Sound event detection with soft labels: a new perspective on evaluation

Sound event detection has been an essential task in the DCASE Challenge since the beginning, with various alterations over the years. The 2023 Challenge presented for the first time a sound event detection task for which the reference labels representing sound class activity were provided as real numbers on the interval from zero to one, in addition to binary labels. In this paper we provide an overview of the sound event detection with soft labels task in DCASE 2023 Challenge, and re-evaluate the challenge submissions using a soft metric. The use of a soft metric allows computing precision, recall and F-score directly using the soft labels, and thus avoids the optimization step for binarizing both the reference and predictions using a threshold. We analyze the behavior of the soft metric on a large number of systems, and show that for the softly labeled reference data, the results obtained with the soft metrics represent very well the system’s ability to follow the data, and is a good proxy for entropy-based measures.