Evaluating Classification Systems Against Soft Labels with Fuzzy Precision and Recall

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

8 Downloads (Pure)

Abstract

Classification systems are normally trained by minimizing the cross-entropy between system outputs and reference labels, which makes the Kullback-Leibler divergence a natural choice for measuring how closely the system can follow the data. Precision and recall provide another perspective for measuring the performance of a classification system. Non-binary references can arise from various sources, and it is often beneficial to use the soft labels for training instead of the binarized data. However, the existing definitions for precision and recall require binary reference labels, and binarizing the data can cause erroneous interpretations. We present a novel method to calculate precision, recall and F-score without quantizing the data. The proposed metrics extend the well established metrics as the definitions coincide when used with binary labels. To understand the behavior of the metrics we show simple example cases and an evaluation of different sound event detection models trained on real data with soft labels.
Original languageEnglish
Title of host publicationProceedings of the 8th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2023)
EditorsMagdalena Fuentes, Toni Heittola, Keisuke Imoto, Annamaria Mesaros, Archontis Politis, Romain Serizel, Tuomas Virtanen
Pages46-50
ISBN (Electronic)978-952-03-3171-9
Publication statusPublished - 2023
Publication typeA4 Article in conference proceedings
EventWorkshop on Detection and Classification of Acoustic Scenes and Events - Tampere, Finland
Duration: 20 Sept 202322 Sept 2023

Conference

ConferenceWorkshop on Detection and Classification of Acoustic Scenes and Events
Country/TerritoryFinland
CityTampere
Period20/09/2322/09/23

Publication forum classification

  • Publication forum level 1

Fingerprint

Dive into the research topics of 'Evaluating Classification Systems Against Soft Labels with Fuzzy Precision and Recall'. Together they form a unique fingerprint.

Cite this