Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions

Shanshan Wang; Toni Heittola; Annamaria Mesaros; Tuomas Virtanen

doi:10.5281/zenodo.5770113

Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions

Shanshan Wang, Toni Heittola, Annamaria Mesaros, Tuomas Virtanen

Computing Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

4 Downloads (Pure)

Abstract

This paper presents the details of the Audio-Visual Scene Classification task in the DCASE 2021 Challenge (Task 1 Subtask B). The task is concerned with classification using audio and video modalities, using a dataset of synchronized recordings. This task has attracted 43 submissions from 13 different teams around the world. Among all submissions, more than half of the submitted systems have better performance than the baseline. The common techniques among the top systems are the usage of large pretrained models such as ResNet or EfficientNet which are trained for the task-specific problem. Fine-tuning, transfer learning, and data augmentation techniques are also employed to boost the performance. More importantly, multi-modal methods using both audio and video are employed by all the top 5 teams. The best system among all achieved a logloss of 0.195 and accuracy of 93.8\%, compared to the baseline system with logloss of 0.662 and accuracy of 77.1%.

Original language	English
Title of host publication	Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021)
Editors	Frederic Font, Annamaria Mesaros, Daniel P.W. Ellis, Eduardo Fonseca, Magdalena Fuentes, Benjamin Elizalde
Pages	45-49
ISBN (Electronic)	978-84-09-36072-7
DOIs	https://doi.org/10.5281/zenodo.5770113
Publication status	Published - 15 Nov 2021
Publication type	A4 Article in conference proceedings
Event	Detection and Classication of Acoustic Scenes and Events - , Spain Duration: 15 Nov 2021 → 19 Nov 2021

Conference

Conference	Detection and Classication of Acoustic Scenes and Events
Country/Territory	Spain
Period	15/11/21 → 19/11/21

Publication forum classification

Publication forum level 0

Access to Document

10.5281/zenodo.5770113Licence: CC BY

DCASE2021Workshop_Wang_20-1Final published version, 647 KBLicence: CC BY

Cite this

Wang, S., Heittola, T., Mesaros, A., & Virtanen, T. (2021). Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions. In F. Font, A. Mesaros, D. P.W. Ellis, E. Fonseca, M. Fuentes, & B. Elizalde (Eds.), Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021) (pp. 45-49) https://doi.org/10.5281/zenodo.5770113

Wang, Shanshan ; Heittola, Toni ; Mesaros, Annamaria et al. / Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions. Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021). editor / Frederic Font ; Annamaria Mesaros ; Daniel P.W. Ellis ; Eduardo Fonseca ; Magdalena Fuentes ; Benjamin Elizalde. 2021. pp. 45-49

@inproceedings{99f523461df840ecb5bdbc8deb769dce,

title = "Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions",

abstract = "This paper presents the details of the Audio-Visual Scene Classification task in the DCASE 2021 Challenge (Task 1 Subtask B). The task is concerned with classification using audio and video modalities, using a dataset of synchronized recordings. This task has attracted 43 submissions from 13 different teams around the world. Among all submissions, more than half of the submitted systems have better performance than the baseline. The common techniques among the top systems are the usage of large pretrained models such as ResNet or EfficientNet which are trained for the task-specific problem. Fine-tuning, transfer learning, and data augmentation techniques are also employed to boost the performance. More importantly, multi-modal methods using both audio and video are employed by all the top 5 teams. The best system among all achieved a logloss of 0.195 and accuracy of 93.8\%, compared to the baseline system with logloss of 0.662 and accuracy of 77.1%.",

author = "Shanshan Wang and Toni Heittola and Annamaria Mesaros and Tuomas Virtanen",

year = "2021",

month = nov,

day = "15",

doi = "10.5281/zenodo.5770113",

language = "English",

pages = "45--49",

editor = "Frederic Font and Annamaria Mesaros and {P.W. Ellis}, Daniel and Eduardo Fonseca and Magdalena Fuentes and Benjamin Elizalde",

booktitle = "Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021)",

note = "Detection and Classication of Acoustic Scenes and Events ; Conference date: 15-11-2021 Through 19-11-2021",

}

Wang, S , Heittola, T , Mesaros, A & Virtanen, T 2021, Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions. in F Font, A Mesaros, D P.W. Ellis, E Fonseca, M Fuentes & B Elizalde (eds), Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021). pp. 45-49, Detection and Classication of Acoustic Scenes and Events, Spain, 15/11/21. https://doi.org/10.5281/zenodo.5770113

Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions. / Wang, Shanshan ; Heittola, Toni ; Mesaros, Annamaria et al.

Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021). ed. / Frederic Font; Annamaria Mesaros; Daniel P.W. Ellis; Eduardo Fonseca; Magdalena Fuentes; Benjamin Elizalde. 2021. p. 45-49.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions

AU - Wang, Shanshan

AU - Heittola, Toni

AU - Mesaros, Annamaria

AU - Virtanen, Tuomas

PY - 2021/11/15

Y1 - 2021/11/15

N2 - This paper presents the details of the Audio-Visual Scene Classification task in the DCASE 2021 Challenge (Task 1 Subtask B). The task is concerned with classification using audio and video modalities, using a dataset of synchronized recordings. This task has attracted 43 submissions from 13 different teams around the world. Among all submissions, more than half of the submitted systems have better performance than the baseline. The common techniques among the top systems are the usage of large pretrained models such as ResNet or EfficientNet which are trained for the task-specific problem. Fine-tuning, transfer learning, and data augmentation techniques are also employed to boost the performance. More importantly, multi-modal methods using both audio and video are employed by all the top 5 teams. The best system among all achieved a logloss of 0.195 and accuracy of 93.8\%, compared to the baseline system with logloss of 0.662 and accuracy of 77.1%.

AB - This paper presents the details of the Audio-Visual Scene Classification task in the DCASE 2021 Challenge (Task 1 Subtask B). The task is concerned with classification using audio and video modalities, using a dataset of synchronized recordings. This task has attracted 43 submissions from 13 different teams around the world. Among all submissions, more than half of the submitted systems have better performance than the baseline. The common techniques among the top systems are the usage of large pretrained models such as ResNet or EfficientNet which are trained for the task-specific problem. Fine-tuning, transfer learning, and data augmentation techniques are also employed to boost the performance. More importantly, multi-modal methods using both audio and video are employed by all the top 5 teams. The best system among all achieved a logloss of 0.195 and accuracy of 93.8\%, compared to the baseline system with logloss of 0.662 and accuracy of 77.1%.

U2 - 10.5281/zenodo.5770113

DO - 10.5281/zenodo.5770113

M3 - Conference contribution

SP - 45

EP - 49

BT - Proceedings of the 6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2021)

A2 - Font, Frederic

A2 - Mesaros, Annamaria

A2 - P.W. Ellis, Daniel

A2 - Fonseca, Eduardo

A2 - Fuentes, Magdalena

A2 - Elizalde, Benjamin

T2 - Detection and Classication of Acoustic Scenes and Events

Y2 - 15 November 2021 through 19 November 2021

ER -

Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions

Abstract

Conference

Publication forum classification

Access to Document

Fingerprint

Cite this