Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study

Annamaria Mesaros; Toni Heittola; Tuomas Virtanen

doi:10.1109/WASPAA.2017.8170047

Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study

Annamaria Mesaros, Toni Heittola, Tuomas Virtanen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

17 Citations (Scopus)

88 Downloads (Pure)

Abstract

Human and machine performance in acoustic scene classification is examined through a parallel experiment using TUT Acoustic Scenes 2016 dataset. The machine learning perspective is presented based on the systems submitted for the 2016 challenge on Detection and Classification of Acoustic Scenes and Events. The human performance, assessed through a listening experiment, was found to be significantly lower than machine performance. Test subjects exhibited different behavior throughout the experiment, leading to significant differences in performance between groups of subjects. An expert listener trained for the task obtained similar accuracy to the average of submitted systems, comparable also to previous studies of human abilities in recognizing everyday acoustic scenes.

Original language	English
Title of host publication	2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)
Publisher	IEEE Computer Society
Pages	319–323
ISBN (Print)	978-1-5386-1631-4
DOIs	https://doi.org/10.1109/WASPAA.2017.8170047
Publication status	Published - 2017
Publication type	A4 Article in conference proceedings
Event	IEEE Workshop on Applications of Signal Processing to Audio and Acoustics - Duration: 1 Jan 1900 → …

Conference

Conference	IEEE Workshop on Applications of Signal Processing to Audio and Acoustics
Period	1/01/00 → …

Keywords

acoustic scene classification
machine learning
human performance
listening experiment

Publication forum classification

Publication forum level 1

Access to Document

10.1109/WASPAA.2017.8170047

mesaros-waspaa2017-humans-vs-machines-asc
© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Accepted author manuscript, 1.13 MBLicence: Other

http://urn.fi/URN:NBN:fi:tty-201802081199

Cite this

@inproceedings{27c5816d5650442eab3ee4c4ab721bdd,

title = "Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study",

abstract = "Human and machine performance in acoustic scene classification is examined through a parallel experiment using TUT Acoustic Scenes 2016 dataset. The machine learning perspective is presented based on the systems submitted for the 2016 challenge on Detection and Classification of Acoustic Scenes and Events. The human performance, assessed through a listening experiment, was found to be significantly lower than machine performance. Test subjects exhibited different behavior throughout the experiment, leading to significant differences in performance between groups of subjects. An expert listener trained for the task obtained similar accuracy to the average of submitted systems, comparable also to previous studies of human abilities in recognizing everyday acoustic scenes.",

keywords = "acoustic scene classification, machine learning, human performance, listening experiment",

author = "Annamaria Mesaros and Toni Heittola and Tuomas Virtanen",

year = "2017",

doi = "10.1109/WASPAA.2017.8170047",

language = "English",

isbn = "978-1-5386-1631-4",

pages = " 319–323",

booktitle = "2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)",

publisher = "IEEE Computer Society",

address = "United States",

note = "IEEE Workshop on Applications of Signal Processing to Audio and Acoustics ; Conference date: 01-01-1900",

}

Mesaros, A , Heittola, T & Virtanen, T 2017, Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study. in 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE Computer Society, pp. 319–323, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 1/01/00. https://doi.org/10.1109/WASPAA.2017.8170047

Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study. / Mesaros, Annamaria ; Heittola, Toni ; Virtanen, Tuomas.
2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE Computer Society, 2017. p. 319–323.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study

AU - Mesaros, Annamaria

AU - Heittola, Toni

AU - Virtanen, Tuomas

PY - 2017

Y1 - 2017

N2 - Human and machine performance in acoustic scene classification is examined through a parallel experiment using TUT Acoustic Scenes 2016 dataset. The machine learning perspective is presented based on the systems submitted for the 2016 challenge on Detection and Classification of Acoustic Scenes and Events. The human performance, assessed through a listening experiment, was found to be significantly lower than machine performance. Test subjects exhibited different behavior throughout the experiment, leading to significant differences in performance between groups of subjects. An expert listener trained for the task obtained similar accuracy to the average of submitted systems, comparable also to previous studies of human abilities in recognizing everyday acoustic scenes.

AB - Human and machine performance in acoustic scene classification is examined through a parallel experiment using TUT Acoustic Scenes 2016 dataset. The machine learning perspective is presented based on the systems submitted for the 2016 challenge on Detection and Classification of Acoustic Scenes and Events. The human performance, assessed through a listening experiment, was found to be significantly lower than machine performance. Test subjects exhibited different behavior throughout the experiment, leading to significant differences in performance between groups of subjects. An expert listener trained for the task obtained similar accuracy to the average of submitted systems, comparable also to previous studies of human abilities in recognizing everyday acoustic scenes.

KW - acoustic scene classification

KW - machine learning

KW - human performance

KW - listening experiment

U2 - 10.1109/WASPAA.2017.8170047

DO - 10.1109/WASPAA.2017.8170047

M3 - Conference contribution

SN - 978-1-5386-1631-4

SP - 319

EP - 323

BT - 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)

PB - IEEE Computer Society

T2 - IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

Y2 - 1 January 1900

ER -

Assessment of human and machine performance in acoustic scene classification: dcase 2016 case study

Abstract

Conference

Keywords

Publication forum classification

Access to Document

Fingerprint

Cite this