A curated dataset of urban scenes for audio-visual scene analysis

Shanshan Wang; Annamaria Mesaros; Toni Heittola; Tuomas Virtanen

doi:10.48550/arXiv.2011.00030

A curated dataset of urban scenes for audio-visual scene analysis

Shanshan Wang
, Annamaria Mesaros
, Toni Heittola
, Tuomas Virtanen

Computing Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

43 Citations (Scopus)

29 Downloads (Pure)

Abstract

This paper introduces a curated dataset of urban scenes for audio-visual scene analysis which consists of carefully selected and recorded material. The data was recorded in multiple European cities, using the same equipment, in multiple locations for each scene, and is openly available. We also present a case study for audio-visual scene recognition and show that joint modeling of audio and visual modalities brings significant performance gain compared to state of the art uni-modal systems. Our approach obtained an 84.8% accuracy compared to 75.8% for the audio-only and 68.4% for the video-only equivalent systems.

Original language	English
Title of host publication	ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher	IEEE
Pages	626-630
Number of pages	5
ISBN (Electronic)	978-1-7281-7605-5
DOIs	https://doi.org/10.48550/arXiv.2011.00030 https://doi.org/10.1109/ICASSP39728.2021.9415085
Publication status	Published - 2021
Publication type	A4 Article in conference proceedings
Event	IEEE International Conference on Acoustics, Speech and Signal Processing - Metro Toronto Convention Centre, Toronto, Canada Duration: 6 Jun 2021 → 11 Jun 2021 https://2021.ieeeicassp.org

Publication series

Name	Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing
ISSN (Print)	1520-6149

Conference

Conference	IEEE International Conference on Acoustics, Speech and Signal Processing
Country/Territory	Canada
City	Toronto
Period	6/06/21 → 11/06/21
Internet address	https://2021.ieeeicassp.org

Funding

This work was supported in part by the European Research Council under the European Unions H2020 Framework Programme through ERC Grant Agreement 637422 EVERYSOUND. The authors wish to thank CSC-IT Centre of Science Ltd., Finland, for providing computational resources. This work was supported in part by the European Research Council under the European Unions H2020 Framework Programme through ERC Grant Agreement 637422 EVERYSOUND. The authors wish to thank CSCIT Centre of Science Ltd., Finland, for providing computational resources.

Keywords

Acoustic scene
Audio-visual data
Pattern recognition
Scene analysis
Transfer learning

Publication forum classification

Publication forum level 1

ASJC Scopus subject areas

Software
Signal Processing
Electrical and Electronic Engineering

Access to Document

wang_ICASSP2021
This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Accepted author manuscript, 127 KBLicence: Unspecified

https://urn.fi/URN:NBN:fi:tuni-202203032286

Cite this

Wang, S., Mesaros, A., Heittola, T., & Virtanen, T. (2021). A curated dataset of urban scenes for audio-visual scene analysis. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 626-630). (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing). IEEE. https://doi.org/10.48550/arXiv.2011.00030, https://doi.org/10.1109/ICASSP39728.2021.9415085

@inproceedings{80ab5a44f68545b09081153cf22f27a1,

title = "A curated dataset of urban scenes for audio-visual scene analysis",

abstract = "This paper introduces a curated dataset of urban scenes for audio-visual scene analysis which consists of carefully selected and recorded material. The data was recorded in multiple European cities, using the same equipment, in multiple locations for each scene, and is openly available. We also present a case study for audio-visual scene recognition and show that joint modeling of audio and visual modalities brings significant performance gain compared to state of the art uni-modal systems. Our approach obtained an 84.8\% accuracy compared to 75.8\% for the audio-only and 68.4\% for the video-only equivalent systems.",

keywords = "Acoustic scene, Audio-visual data, Pattern recognition, Scene analysis, Transfer learning",

author = "Shanshan Wang and Annamaria Mesaros and Toni Heittola and Tuomas Virtanen",

note = "Funding Information: This work was supported in part by the European Research Council under the European Unions H2020 Framework Programme through ERC Grant Agreement 637422 EVERYSOUND. The authors wish to thank CSC-IT Centre of Science Ltd., Finland, for providing computational resources. Funding Information: This work was supported in part by the European Research Council under the European Unions H2020 Framework Programme through ERC Grant Agreement 637422 EVERYSOUND. The authors wish to thank CSCIT Centre of Science Ltd., Finland, for providing computational resources. Publisher Copyright: {\textcopyright} 2021 IEEE. jufoid=57409; IEEE International Conference on Acoustics, Speech and Signal Processing ; Conference date: 06-06-2021 Through 11-06-2021",

year = "2021",

doi = "10.48550/arXiv.2011.00030",

language = "English",

series = "Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing",

publisher = "IEEE",

pages = "626--630",

booktitle = "ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

address = "United States",

url = "https://2021.ieeeicassp.org",

}

Wang, S, Mesaros, A, Heittola, T & Virtanen, T 2021, A curated dataset of urban scenes for audio-visual scene analysis. in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, IEEE, pp. 626-630, IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, Ontario, Canada, 6/06/21. https://doi.org/10.48550/arXiv.2011.00030, https://doi.org/10.1109/ICASSP39728.2021.9415085

A curated dataset of urban scenes for audio-visual scene analysis. / Wang, Shanshan; Mesaros, Annamaria; Heittola, Toni et al.
ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021. p. 626-630 (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - A curated dataset of urban scenes for audio-visual scene analysis

AU - Wang, Shanshan

AU - Mesaros, Annamaria

AU - Heittola, Toni

AU - Virtanen, Tuomas

N1 - Funding Information: This work was supported in part by the European Research Council under the European Unions H2020 Framework Programme through ERC Grant Agreement 637422 EVERYSOUND. The authors wish to thank CSC-IT Centre of Science Ltd., Finland, for providing computational resources. Funding Information: This work was supported in part by the European Research Council under the European Unions H2020 Framework Programme through ERC Grant Agreement 637422 EVERYSOUND. The authors wish to thank CSCIT Centre of Science Ltd., Finland, for providing computational resources. Publisher Copyright: © 2021 IEEE. jufoid=57409

PY - 2021

Y1 - 2021

N2 - This paper introduces a curated dataset of urban scenes for audio-visual scene analysis which consists of carefully selected and recorded material. The data was recorded in multiple European cities, using the same equipment, in multiple locations for each scene, and is openly available. We also present a case study for audio-visual scene recognition and show that joint modeling of audio and visual modalities brings significant performance gain compared to state of the art uni-modal systems. Our approach obtained an 84.8% accuracy compared to 75.8% for the audio-only and 68.4% for the video-only equivalent systems.

AB - This paper introduces a curated dataset of urban scenes for audio-visual scene analysis which consists of carefully selected and recorded material. The data was recorded in multiple European cities, using the same equipment, in multiple locations for each scene, and is openly available. We also present a case study for audio-visual scene recognition and show that joint modeling of audio and visual modalities brings significant performance gain compared to state of the art uni-modal systems. Our approach obtained an 84.8% accuracy compared to 75.8% for the audio-only and 68.4% for the video-only equivalent systems.

KW - Acoustic scene

KW - Audio-visual data

KW - Pattern recognition

KW - Scene analysis

KW - Transfer learning

U2 - 10.48550/arXiv.2011.00030

DO - 10.48550/arXiv.2011.00030

M3 - Conference contribution

AN - SCOPUS:85108360575

T3 - Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing

SP - 626

EP - 630

BT - ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - IEEE

T2 - IEEE International Conference on Acoustics, Speech and Signal Processing

Y2 - 6 June 2021 through 11 June 2021

ER -

Wang S, Mesaros A, Heittola T, Virtanen T. A curated dataset of urban scenes for audio-visual scene analysis. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2021. p. 626-630. (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing). doi: 10.48550/arXiv.2011.00030, 10.1109/ICASSP39728.2021.9415085

A curated dataset of urban scenes for audio-visual scene analysis

Abstract

Publication series

Conference

Funding

Keywords

Publication forum classification

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this