Leveraging Category Information for Single-Frame Visual Sound Source Separation

Lingyu Zhu; Esa Rahtu

doi:10.1109/EUVIP50544.2021.9484036

Leveraging Category Information for Single-Frame Visual Sound Source Separation

Lingyu Zhu, Esa Rahtu

Computing Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

6 Citations (Scopus)

14 Downloads (Pure)

Abstract

Visual sound source separation aims at identifying sound components from a given sound mixture with the presence of visual cues. Prior works have demonstrated impressive results, but with the expense of large multi-stage architectures and complex data representations (e.g. optical flow trajectories). In contrast, we study simple yet efficient models for visual sound separation using only a single video frame. Furthermore, our models are able to exploit the information of the sound source category in the separation process. To this end, we propose two models where we assume that i) the category labels are available at the training time, or ii) we know if the training sample pairs are from the same or different category. The experiments with the MUSIC dataset show that our model obtains comparable or better performance compared to several recent baseline methods. The code is available at https://github.com/ly-zhu/Leveraging-Category-Information-for-Single-Frame-Visual-Sound-Source-Separation.

Original language	English
Title of host publication	Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021
Editors	A. Beghdadi, F. Alaya Cheikh, J.M.R.S. Tavares, A. Mokraoui, G. Valenzise, L. Oudre, M.A. Qureshi
Publisher	IEEE
Number of pages	6
ISBN (Electronic)	9781665432306
ISBN (Print)	9781665432313
DOIs	https://doi.org/10.1109/EUVIP50544.2021.9484036
Publication status	Published - 20 Jul 2021
Publication type	A4 Article in conference proceedings
Event	European Workshop on Visual Information Processing - Paris, France Duration: 23 Jun 2021 → 25 Jun 2021

Publication series

Name	European Workshop on Visual Information Processing
ISSN (Print)	2164-974X
ISSN (Electronic)	2471-8963

Conference

Conference	European Workshop on Visual Information Processing
Country/Territory	France
City	Paris
Period	23/06/21 → 25/06/21

Keywords

attention mechanism
self-supervised learning
sound source localization
visual sound separation

Publication forum classification

Publication forum level 1

ASJC Scopus subject areas

Computer Vision and Pattern Recognition
Information Systems
Signal Processing

Access to Document

10.1109/EUVIP50544.2021.9484036

EUVIP_2021
©2021 IEEE
Accepted author manuscript, 2.95 MBLicence: Unspecified

https://urn.fi/URN:NBN:fi:tuni-202210267868Licence: Unspecified

Cite this

Zhu, L., & Rahtu, E. (2021). Leveraging Category Information for Single-Frame Visual Sound Source Separation. In A. Beghdadi, F. A. Cheikh, J. M. R. S. Tavares, A. Mokraoui, G. Valenzise, L. Oudre, & M. A. Qureshi (Eds.), Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021 Article 9484036 (European Workshop on Visual Information Processing). IEEE. https://doi.org/10.1109/EUVIP50544.2021.9484036

Zhu, Lingyu ; Rahtu, Esa. / Leveraging Category Information for Single-Frame Visual Sound Source Separation. Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021. editor / A. Beghdadi ; F. Alaya Cheikh ; J.M.R.S. Tavares ; A. Mokraoui ; G. Valenzise ; L. Oudre ; M.A. Qureshi. IEEE, 2021. (European Workshop on Visual Information Processing).

@inproceedings{d128ebf2933a4dbeb9bd120634ebc43e,

title = "Leveraging Category Information for Single-Frame Visual Sound Source Separation",

abstract = "Visual sound source separation aims at identifying sound components from a given sound mixture with the presence of visual cues. Prior works have demonstrated impressive results, but with the expense of large multi-stage architectures and complex data representations (e.g. optical flow trajectories). In contrast, we study simple yet efficient models for visual sound separation using only a single video frame. Furthermore, our models are able to exploit the information of the sound source category in the separation process. To this end, we propose two models where we assume that i) the category labels are available at the training time, or ii) we know if the training sample pairs are from the same or different category. The experiments with the MUSIC dataset show that our model obtains comparable or better performance compared to several recent baseline methods. The code is available at https://github.com/ly-zhu/Leveraging-Category-Information-for-Single-Frame-Visual-Sound-Source-Separation. ",

keywords = "attention mechanism, self-supervised learning, sound source localization, visual sound separation",

author = "Lingyu Zhu and Esa Rahtu",

note = "JUFOID=71968 Funding Information: Acknowledgement This work is supported by the Academy of Finland (projects 327910 & 324346). Publisher Copyright: {\textcopyright} 2021 IEEE.; European Workshop on Visual Information Processing ; Conference date: 23-06-2021 Through 25-06-2021",

year = "2021",

month = jul,

day = "20",

doi = "10.1109/EUVIP50544.2021.9484036",

language = "English",

isbn = "9781665432313",

series = "European Workshop on Visual Information Processing",

publisher = "IEEE",

editor = "A. Beghdadi and Cheikh, {F. Alaya} and J.M.R.S. Tavares and A. Mokraoui and G. Valenzise and L. Oudre and M.A. Qureshi",

booktitle = "Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021",

address = "United States",

}

Zhu, L & Rahtu, E 2021, Leveraging Category Information for Single-Frame Visual Sound Source Separation. in A Beghdadi, FA Cheikh, JMRS Tavares, A Mokraoui, G Valenzise, L Oudre & MA Qureshi (eds), Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021., 9484036, European Workshop on Visual Information Processing, IEEE, European Workshop on Visual Information Processing, Paris, France, 23/06/21. https://doi.org/10.1109/EUVIP50544.2021.9484036

Leveraging Category Information for Single-Frame Visual Sound Source Separation. / Zhu, Lingyu; Rahtu, Esa.
Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021. ed. / A. Beghdadi; F. Alaya Cheikh; J.M.R.S. Tavares; A. Mokraoui; G. Valenzise; L. Oudre; M.A. Qureshi. IEEE, 2021. 9484036 (European Workshop on Visual Information Processing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Leveraging Category Information for Single-Frame Visual Sound Source Separation

AU - Zhu, Lingyu

AU - Rahtu, Esa

PY - 2021/7/20

Y1 - 2021/7/20

N2 - Visual sound source separation aims at identifying sound components from a given sound mixture with the presence of visual cues. Prior works have demonstrated impressive results, but with the expense of large multi-stage architectures and complex data representations (e.g. optical flow trajectories). In contrast, we study simple yet efficient models for visual sound separation using only a single video frame. Furthermore, our models are able to exploit the information of the sound source category in the separation process. To this end, we propose two models where we assume that i) the category labels are available at the training time, or ii) we know if the training sample pairs are from the same or different category. The experiments with the MUSIC dataset show that our model obtains comparable or better performance compared to several recent baseline methods. The code is available at https://github.com/ly-zhu/Leveraging-Category-Information-for-Single-Frame-Visual-Sound-Source-Separation.

AB - Visual sound source separation aims at identifying sound components from a given sound mixture with the presence of visual cues. Prior works have demonstrated impressive results, but with the expense of large multi-stage architectures and complex data representations (e.g. optical flow trajectories). In contrast, we study simple yet efficient models for visual sound separation using only a single video frame. Furthermore, our models are able to exploit the information of the sound source category in the separation process. To this end, we propose two models where we assume that i) the category labels are available at the training time, or ii) we know if the training sample pairs are from the same or different category. The experiments with the MUSIC dataset show that our model obtains comparable or better performance compared to several recent baseline methods. The code is available at https://github.com/ly-zhu/Leveraging-Category-Information-for-Single-Frame-Visual-Sound-Source-Separation.

KW - attention mechanism

KW - self-supervised learning

KW - sound source localization

KW - visual sound separation

U2 - 10.1109/EUVIP50544.2021.9484036

DO - 10.1109/EUVIP50544.2021.9484036

M3 - Conference contribution

AN - SCOPUS:85111421997

SN - 9781665432313

T3 - European Workshop on Visual Information Processing

BT - Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021

A2 - Beghdadi, A.

A2 - Cheikh, F. Alaya

A2 - Tavares, J.M.R.S.

A2 - Mokraoui, A.

A2 - Valenzise, G.

A2 - Oudre, L.

A2 - Qureshi, M.A.

PB - IEEE

T2 - European Workshop on Visual Information Processing

Y2 - 23 June 2021 through 25 June 2021

ER -

Zhu L, Rahtu E. Leveraging Category Information for Single-Frame Visual Sound Source Separation. In Beghdadi A, Cheikh FA, Tavares JMRS, Mokraoui A, Valenzise G, Oudre L, Qureshi MA, editors, Proceedings of the 2021 9th European Workshop on Visual Information Processing, EUVIP 2021. IEEE. 2021. 9484036. (European Workshop on Visual Information Processing). doi: 10.1109/EUVIP50544.2021.9484036

Leveraging Category Information for Single-Frame Visual Sound Source Separation

Abstract

Publication series

Conference

Keywords

Publication forum classification

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this