Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network

Lingyu Zhu; Esa Rahtu

doi:10.1007/978-3-030-69544-6_25

Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network

Lingyu Zhu, Esa Rahtu

Computing Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

3 Citations (Scopus)

8 Downloads (Pure)

Abstract

The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the source separation. A key element in COF is a novel opponent filter module that identifies and relocates residual components between sources. The system is guided by the appearance and motion of the source, and, for this purpose, we study different representations based on video frames, optical flows, dynamic images, and their combinations. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained in an end-to-end manner using a large set of unlabelled videos. We compare COF with recent baselines and obtain the state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL).

Original language	English
Title of host publication	Computer Vision – ACCV 2020
Subtitle of host publication	15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI
Editors	Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, Jianbo Shi
Publisher	Springer
Pages	409-426
Number of pages	18
ISBN (Electronic)	978-3-030-69544-6
ISBN (Print)	978-3-030-69543-9
DOIs	https://doi.org/10.1007/978-3-030-69544-6_25
Publication status	Published - 26 Feb 2021
Publication type	A4 Article in conference proceedings
Event	Asian Conference on Computer Vision - Virtual, Online Duration: 30 Nov 2020 → 4 Dec 2020 Conference number: 15

Publication series

Name	Lecture Notes in Computer Science
Volume	12627
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Asian Conference on Computer Vision
Period	30/11/20 → 4/12/20

Keywords

audio-visual learning
sound source separation
sound source localization
opponent filter
dynamic image

Publication forum classification

Publication forum level 1

Access to Document

10.1007/978-3-030-69544-6_25

accv2020CameraReadyAccepted author manuscript, 3.46 MBLicence: Unspecified

https://urn.fi/URN:NBN:fi:tuni-202210267869Licence: Unspecified

Cite this

Zhu, L., & Rahtu, E. (2021). Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network. In H. Ishikawa, C.-L. Liu, T. Pajdla, & J. Shi (Eds.), Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI (pp. 409-426). (Lecture Notes in Computer Science; Vol. 12627). Springer. https://doi.org/10.1007/978-3-030-69544-6_25

Zhu, Lingyu ; Rahtu, Esa. / Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network. Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI. editor / Hiroshi Ishikawa ; Cheng-Lin Liu ; Tomas Pajdla ; Jianbo Shi. Springer, 2021. pp. 409-426 (Lecture Notes in Computer Science).

@inproceedings{9ba44d4207d74270906076eaf3322006,

title = "Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network",

abstract = "The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the source separation. A key element in COF is a novel opponent filter module that identifies and relocates residual components between sources. The system is guided by the appearance and motion of the source, and, for this purpose, we study different representations based on video frames, optical flows, dynamic images, and their combinations. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained in an end-to-end manner using a large set of unlabelled videos. We compare COF with recent baselines and obtain the state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL).",

keywords = "audio-visual learning, sound source separation, sound source localization, opponent filter, dynamic image",

author = "Lingyu Zhu and Esa Rahtu",

note = "JUFOID=62555 Funding Information: This work is supported by the Academy of Finland (projects 327910 & 324346). Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; Asian Conference on Computer Vision ; Conference date: 30-11-2020 Through 04-12-2020",

year = "2021",

month = feb,

day = "26",

doi = "10.1007/978-3-030-69544-6_25",

language = "English",

isbn = "978-3-030-69543-9",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "409--426",

editor = "Hiroshi Ishikawa and Cheng-Lin Liu and Tomas Pajdla and Jianbo Shi",

booktitle = "Computer Vision – ACCV 2020",

}

Zhu, L & Rahtu, E 2021, Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network. in H Ishikawa, C-L Liu, T Pajdla & J Shi (eds), Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI. Lecture Notes in Computer Science, vol. 12627, Springer, pp. 409-426, Asian Conference on Computer Vision, 30/11/20. https://doi.org/10.1007/978-3-030-69544-6_25

Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network. / Zhu, Lingyu; Rahtu, Esa.
Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI. ed. / Hiroshi Ishikawa; Cheng-Lin Liu; Tomas Pajdla; Jianbo Shi. Springer, 2021. p. 409-426 (Lecture Notes in Computer Science; Vol. 12627).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network

AU - Zhu, Lingyu

AU - Rahtu, Esa

N1 - Conference code: 15

PY - 2021/2/26

Y1 - 2021/2/26

N2 - The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the source separation. A key element in COF is a novel opponent filter module that identifies and relocates residual components between sources. The system is guided by the appearance and motion of the source, and, for this purpose, we study different representations based on video frames, optical flows, dynamic images, and their combinations. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained in an end-to-end manner using a large set of unlabelled videos. We compare COF with recent baselines and obtain the state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL).

AB - The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the source separation. A key element in COF is a novel opponent filter module that identifies and relocates residual components between sources. The system is guided by the appearance and motion of the source, and, for this purpose, we study different representations based on video frames, optical flows, dynamic images, and their combinations. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained in an end-to-end manner using a large set of unlabelled videos. We compare COF with recent baselines and obtain the state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL).

KW - audio-visual learning

KW - sound source separation

KW - sound source localization

KW - opponent filter

KW - dynamic image

U2 - 10.1007/978-3-030-69544-6_25

DO - 10.1007/978-3-030-69544-6_25

M3 - Conference contribution

SN - 978-3-030-69543-9

T3 - Lecture Notes in Computer Science

SP - 409

EP - 426

BT - Computer Vision – ACCV 2020

A2 - Ishikawa, Hiroshi

A2 - Liu, Cheng-Lin

A2 - Pajdla, Tomas

A2 - Shi, Jianbo

PB - Springer

T2 - Asian Conference on Computer Vision

Y2 - 30 November 2020 through 4 December 2020

ER -

Zhu L, Rahtu E. Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network. In Ishikawa H, Liu CL, Pajdla T, Shi J, editors, Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI. Springer. 2021. p. 409-426. (Lecture Notes in Computer Science). doi: 10.1007/978-3-030-69544-6_25

Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network

Abstract

Publication series

Conference

Keywords

Publication forum classification

Access to Document

Fingerprint

Cite this