Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network

Lingyu Zhu, Esa Rahtu

Tutkimustuotos: KonferenssiartikkeliTieteellinenvertaisarvioitu

3 Sitaatiot (Scopus)
11 Lataukset (Pure)

Abstrakti

The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the source separation. A key element in COF is a novel opponent filter module that identifies and relocates residual components between sources. The system is guided by the appearance and motion of the source, and, for this purpose, we study different representations based on video frames, optical flows, dynamic images, and their combinations. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained in an end-to-end manner using a large set of unlabelled videos. We compare COF with recent baselines and obtain the state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL).
AlkuperäiskieliEnglanti
OtsikkoComputer Vision – ACCV 2020
Alaotsikko15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI
ToimittajatHiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, Jianbo Shi
KustantajaSpringer
Sivut409-426
Sivumäärä18
ISBN (elektroninen)978-3-030-69544-6
ISBN (painettu)978-3-030-69543-9
DOI - pysyväislinkit
TilaJulkaistu - 26 helmik. 2021
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaAsian Conference on Computer Vision - Virtual, Online
Kesto: 30 marrask. 20204 jouluk. 2020
Konferenssinumero: 15

Julkaisusarja

NimiLecture Notes in Computer Science
Vuosikerta12627
ISSN (painettu)0302-9743
ISSN (elektroninen)1611-3349

Conference

ConferenceAsian Conference on Computer Vision
Ajanjakso30/11/204/12/20

Julkaisufoorumi-taso

  • Jufo-taso 1

Sormenjälki

Sukella tutkimusaiheisiin 'Visually Guided Sound Source Separation Using Cascaded Opponent Filter Network'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä