Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases

Huang Xie, Okko Räsänen, Konstantinos Drossos, Tuomas Virtanen

Tutkimustuotos: KonferenssiartikkeliScientificvertaisarvioitu

4 Lataukset (Pure)

Abstrakti

We investigate unsupervised learning of correspondences between sound events and textual phrases through aligning audio clips with textual captions describing the content of a whole audio clip. We align originally unaligned and unannotated audio clips and their captions by scoring the similarities between audio frames and words, as encoded by modality-specific encoders and using a ranking-loss criterion to optimize the model. After training, we obtain clip-caption similarity by averaging frame-word similarities and estimate event-phrase correspondences by calculating frame-phrase similarities. We evaluate the method with two cross-modal tasks: audio-caption retrieval, and phrase-based sound event detection (SED). Experimental results show that the proposed method can globally associate audio clips with captions as well as locally learn correspondences between individual sound events and textual phrases in an unsupervised manner.
AlkuperäiskieliEnglanti
OtsikkoICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
KustantajaIEEE
Sivut8867-8871
Sivumäärä5
ISBN (elektroninen)978-1-6654-0540-9
ISBN (painettu)978-1-6654-0541-6
DOI - pysyväislinkit
TilaJulkaistu - toukok. 2022
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaIEEE International Conference on Acoustics, Speech and Signal Processing - Singapore, Singapore
Kesto: 23 toukok. 202227 toukok. 2022

Julkaisusarja

NimiProceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing
ISSN (elektroninen)2379-190X

Conference

ConferenceIEEE International Conference on Acoustics, Speech and Signal Processing
Maa/AlueSingapore
KaupunkiSingapore
Ajanjakso23/05/2227/05/22

Julkaisufoorumi-taso

  • Jufo-taso 1

Sormenjälki

Sukella tutkimusaiheisiin 'Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä