Self-Supervised Learning of Audio Representations from Audio-Visual Data Using Spatial Alignment

Research output: Contribution to journalArticleScientificpeer-review

4 Downloads (Pure)

Abstract

Learning from audio-visual data offers many possibilities to express correspondence between the audio and visual content, similar to the human perception that relates aural and visual information. In this work, we present a method for self-supervised representation learning based on audio-visual spatial alignment (AVSA), a more sophisticated alignment task than the audio-visual correspondence (AVC). In addition to the correspondence, AVSA also learns from the spatial location of acoustic and visual content. Based on 360° video and Ambisonics audio, we propose selection of visual objects using object detection, and beamforming of the audio signal towards the detected objects, attempting to learn the spatial alignment between objects and the sound they produce. We investigate the use of spatial audio features to represent the audio input, and different audio formats: Ambisonics, mono, and stereo. Experimental results show a 10% improvement on AVSA for the first order ambisonics intensity vector (FOA-IV) in comparison with log-mel spectrogram features; the addition of object-oriented crops also brings significant performance increases for the human action recognition downstream task. A number of audio-only downstream tasks are devised for testing the effectiveness of the learnt audio feature representation, obtaining performance comparable to state-of-the-art methods on acoustic scene classification from ambisonic and binaural audio.

Original languageEnglish
Pages (from-to)1467-1479
Number of pages13
JournalIEEE Journal on Selected Topics in Signal Processing
Volume16
Issue number6
Early online date8 Jun 2022
DOIs
Publication statusPublished - 14 Oct 2022
Publication typeA1 Journal article-refereed

Keywords

  • Audio classification
  • audio-visual corres-pondence
  • audio-visual data
  • audio-visual spatial alignment
  • feature learning
  • self-supervised learning

Publication forum classification

  • Publication forum level 2

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Self-Supervised Learning of Audio Representations from Audio-Visual Data Using Spatial Alignment'. Together they form a unique fingerprint.

Cite this