Multi-task Regularization Based on Infrequent Classes for Audio Captioning

Emre Cakir, Konstantinos Drossos, Tuomas Virtanen

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

4 Downloads (Pure)


Audio captioning is a multi-modal task, focusing on using natural language for describing the contents of general audio. Most audio captioning methods are based on deep neural networks, employing an encoder-decoder scheme and a dataset with audio clips and corresponding natural language descriptions (i.e. captions). A significant challenge for audio captioning is the distribution of words in the captions: some words are very frequent but acoustically non-informative, i.e. the function words (e.g. "a", "the"), and other words are infrequent but informative, i.e. the content words (e.g. adjectives, nouns). In this paper we propose two methods to mitigate this class imbalance problem. First, in an autoencoder setting for audio captioning, we weigh each word's contribution to the training loss inversely proportional to its number of occurrences in the whole dataset. Secondly, in addition to multi-class, word-level audio captioning task, we define a multi-label side task based on clip-level content word detection by training a separate decoder. We use the loss from the second task to regularize the jointly trained encoder for the audio captioning task. We evaluate our method using Clotho, a recently published, wide-scale audio captioning dataset, and our results show an increase of 37% relative improvement with SPIDEr metric over the baseline method.
Original languageEnglish
Title of host publicationProceedings of the Fifth Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2020))
EditorsNobutaka Ono, Noboru Harada, Yohei Kawaguchi, Annamaria Mesaros, Keisuke Imoto, Yuma Koizumi, Tatsuya Komatsu
Number of pages5
ISBN (Electronic)978-4-600-00566-5
Publication statusPublished - 2020
Publication typeA4 Article in conference proceedings
EventWorkshop on Detection and Classification of Acoustic Scenes and Events - Tokyo, Japan
Duration: 2 Nov 20203 Nov 2020


WorkshopWorkshop on Detection and Classification of Acoustic Scenes and Events
Abbreviated titleDCASE 2020
Internet address


  • audio captioning
  • Clotho
  • multi-task
  • regularization
  • content words
  • infrequent classes

Publication forum classification

  • Publication forum level 0


Dive into the research topics of 'Multi-task Regularization Based on Infrequent Classes for Audio Captioning'. Together they form a unique fingerprint.

Cite this