Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models

Systems that can find correspondences between multiple modal-
ities, such as between speech and images, have great potential
to solve different recognition and data analysis tasks in an un-
supervised manner. This work studies multimodal learning in
the context of visually grounded speech (VGS) models, and fo-
cuses on their recently demonstrated capability to extract spa-
tiotemporal alignments between spoken words and the corre-
sponding visual objects without ever been explicitly trained for
object localization or word recognition. As the main contribu-
tions, we formalize the alignment problem in terms of an au-
diovisual alignment tensor that is based on earlier VGS work,
introduce systematic metrics for evaluating model performance
in aligning visual objects and spoken words, and propose a new
VGS model variant for the alignment task utilizing cross-modal
attention layer. We test our model and a previously proposed
model in the alignment task using SPEECH-COCO captions
coupled with MSCOCO images. We compare the alignment
performance using our proposed evaluation metrics to the se-
mantic retrieval task commonly used to evaluate VGS models.
We show that cross-modal attention layer not only helps the
model to achieve higher semantic cross-modal retrieval perfor-
mance, but also leads to substantial improvements in the align-
ment performance between image object and spoken words.