Improving Multimodal Learning with Multi-Loss Gradient Modulation


Konstantinos Kontras (Department of Electrical Engineering, KU Leuven, Belgium, KU Leuven), Christos Chatzichristos (KU Leuven), Matthew B. Blaschko (KU Leuven), Maarten De Vos (KU Leuven)
The 35th British Machine Vision Conference
PDFPosterSupplementary

Abstract

Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, enhancing robustness, and improving contextual understanding and performance. However, combining such modalities presents chal- lenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of informa- tion from other modalities and leading to sub-optimal model performance. To address this issue the vast majority of previous works suggests to assess the unimodal contribu- tions and dynamically adjust the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon con- vergence. We achieve superior results across three audio-video datasets: on CREMA-D, models with ResNet backbone encoders surpass the previous best by 1.9% to 12.4%, and Conformer backbone models deliver improvements ranging from 2.8% to 14.1% across different fusion methods. On AVE, improvements range from 2.7% to 7.7%, while on UCF101, gains reach up to 6.1%. Finally, we demonstrate on the CMU-MOSEI dataset, that our approach is valid when three modalities (audio, video, and text) are available.

Citation

@inproceedings{Kontras_2024_BMVC,
author    = {Konstantinos Kontras and Christos Chatzichristos and Matthew B. Blaschko and Maarten De Vos},
title     = {Improving Multimodal Learning with Multi-Loss Gradient Modulation},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0977.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection