InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning


Babak Ehteshami Bejnordi (QualComm), Gaurav Kumar (QualComm), Amelie Royer (Kyutai), Christos Louizos (QualComm), Tijmen Blankevoort (Facebook), Mohsen Ghafoorian (Qualcomm)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Jointly learning multiple tasks with a unified model can improve accuracy and data efficiency, but faces the challenge of task interference, where optimizing one task objective may inadvertently compromise the performance of another. A solution to mitigate this issue is to allocate task-specific parameters, free from interference, on top of shared features. However, manually designing such architectures is cumbersome, as practitioners need to balance between the overall performance across all tasks and the higher computational cost induced by the newly added parameters. In this work, we propose InterroGate, a novel MTL architecture designed to mitigate task interference while enhancing computational efficiency during inference. InterroGate features a learnable gating mechanism to automatically balance the shared and task-specific representations while preserving the performance of all tasks. The patterns of parameter sharing and specialization dynamically learned during training, become fixed at inference, resulting in a static, optimized MTL architecture. Through extensive empirical evaluations, we demonstrate SoTA results on three MTL benchmarks.

Citation

@inproceedings{Bejnordi_2024_BMVC,
author    = {Babak Ehteshami Bejnordi and Gaurav Kumar and Amelie Royer and Christos Louizos and Tijmen Blankevoort and Mohsen Ghafoorian},
title     = {InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learning},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0054.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection