AttEntropy: On the Generalization Ability of Supervised Semantic Segmentation Transformers to New Objects in New Domains


Krzysztof Baron-Lis (Waabi), Matthias Rottmann (University of Wuppertal), Annika Mütze (Bergische Universität Wuppertal), Sina Honari (Samsung), Pascal Fua (EPFL - EPF Lausanne), Mathieu Salzmann (Swiss Data Science Center)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

In addition to impressive performance, vision transformers have demonstrated remarkable abilities to encode information they were not trained to extract. For example, this information can be used to perform segmentation or single-view depth estimation even though the networks were only trained for image recognition. We show that a similar phenomenon occurs when explicitly training transformers for semantic segmentation in a supervised manner for a set of categories: Once trained, they provide valuable information even about categories absent from the training set and this information can be used to segment objects from these never-seen-before classes in domains as varied as road obstacles, aircraft parked at a terminal, lunar rocks, and maritime hazards.

Citation

@inproceedings{Baron-Lis_2024_BMVC,
author    = {Krzysztof Baron-Lis and Matthias Rottmann and Annika Mütze and Sina Honari and Pascal Fua and Mathieu Salzmann},
title     = {AttEntropy: On the Generalization Ability of Supervised Semantic Segmentation Transformers to New Objects in New Domains},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0215.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection