Linear Calibration Approach to Knowledge-free Group Robust Classification


Ryota Ishizaki (Tokyo University of Science), Shunya Yamagami (Tokyo University of Science), Yuta Goto (Tokyo University of Science), Go Irie (Tokyo University of Science)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Large-scale pre-trained vision-language models such as CLIP have shown remarkable performance on various downstream tasks. However, such a model often learns not only the information that is truly useful for classification, but also group attributes that are spuriously correlated with classes, leading to misclassification of an image into a group with the same group attributes but with the wrong class. The goal of this paper is to develop a method for learning a classifier that is robust to the group attributes. Unlike existing methods, our method is (i) knowledge-free: does not use any information of group attributes for training, (ii) linear: a lightweight method that trains only a single linear projection, and (iii) calibration-based: does not change the original classifier at all. The negative effects of the group attributes can be canceled by projecting the classification space to the orthogonal complement of the subspace spanned by the group attributes. To achieve this, we propose Spurious Subspace Mining (SSM) to discover the subspace from a random set of text embeddings without any supervision. Experimental results on two standard benchmark datasets, Waterbirds and CelebA, show that the proposed method outperforms various existing methods and improves zero-shot baseline by 35.2% in worst-group accuracy.

Citation

@inproceedings{Ishizaki_2024_BMVC,
author    = {Ryota Ishizaki and Shunya Yamagami and Yuta Goto and Go Irie},
title     = {Linear Calibration Approach to Knowledge-free Group Robust Classification},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0038.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection