Neural Collapse Inspired Contrastive Continual Learning


Antoine Montmaur (ENSEA), Nicolas Larue (ENSEA), Ngoc-Son Vu (ENSEA)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

In recent advances, contrastive learning has enhanced representation quality by emphasizing transferable features across tasks, while the newly identified phenomenon of neural collapse (NC) optimizes separation capacity. Recognizing that catastrophic forgetting, the primary challenge in continual learning, results from overlapping representations between tasks, and inspired by the optimal classification ability of NC, we propose innovative strategies to minimize representational overlap. We first introduce neural continual collapse (NCC), a loss function that guides representations towards neural collapse by employing predefined hard prototypes to attract samples within a class. Additionally, we propose simplex structure distillation (SSD), a distillation technique that uses hard prototypes to strengthen knowledge consolidation. SSD improves learning stability and decreases reliance on replay buffers by gradually aligning structural distributions as tasks progress. These methods excel in challenging replay-free setups and surpass state-of-the-art (SOTA) replay-based methods.

Citation

@inproceedings{Montmaur_2024_BMVC,
author    = {Antoine Montmaur and Nicolas Larue and Ngoc-Son Vu},
title     = {Neural Collapse Inspired Contrastive Continual Learning},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0579.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection