Are Sparse Neural Networks Better Hard Sample Learners?


Qiao Xiao (Eindhoven University of Technology), Boqian Wu (University of Twente), Lu Yin (University of Surrey), Christopher Neil Gadzinski (University of Luxemburg), Tianjin Huang (University of Exeter), Mykola Pechenizkiy (Eindhoven University of Technology), Decebal Constantin Mocanu (University of Luxemburg)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

While deep learning has demonstrated impressive progress, it remains a daunting challenge to learn from hard samples as these samples are usually noisy and intricate. These hard samples play a crucial role in the optimal performance of deep neural networks. Most research on Sparse Neural Networks (SNNs) has focused on standard training data, leaving gaps in understanding their effectiveness on complex and challenging data. This paper's extensive investigation across scenarios reveals that most SNNs trained on challenging samples can often match or surpass dense models in accuracy at certain sparsity levels, especially with limited data. We observe that layer-wise density ratios tend to play an important role in SNN performance, particularly for methods that train from scratch without pre-trained initialization. These insights enhance our understanding of SNNs' behavior and potential for efficient learning approaches in data-centric AI. Our code is publicly available at: https://github.com/QiaoXiao7282/hard_sample_learners.

Citation

@inproceedings{Xiao_2024_BMVC,
author    = {Qiao Xiao and Boqian Wu and Lu Yin and Christopher Neil Gadzinski and Tianjin Huang and Mykola Pechenizkiy and Decebal Constantin Mocanu},
title     = {Are Sparse Neural Networks Better Hard Sample Learners?},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0290.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection