Layer-wise Learning of CNNs by Self-tuning Learning Rate and Early Stopping at Each Layer


Melika Sadeghi Tabrizi (University of Tehran, University of Tehran), Ali Karimi (Kharazmi University), Ahmad Kalhor (University of Tehran), Babak N Araabi (University of Tehran, University of Tehran), Mona Ahmadian (University of Surrey)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

In this paper, we propose a novel layer-wise learning strategy for Convolutional Neural Networks (CNNs), through which the learning rate and early stop are independently tuned at each layer. This strategy maximizes each layer's Separation Index (SI) to improve overall model performance. For this reason, our layer-wise learning strategy is different in three main aspects compared to state-of-the-art learning strategies. Firstly, the learning rate of each layer is adjusted based on the initial value of SI for that layer. Secondly, if the SI value of a layer does not increase over several consecutive epochs, it indicates that the layer cannot extract more significant features, so the training process for that layer is stopped, and training for the next layer starts. Thirdly, we demonstrate the superiority of triplet loss as a ranking loss in layer-wise learning, outperforming other state-of-the-art loss functions. Experiments on CNNs architectures such as LeNet, AlexNet, VGG, ResNet, EfficientNet, and DenseNet, and datasets such as CIFAR-100, CIFAR-10, and STL-10 demonstrated that it performs better in terms of accuracy and training time compared to state-of-the-art learning strategies.

Citation

@inproceedings{Tabrizi_2024_BMVC,
author    = {Melika Sadeghi Tabrizi and Ali Karimi and Ahmad Kalhor and Babak N Araabi and Mona Ahmadian},
title     = {Layer-wise Learning of CNNs by Self-tuning Learning Rate and Early Stopping at Each Layer},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0947.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection