Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization


Roisin Luo (University of Galway), Alexandru Drimbarean (FotoNation), James McDermott (University of Galway), Colm O'Riordan (University of Galway)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

This paper explores a novel paradigm in low-bit (*ie.* 4-bits or lower) quantization, differing from existing state-of-the-art methods, by framing optimal quantization as an architecture search problem within convolutional neural networks (ConvNets). Our framework, dubbed **CoRa** (Optimal Quantization Residual **Co**nvolutional Operator Low-**Ra**nk Adaptation), is motivated by two key aspects. Firstly, quantization residual knowledge, *ie.* the lost information between floating-point weights and quantized weights, has long been neglected by the research community. Reclaiming the critical residual knowledge, with an infinitesimal extra parameter cost, can reverse performance degradation without training. Secondly, state-of-the-art quantization frameworks search for optimal quantized weights to address the performance degradation. Yet, the vast search spaces in weight optimization pose a challenge for the efficient optimization in large models. For example, state-of-the-art BRECQ necessitates $2 \times 10^4$ iterations to quantize models. Fundamentally differing from existing methods, **CoRa** searches for the optimal architectures of low-rank adapters, reclaiming critical quantization residual knowledge, within the search spaces smaller compared to the weight spaces, by many orders of magnitude. The low-rank adapters approximate the quantization residual weights, discarded in previous methods. We evaluate our approach over multiple pre-trained ConvNets on ImageNet. **CoRa** achieves comparable performance against both state-of-the-art quantization-aware training and post-training quantization baselines, in $4$-bit and $3$-bit quantization, by using less than $250$ iterations on a small calibration set with $1600$ images. Thus, **CoRa** establishes a new state-of-the-art in terms of the optimization efficiency in low-bit quantization.

Citation

@inproceedings{Luo_2024_BMVC,
author    = {Roisin Luo and Alexandru Drimbarean and James McDermott and Colm O'Riordan},
title     = {Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0184.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection