MMPrune4U: Regularizing Multimodal Feature Distortion in Weight Pruning for Deep Neural Network Compression


Sudip Das (Valeo), Kaixin Xu (I2R, A*STAR), Nushrat Hussain (Indian Statistical Institute), Ziyuan Zhao (I2R, A*STAR), Arindam Das (Valeo), Weisi Lin (Nanyang Technological University), Ujjwal Bhattacharya (Indian Statistical Institute, Dhirubhai Ambani Institute Of Information and Communication Technology)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)

Abstract

Despite the remarkable success of multimodal models in automotive applications, their practical benefits are often accompanied by a large number of parameters, including redundant and excessive weights. This poses hurdles to their deployment on embedded devices due to the substantial computational costs compared to unimodal models. Model sparsification is among the common solutions to reduce the resources required for computation and increase throughput of the system. Although many recent studies in model sparsification and pruning achieve remarkable performance for unimodal models, they overlook capturing the layer-wise sensitivity towards accuracy and behaviors for distinct modalities in response to the pruning, leading to information loss in the downstream tasks of the pruned model. We introduce MMPrune4U, a layer-adaptive weight pruning method explicitly designed to support multimodal 3D scene understanding that incorporates a regularizer based on log-Sobolev inequality. This approach uncovers a crucial property related to the distortion of features resulting from pruning weights across multiple layers while keeping a predefined pruning ratio. As per the changes in the output distribution of the each layer during pruning compared to unpruned model, we regularize the distortion through the functional Fisher information. We formulate our layer-adaptive pruning by considering the layerwise impact to the downstream tasks and optimize the objective function through combinatorial optimization challenge, which we effectively address using dynamic programming techniques. The proposed MMPrune4U method demonstrates superior performance in comparison to the existing state-of-the-art methods, as shown by experimental results on both nuScenes and SemanticKITTI datasets.

Citation

@inproceedings{Das_2024_BMVC,
author    = {Sudip Das and Kaixin Xu and Nushrat Hussain and Ziyuan Zhao and Arindam Das and Weisi Lin and Ujjwal Bhattacharya},
title     = {MMPrune4U: Regularizing Multimodal Feature Distortion in Weight Pruning for Deep Neural Network Compression},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0104.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection