Anchor-Based Masked Generative Distillation for Pixel-Level Prediction Tasks


Xie Yu (Beijing University of Aeronautics and Astronautics), Wentao Zhang (Beijing University of Aeronautics and Astronautics)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Knowledge distillation (KD) has made significant achievements in pixel-level prediction tasks. In recent years, owing to the mask generative distillation (MGD) paradigm, there has been a tendency towards unified distillation architecture, which achieved remarkable performance across multiple tasks. However, due to the naive mask designing and feature generating, MGD does not demonstrate the potential of Mask Image Modeling (MIM) on pixel-level prediction tasks. To further exploit the strength of MIM in knowledge distillation, we design Anchor-Based Masked Generative Distillation (AMGD) by both improving the masking stage and generating stage. Specifically, we involve the concepts of "shape and size" into mask designing and implement an advanced generator according to the attributes of the masks. Comparing to MGD, AMGD can fully take advantage of objects’ semantic information and thus benefits the distillation on pixel-level prediction tasks. Extensive experiments on three mainstream tasks with various benchmarks demonstrate the effectiveness as well as the generalization ability of AMGD.

Citation

@inproceedings{Yu_2024_BMVC,
author    = {Xie Yu and Wentao Zhang},
title     = {Anchor-Based Masked Generative Distillation for Pixel-Level Prediction Tasks},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0365.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection