GLPI: A Global Layered Prompt Integration approach for Explicit Visual Prompt


Yufei Gao (Zhengzhou University), Bin Fu (Zhengzhou University), Lei Shi (Zhengzhou University), Chengming Liu (Zhengzhou University), yucheng shi (Zhengzhou University)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)

Abstract

In the era of large models, prompt learning of pre-trained visual models has shown significant flexibility in various downstream tasks. Explicit Visual Prompt (EVP) serves as an outstanding unified framework applicable to foreground segmentation, achieving superior performance by fine-tuning with features from frozen patch embeddings and high-frequency components. However, in the process of training large models with EVP, the approach of freezing parameters and centrally updating the prompt embeddings may pose difficulties for long-distance backpropagation. These challenges can affect the generalization performance of the model, potentially limiting its ability to fully adapt and represent across different tasks and data. Besides, compared with other tuning methods, EVP requires more steps to perform competitively. To address these issues, this paper proposes Global Layered Prompt Integration(GLPI), which filters and combines the prompt information of adjacent encoder layers with adaptive threshold values to obtain an integration prompt closer to the downstream tasks. The optimal prompts with global information are constructed to enable the model to process images from a wider range of perspectives. Extensive experiments conducted on foreground segmentation tasks demonstrate that GLPI outperforms EVP and other advanced approaches.

Citation

@inproceedings{Gao_2024_BMVC,
author    = {Yufei Gao and Bin Fu and Lei Shi and Chengming Liu and yucheng shi},
title     = {GLPI: A Global Layered Prompt Integration approach for Explicit Visual Prompt},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0627.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection