Hierarchical Prompt Learning for Scene Graph Generation


Xuhan Zhu (University of Chinese Academy of Sciences), Yifei Xing (Chinese Academy of Sciences), Ruiping Wang (Institute of Computing Technology, Chinese Academy of Sciences), Yaowei Wang (Harbin Institute of Technology, Shenzhen), Xiangyuan Lan (Peng Cheng Laboratory)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Scene Graph Generation (SGG) delivers structured knowledge representing complex scenes, which is applied in many computer vision fields. However, existing SGG models falter in predicting novel and informative predicates, undermining their applicability for higher-level visual tasks. Drawing inspiration from the success of prompt learning in zero-shot knowledge transfer, we propose a prompt-learning-based method to address novel and informative predicate learning challenges in SGG. Specifically, we perform a comprehensive analysis of three basic prompts in SGG, considering their computational efficiency and learning ability. Subsequently, we build upon these basic prompts to construct a Hierarchical Prompt (HP) learning method to enhance informative predicate learning. HP utilizes the composition of basic prompts constrained to progressively narrowed class groups and encourages the corresponding prompts to focus on the learning of increasingly informative predicates. HP is a plug-and-play solution applicable to various models. Extensive evaluations on SGG benchmarks demonstrate the excellent ability of HP to improve the performance of informative predicates across different baselines. We also introduce a novel predicate generalization task with a new benchmark. Experiments on it demonstrate the superiority of HP in base-to-novel predicate generalization.

Citation

@inproceedings{Zhu_2024_BMVC,
author    = {Xuhan Zhu and Yifei Xing and Ruiping Wang and Yaowei Wang and Xiangyuan Lan},
title     = {Hierarchical Prompt Learning for Scene Graph Generation},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0183.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection