Prompting Diffusion Representations for Cross-Domain Semantic Segmentation


Rui Gong (Amazon), Martin Danelljan (ETH Zurich), Han Sun (EPFL - EPF Lausanne), Julio Delgado Mangas (Meta, Reality labs), Nikolay Marin (Amazon), Luc Van Gool (INSAIT - Sofia Un.)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

While originally designed for image generation, diffusion models have recently shown to provide excellent pretrained feature representations for semantic segmentation. Intrigued by this result, we set out to explore how well diffusion-pretrained representations generalize to new domains, a crucial ability for any representation. We find that diffusion-pretraining achieves extraordinary domain generalization results for semantic segmentation, outperforming both supervised and self-supervised backbone networks. Motivated by this, we investigate how to utilize the model's unique ability of taking an input prompt, in order to further enhance its cross-domain performance. We introduce a scene prompt and a prompt randomization strategy to help further disentangle the domain-invariant information when training the segmentation head. Moreover, we propose a simple but highly effective approach for test-time domain adaptation, based on learning a scene prompt on the target domain in an unsupervised manner. Extensive experiments conducted on four synthetic-to-real and clear-to-adverse weather benchmarks demonstrate the effectiveness of our approaches. Without resorting to any complex techniques, such as image translation, augmentation, or rare-class sampling, we set a new state-of-the-art on all benchmarks.

Citation

@inproceedings{Gong_2024_BMVC,
author    = {Rui Gong and Martin Danelljan and Han Sun and Julio Delgado Mangas and Nikolay Marin and Luc Van Gool},
title     = {Prompting Diffusion Representations for Cross-Domain Semantic Segmentation},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0103.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection