Deep Unfolding Network with Spatial-spectral Perception Enhanced for Pan-sharpening


Mengjiao Zhao (Zhejiang University), Mengting Ma (Zhejiang University), Xiangdong Li (Zhejiang University), Ao Gao (Zhejiang University), Siyang Song (University of Exeter), Wei Zhang (Zhejiang University)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Pan-sharpening aims to perform super-resolution processing on low-resolution multispectral (LR-MS) images guided by high-resolution panchromatic (PAN) images. Existing pan-sharpening methods have two main issues. Firstly, deep learning-based methods are mostly designed based on black-box principles, lacking sufficient interpretability. Secondly, model-based methods enhance interpretability but do not fully consider domain-specific prior knowledge, namely complex spatial and spectral relationships, which limits their performance. To address these challenges, we propose a novel deep unfolding network with spatial-spectral perception enhanced for pan-sharpening, namely SSPEDUN. Specifically, we model the pan-sharpening problem as the minimization of a variational model with spatial reconstruction priors and spectral modulation priors. The spatial reconstruction prior reconstructs high-quality spatial information based on observed image spatial relationships, while the spectral modulation prior accurately modulates the spectral relationships between images. Then, we design an efficient iterative proximal gradient descent algorithm to alternately solve the data subproblem and the prior subproblem of the model, and then unfold this algorithm into a deep network. In the deep unfolding network, we introduce a data projection module to address data mapping during the optimization process and carefully design a Perception Enhancement Module (PEM) as the prior module to precisely model spatial and spectral relationships. Extensive experiments on three satellite datasets demonstrate the superiority of our method. The source code is available in our supplementary material.

Citation

@inproceedings{Zhao_2024_BMVC,
author    = {Mengjiao Zhao and Mengting Ma and Xiangdong Li and Ao Gao and Siyang Song and Wei Zhang},
title     = {Deep Unfolding Network with Spatial-spectral Perception Enhanced for Pan-sharpening},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0250.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection