Efficient Data Source Relevance Quantification for Multi-Source Neural Networks


Jakob Gawlikowski (Technical University of Munich (TUM)), Nina Maria Gottschling (German Aerospace Center (DLR))
The 35th British Machine Vision Conference
PDFPosterSupplementary

Abstract

In deep learning-based data fusion, accurately quantifying the relevance of individual data sources offers enhanced analytical capabilities and insights into source-wise information. However, current methodologies are computationally expensive, requiring multiple forward passes or separate backpropagation for each output. In this paper, we present Relevance Forward Propagation (RFP). This approach efficiently computes data source relevance values for all network outputs on the fly within a single forward pass by utilizing the aggregation of source-wise relevance values. We mathematically prove and experimentally validate the equivalence of the resulting source-wise relevance values to those computed with the well-established backpropagation-based Layer-Wise Relevance Propagation. We validate the effectiveness and efficiency of RFP against several existing approaches. Using a data fusion MNIST, we explore factors affecting data source relevance, such as noise and disparities in data source complexities. We extend these insights to practical domains by addressing the fusion of satellite data from Synthetic Aperture Radar (SAR) and optical satellites. Our method demonstrates adaptability within complex settings in scenarios where clouds affect optical data. Our proposed approach efficiently quantifies the relevance values of individual data sources within the prediction step of deep learning-based fusion, extending to real-world complexities and challenges encountered in satellite data fusion.

Citation

@inproceedings{Gawlikowski_2024_BMVC,
author    = {Jakob Gawlikowski and Nina Maria Gottschling},
title     = {Efficient Data Source Relevance Quantification for Multi-Source Neural Networks},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0885.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection