A self-supervised cyclic neural-analytic approach for novel view synthesis and 3D reconstruction


Dragos Costea (University Politehnica of Bucharest), Alina Marcu (Institute of Mathematics of the Romanian Academy), Marius Leordeanu (Norwegian Research Center (NORCE))
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Generating novel views from recorded videos is crucial for enabling autonomous UAV navigation. Recent advancements in neural rendering have facilitated the rapid development of methods capable of rendering new trajectories. However, these methods often fail to generalize well to regions far from the training data without an optimized flight path, leading to suboptimal reconstructions. We propose a self-supervised cyclic neural-analytic pipeline that combines high-quality neural rendering outputs with precise geometric insights from analytical methods. Our solution enhances both RGB and mesh reconstructions for novel view synthesis, particularly in undersampled areas and regions entirely distinct from the training dataset. We leverage an effective transformer-based architecture for image reconstruction to refine and adapt the synthesis process, enabling effective handling of novel, unseen poses without relying on extensive labeled datasets. Our findings demonstrate substantial improvements in rendering views of novel and also 3D reconstruction, which to the best of our knowledge is a first, setting a new standard for autonomous navigation in complex outdoor environments. To foster further research and application, we will make our code publicly available.

Citation

@inproceedings{Costea_2024_BMVC,
author    = {Dragos Costea and Alina Marcu and Marius Leordeanu},
title     = {A self-supervised cyclic neural-analytic approach for novel view synthesis and 3D reconstruction},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0524.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection