MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation


Kim Yu-Ji (Pohang University of Science and Technology), Hyunwoo Ha (Pohang University of Science and Technology), Kim Youwang (Pohang University of Science and Technology), Jaeheung Surh (Bucketplace), Hyowon Ha (Bucketplace), Tae-Hyun Oh (POSTECH)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

Reconstructing 3D from a single view image is a long-standing challenge. One of the popular approaches to tackle this problem is learning-based methods, but dealing with the test cases unfamiliar with training data (Out-of-distribution; OoD) introduces an additional challenge. To adapt for unseen samples in test time, we propose MeTTA, a test-time adaptation (TTA) exploiting generative prior. We design joint optimization of 3D geometry, appearance, and pose to handle OoD cases with only a single view image. However, the alignment between the reference image and the 3D shape via the estimated viewpoint could be erroneous, which leads to ambiguity. To address this ambiguity, we carefully design learnable virtual cameras and their self-calibration. In our experiments, we demonstrate that MeTTA effectively deals with OoD scenarios at failure cases of existing learning-based 3D reconstruction models and enables obtaining a realistic appearance with physically based rendering (PBR) textures.

Citation

@inproceedings{Yu-Ji_2024_BMVC,
author    = {Kim Yu-Ji and Hyunwoo Ha and Kim Youwang and Jaeheung Surh and Hyowon Ha and Tae-Hyun Oh},
title     = {MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0018.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection