Key-point Guided Deformable Image Manipulation Using Diffusion Model


Seok-Hwan Oh (Korea Advanced Institute of Science & Technology), Guil Jung (KAIST), Myeong-Gee Kim (Barreleye, inc.), Sang-yun Kim (KAIST), Young-Min Kim (KAIST), hyeonjik lee (KAIST), Hyuksool Kwon (Seoul National University), Hyeonmin Bae (Korea Advanced Institute of Science and Technology)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

In this paper, we introduce a Key-point-guided Diffusion probabilistic Model (KDM) that gains precise control over images by manipulating the object's key point. We propose a two-stage generative model incorporating an optical flow map as an intermediate output. By doing so, a dense pixel-wise understanding of the semantic relation between the image and sparse key point is configured, leading to more realistic image generation. Additionally, the integration of optical flow helps regulate the inter-frame variance of sequential images, demonstrating an authentic sequential image generation. The KDM is evaluated with diverse key-point conditioned image synthesis tasks, including facial image generation, human pose synthesis, and echocardiography video prediction, demonstrating the KDM is proving consistency enhanced and photo-realistic images compared with state-of-the-art models

Citation

@inproceedings{Oh_2024_BMVC,
author    = {Seok-Hwan Oh and Guil Jung and Myeong-Gee Kim and Sang-yun Kim and Young-Min Kim and hyeonjik lee and Hyuksool Kwon and Hyeonmin Bae},
title     = {Key-point Guided Deformable Image Manipulation Using Diffusion Model},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0114.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection