TransHuPR: Cross-View Fusion Transformer for Human Pose Estimation Using mmWave Radar


Niraj Prakash Kini (National Yang Ming Chiao Tung University), Ruey-Horng Shiue (National Yang Ming Chiao Tung University), ryan chandra (National Yangmin Chiaotung University), Wen-Hsiao Peng (National Yang Ming Chiao Tung University), Ching-Wen Ma (National Yang Ming Chiao Tung University), Jenq-Neng Hwang (University of Washington)
The 35th British Machine Vision Conference
PDFPosterVideo (Right click to download)Supplementary

Abstract

We present a novel Cross-View Fusion Transformer for Human Pose Estimation task based on mmWave Radar (TransHuPR). It is an mmWave Radar-based 2D Human Pose Estimation (HPE). Our work incorporates a 2D front projection view of the 3D pointcloud representation of the radar data as an input modality. The fusion transformer effectively fuses features derived from 2D front projection views of 2 independent radars and delivers high-quality predictions of human pose keypoints. We also introduce a new dataset consisting of fast actions with high frame rates as continuous radar sequences. Unlike other publicly available datasets, our dataset stands out because of its size, which ensures good generalization. We also incorporate single-action and mixed-action sequences, making the dataset more challenging. We use a non-expensive multi-radar system, which can be easily replicated. Our proposed method demonstrates significant improvements over existing methods in terms of both average precision scores and qualitative analysis. The dataset and code will be made publicly available.

Citation

@inproceedings{Kini_2024_BMVC,
author    = {Niraj Prakash Kini and Ruey-Horng Shiue and ryan chandra and Wen-Hsiao Peng and Ching-Wen Ma and Jenq-Neng Hwang},
title     = {TransHuPR: Cross-View Fusion Transformer for Human Pose Estimation Using mmWave Radar},
booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
publisher = {BMVA},
year      = {2024},
url       = {https://papers.bmvc2024.org/0686.pdf}
}


Copyright © 2024 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection