Ego gesture intention grounding facilitated by video prediction
1 Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China
2 Shaanxi Dechuang Digital Industrial Intelligence Technology Co., Ltd., Xi’an, China
Abstract

Egocentric interaction grounding requires a system to identify one target when several alternatives are plausible. We propose a framework that combines video-conditioned interaction hypotheses, localization-guided direct preference optimization (DPO), confidence-weighted box fusion, and exponential moving average (EMA) smoothing. We construct preference pairs from training-set localization quality to adapt the hypothesis generator, then combine grounded hypotheses using detector confidence at inference. On RefEgo, our method improves final-output mean intersection over union (mIoU) from  0.177 to 0.232 compared with an adapted reasoning-and-grounding baseline. The detector-proposal oracle reference increases from 0.422 to 0.434, while the reference-to-output gap decreases from 0.245 to  0.202. The fractions of target-present evaluation records meeting intersection over union thresholds of 0.5  and 0.7 are 0.257 and 0.193, respectively. Candidate-generation analysis, a separate pre-/post-DPO  comparison, and component ablations complement the main comparison. These results support combining localization-guided preference learning with confidence-weighted spatial aggregation for more accurate final target localization in egocentric video.

Keywords

ego gesture intention grounding; video prediction; egocentric video; target selection; preference learning; box fusion; temporal consistency

Preview