Highlights
We remove sequential dependencies by predicting the full spatio-temporal tube in just two decoding rounds.
We improve both temporal and spatial grounding by preventing error propagation across the predicted tube.
We study four decoding strategies for generative STVG and introduce Parallel Tube Decoding, reducing tube generation from 4 + 7T rounds under Quantized Token Decoding and 1 + T under Sequential Block Decoding to just two rounds, independent of the number of boxes.
We enable PTD through Decoupled Block Attention, which removes cross-box dependencies while preserving shared multimodal evidence, keeping each box grounded in the visual content at its own temporal position.
Policy optimization with complementary temporal and spatial rewards directly improves event boundaries and bounding-box geometry.
PTD cuts Tube Completion Latency by 79× and raises spatial decoding throughput by 92× while improving localization quality. With a compact 4B backbone, it performs strongly on STVG and generalizes zero-shot to temporal grounding, evidence-grounded VideoQA, and referring video object tracking.
Observe how quickly each method predicts the boxes overlaid on the videos to compare the decoding speeds of Quantized Token Decoding, Sequential Token Decoding, and our Parallel Tube Decoding (PTD).
Comparison of decoding strategies on VidSTG. We report temporal and video IoU for declarative and interrogative queries, together with Tube Completion Latency (TCL) and Boxes Per Second (BPS). Parallel Tube Decoding achieves the strongest grounding performance, lowest latency, and highest throughput.
Results on VidSTG. Comparison with backbone baselines and prior multimodal large language models for declarative and interrogative spatio-temporal grounding. Our compact Qwen3-VL-4B model with GRPO and PTD delivers the strongest overall results.
Results on HC-STVG. Comparison with backbone baselines and prior methods on HC-STVG v1 and v2. PTD achieves strong temporal localization and the best video IoU across both benchmark versions.
We compare PTD with competitive STVG methods on challenging video-grounding examples. Like many existing approaches, STVG-R1 and DEViL use the MLLM for temporal localization but construct the spatial tube using external detection or tracking modules. In contrast, PTD directly generates both the temporal interval and complete spatial tube, producing more accurate temporal boundaries and spatial localization. The three videos highlight fine-grained target identification among visually similar distractors, large changes in object scale as the target approaches the camera, and event-specific temporal localization rather than simply predicting the full period in which the target is visible.
Zero-shot temporal grounding. Results on Charades-STA and ActivityNet Captions without task-specific training. Our model improves over the strongest prior zero-shot methods, including under stricter temporal-overlap thresholds.
Generalization to grounded VideoQA and video object tracking. Left: zero-shot evidence-aware question answering on ReXTime. Right: referring video object tracking on Ref-DAVIS, Ref-YT-VOS, and ReasonVOS using SAM2 or SAM3.
Ablation of localization-aware rewards. Temporal and spatial rewards provide complementary improvements for declarative and interrogative grounding. Combining both produces the strongest overall spatio-temporal localization performance.
Qualitative results illustrating the effect of the spatial reward. Green, red, and blue denote the ground-truth, SFT, and GRPO with spatial reward predictions, respectively. In the first example, the spatial reward maintains tight localization of the rider under rapid motion. In the second, it preserves target consistency despite substantial changes in object scale and interference from nearby entities. These examples show how the spatial reward improves both bounding-box precision and consistent target localization across time.
Qualitative results illustrating the effect of the temporal reward. Green, red, and blue denote the ground-truth, SFT, and GRPO with temporal reward predictions, respectively. In the first example, SFT localizes only the most salient portion of the event despite accurate spatial boxes, whereas the temporal reward recovers the complete interaction. In the second, the temporal reward prevents the prediction from extending beyond the queried event while the target remains visible. These examples show how the temporal reward improves boundary prediction, ensuring that the generated tube aligns with the complete temporal extent of the queried event.
Analysis of decoding efficiency and trajectory-level dependency. (a) Tube completion latency as the number of grounded frames increases. PTD maintains nearly constant latency, while token-based and block decoding scale with tube length. (b) Attention distribution across tube decoding for Sequential Block Decoding (dotted) and PTD (solid). Sequential decoding progressively shifts attention from the video toward prior text. (c) History-correction analysis for Sequential Block Decoding. Replacing an erroneous box Bi with its ground-truth box improves subsequent predictions, with the effect gradually decreasing as the decoding distance j − i increases.
Citation
If you find Parallel Tube Decoding useful for your research, please consider citing our work.
@misc{rasheed2026locateanythingvideos,
title = {Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding},
author = {Hanoona Rasheed and Haania Siddiqui and Ming-Hsuan Yang and Fahad Shahbaz Khan and Salman Khan},
year = {2026}
}