SAPO: Breaking the Video Temporal Grounding Bottleneck with Self-Supervised Attention Policy Optimization

Video vision-language models (VLMs) excel at semantic understanding, yet they often "know but can't say precisely" when it comes to precise temporal grounding. A recent study from arXiv reveals the reason behind this contradiction and proposes a solution that requires no human annotation—Self-Supervised Attention Policy Optimization (SAPO).

The Temporal Generation Gap: The Divide Between What VLMs "Know" and What They "Say"

The research team analyzed four VLMs across two grounding datasets and found a consistent phenomenon: in about 66% of valid samples, the time interval indicated by the model's internal attention mechanism was more accurate than the timestamp the model ultimately predicted. They named this phenomenon the "temporal generation gap."

The significance of this finding is that it shows VLMs do not lack grounding ability; rather, they lose precision when converting internal representations into concrete timestamps. In other words, the model "knows it in its head" but "can't say it precisely out loud." This insight points to a new direction for improving temporal grounding: instead of laboring to enhance the model's perception, the goal is to help the model express the information it already knows more accurately.

mIoU (mean Intersection over Union) is the most commonly used evaluation metric in temporal grounding tasks. For each sample, the intersection of the predicted time interval and the ground-truth interval is divided by their union, yielding an IoU value between 0 and 1; averaging over all samples gives mIoU. An IoU of 1 means the prediction perfectly overlaps the ground truth, while 0 means no overlap at all. Because it penalizes both interval position shifts and length deviations, it reflects grounding quality better than simple timestamp error. The "relative mIoU gain" reported in this article refers to the percentage improvement of the improved mIoU relative to the baseline—for example, a 33.7% gain means a substantial jump in grounding precision on Qwen2.5-VL-3B.

SAPO's Core Idea: Using Attention as a Self-Supervised Signal

Since precise temporal annotations are costly, the research team turned to unlabeled video-query pairs. SAPO's key innovation is to use the model's internal temporal attention as a self-supervised signal to drive Group Relative Policy Optimization (GRPO).

Specifically, SAPO uses attention intervals as "pseudo-labels" to evaluate prediction quality and designs a selection criterion that requires no ground-truth annotations to pick out high-value training samples. This approach bypasses dependence on human annotation while directly optimizing against the "temporal generation gap."

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm proposed by the DeepSeek^Ad team, originally used to improve the reasoning capabilities of large language models. Unlike traditional PPO (Proximal Policy Optimization), which requires training an additional value network (critic) to estimate a baseline, GRPO samples a group of multiple outputs for the same input and directly uses the mean reward within the group as the baseline to compute the advantage function. This eliminates the value network and significantly reduces memory and computational overhead. In this study, GRPO is used to fine-tune the VLM's timestamp generation policy: the model generates multiple candidate time intervals for the same video-query pair, SAPO scores them with a reward constructed from attention intervals, and after relative comparison within the group the policy is updated, making the model tend to output results closer to the intervals indicated by its internal attention.

Temporal attention in vision-language models (VLMs) refers to the weights assigned to each frame by the cross-modal attention layers as the model processes a sequence of video frames. When a user asks a query such as "when did a certain event happen," frames semantically relevant to the query typically receive higher attention weights, and these weights form a distribution curve along the time axis. By smoothing this curve and taking contiguous intervals above a threshold, one obtains the event time span that the model "internally believes." SAPO treats this attention-derived interval as a pseudo-label: it reflects the model's internal judgment before generating the final text output, thereby bypassing human annotation and naturally aligning with the model's own representations.

Experimental Results: Stable Gains Across Models and Datasets

SAPO delivers stable temporal grounding improvements across multiple datasets and models, covering initial grounding abilities ranging from weak to strong. The relative mIoU gain reaches 33.7% on Qwen2.5-VL-3B, and even on the stronger Time-R1-7B there is a 5.4% improvement.

Interestingly, this improvement transfers across datasets, and while grounding precision improves, the model's performance on general video understanding tasks is not degraded. This means SAPO provides a "lossless" capability enhancement.

Implications and Outlook

The value of SAPO lies not only in its specific technical approach but also in the reusable idea it reveals: the model's internal representations may already contain the information needed to complete the task, and the key is how to design self-supervised signals to release it. For tasks with expensive annotations such as video temporal grounding, this "digging inward" strategy is especially attractive.

In the future, similar ideas may be extended to other multimodal tasks that require precise output, such as spatial grounding or fine-grained action recognition. SAPO reminds us that sometimes improving model performance does not require more annotated data, but rather smarter use of the knowledge the model already possesses.

One-line Summary

SAPO uses VLMs' internal attention as a self-supervised signal to narrow the temporal generation gap without annotations.

Paragraph Summary

This study reveals the "temporal generation gap" of video vision-language models (VLMs) in temporal grounding tasks: in about 66% of valid samples, the time interval indicated by the model's internal attention is more accurate than the timestamp it ultimately outputs, showing that the model does not lack grounding ability but loses precision when converting internal representations into concrete timestamps. To address this, the authors propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal temporal attention as pseudo-labels and reward signals to drive Group Relative Policy Optimization (GRPO), thereby directly optimizing temporal generation on unlabeled video-query pairs. Experiments show that SAPO stably improves grounding precision across multiple models and datasets, with a relative mIoU gain of 33.7% on Qwen2.5-VL-3B and 5.4% on Time-R1-7B, and the improvement transfers across datasets without harming general video understanding ability.