OmniGround: Closing the Omni-Modal Temporal Grounding Gap with Attention-Consistency Training

Omni-modal large models can understand video, audio, and text simultaneously, yet they consistently underperform on temporal grounding tasks. The OmniGround team discovered that the model's internal attention mechanism is actually more accurate than the timestamps it outputs. They therefore proposed the TACG method, which uses attention-consistency training to align this internal signal with the predictions, and built a 580K-sample dataset to train the OmniGround model, setting a new SOTA for omni-modal temporal grounding on LongVALE and UnAV-100.

The Temporal Generation Gap: Attention Is More Accurate Than Prediction

Rather than stopping at the final predictions, the research team dug into the model's internals to inspect temporal evidence. They found a counterintuitive phenomenon: the model's internal attention localizes the target moment significantly more accurately than the timestamps it actually generates. In other words, the model "knows" where the answer is internally, but "can't say it precisely."

This finding points to a key insight—the bottleneck in temporal grounding is not a lack of understanding capability, but a misalignment between internal representations and output generation. The authors named this phenomenon the "temporal generation gap." It explains why omni-modal models perform well on video understanding benchmarks yet repeatedly fail on tasks requiring precise temporal localization.

For audio-visual applications, temporal grounding is a foundational capability: retrieving specific sound events from video, localizing dialogue segments, and aligning multimodal event boundaries all depend on it. The existence of this gap means existing omni-modal models are still some distance from practical deployment.

Omni-modal temporal grounding refers to the task where, given a video or audio-visual clip and a natural language query, the model must output the start/end timestamps of the event described by the query on the timeline. The evaluation metric is typically mIoU (mean Intersection over Union), the average of the intersection-over-union between predicted and ground-truth time intervals. Unlike tasks such as video retrieval or video question answering, which only require semantic labels or text as output, temporal grounding demands fine-grained temporal perception from the model, outputting continuous time coordinates rather than discrete categories—thus imposing higher precision requirements on the model's internal temporal representations. Omni-modal models must simultaneously process three modalities—visual frame sequences, audio waveforms, and text queries—and align them on a unified timeline, which is precisely the core difficulty of this task.

TACG: Treating Internal Attention as a Temporal Anchor

To address this gap, the team proposed TACG (Temporal Attention-Consistency Grounding), a post-training method. Its core idea is to treat the model's internal attention as a reliable temporal anchor and, through attention-consistency training, make the attention distribution align with both the model predictions and the ground-truth annotations.

The elegance of TACG's design lies in introducing no additional temporal cues. It relies on no external timestamp encoder and modifies no model architecture; instead, during training it "calibrates" the model using consistency constraints among attention, predictions, and annotations. This amounts to teaching the model to more accurately translate the temporal information it already holds internally into output timestamps.

This post-training strategy has high engineering value: no training from scratch, no additional modality inputs, and it can be layered directly onto existing omni-modal models, lowering the barrier to deployment.

Attention-consistency training is a post-training strategy that uses the model's internal attention distribution as a supervisory signal. In omni-modal models, attention weights reflect the degree to which query tokens depend on video/audio frames, and peak positions often correspond to the temporal location of the event referred to by the query. TACG constructs constraints to make the attention distribution, the model's timestamp predictions, and the ground-truth annotations converge: attention peaks should fall within the ground-truth interval, and predicted timestamps should also align with attention peaks. This training requires no additional annotations and does not change the model structure at inference—essentially, it makes the model's existing internal knowledge explicit. Similar ideas have precedents in visual grounding and interpretability research, but applying them to omni-modal temporal grounding is a first.

580K-Sample Dataset and the OmniGround Model

Beyond the method, the team also reconstructed temporal question-answering from dense audio-visual annotations and organized general omni-modal instruction data to build a 580K-sample training dataset. This dataset's scale is considerable for the field of omni-modal temporal grounding, providing ample supervision for training.

Based on TACG and this dataset, the team trained OmniGround—a general omni-modal model with stronger temporal grounding capabilities. Experimental results show clear improvements: 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100, both reaching SOTA levels for omni-modal temporal grounding.

More importantly, OmniGround remains competitive on general audio-visual understanding and pure visual grounding tasks, without sacrificing other capabilities for the sake of temporal grounding. This shows that TACG's calibration is "lossless" and does not undermine the model's underlying multimodal understanding.

mIoU (mean Intersection over Union) is the core evaluation metric for temporal grounding tasks. For each sample, the IoU is computed as the intersection length of the predicted and ground-truth time intervals divided by their union length; mIoU is the average across all samples, ranging from 0 to 1 and typically expressed as a percentage. Higher mIoU indicates better overlap between predicted and ground-truth intervals. LongVALE and UnAV-100 are two commonly used audio-visual temporal grounding benchmarks: LongVALE emphasizes joint vision-audio-language understanding, while UnAV-100 focuses on audio event localization. OmniGround achieves 37.5 and 35.7 mIoU on these two benchmarks respectively—high levels for precise temporal localization tasks, since time interval prediction is highly sensitive to boundary errors.

Implications for Omni-Modal Model Training

OmniGround's work offers a noteworthy paradigm: rather than stacking complex decoding strategies at the output end, it is better to return to the model's internals and check whether intermediate representations already contain wasted information. The discovery of the temporal generation gap reminds researchers that evaluation of omni-modal models should not focus only on final metrics—the diagnostic value of internal attention is equally important.

From an application perspective, enhanced temporal grounding capabilities will directly benefit scenarios such as video retrieval, audio-visual event detection, and multimodal content understanding. OmniGround achieving SOTA without injecting additional temporal cues also implies that existing omni-modal models may commonly suffer from similar "attention-output misalignment" issues, and TACG may generalize to broader model families.

Of course, the details disclosed in the paper are currently limited: the generalizability of TACG across different architectures, the specific form of the attention-consistency constraints, and the data distribution of the 580K dataset all await further verification. But based on the existing results, this "look inward" training approach offers a clear and reproducible direction for omni-modal temporal grounding.

One-line Summary

OmniGround finds that model attention is more accurate than output timestamps, and calibration with TACG sets a new SOTA for omni-modal temporal grounding.

Paragraph Summary

Omni-modal large models can simultaneously understand video, audio, and text but perform poorly on temporal grounding tasks. The OmniGround team looked inside the model and found that the attention mechanism localizes target moments significantly more accurately than the timestamps the model actually generates, naming this misalignment the "temporal generation gap." To address it, the team proposed TACG, a post-training method that uses attention-consistency training to align the internal attention distribution with both predictions and ground-truth annotations, without introducing additional temporal cues or modifying the model architecture. The team also built a 580K-sample training dataset and trained the OmniGround model, achieving 37.5 mIoU on LongVALE and 35.7 mIoU on UnAV-100—a new SOTA for omni-modal temporal grounding—while remaining competitive on general audio-visual understanding and pure visual grounding.