HLR-LLM: Teaching Audio-Visual Large Models to Hear Where Sound Comes From

When AI Can "Hear" but Doesn't Know Where the Sound Is Coming From

Audio-visual large language models (AV-LLMs) have made remarkable progress recently, capable of understanding both visual scenes and audio content simultaneously. However, most of these models treat audio as monaural semantic information—they can identify "what sound is this," but cannot answer "which direction is the sound coming from" or "which object in the scene is it related to."

This limitation is not trivial. In real-world scenarios, spatial information is a key dimension for understanding the environment: a car approaching from the left, a shout from behind, an instrument playing in the lower right corner of the frame—all of these require the model to have spatial awareness for accurate reasoning.

A recent research effort has proposed a systematic solution to this problem, building both an evaluation benchmark and an improved model.

Monaural audio refers to sound signals recorded and played back with a single microphone or a single channel. Traditional tasks such as speech recognition and audio event classification are almost all based on monaural audio, because it is sufficient to carry semantic information like speech content, timbre, and rhythm. But monaural audio superimposes sounds from all directions into a single waveform, physically discarding spatial localization cues such as interaural time difference (ITD), interaural level difference (ILD), and head-related transfer function (HRTF). As a result, the sound source direction cannot be inferred from a monaural signal alone. This is precisely the root cause of why current audio-visual large models can "hear but cannot locate": both the training corpus and the model input assume monaural audio, the spatial dimension is structurally erased, and improvements in semantic understanding do not automatically restore it.

HLR-AVSceneQA: A Hierarchical Spatial Audio-Visual Evaluation Benchmark

The research team first points out the shortcomings of existing evaluations: prior spatial audio-visual research tends to focus on only one specific capability, such as spatial correspondence, direction judgment, or distance reasoning, lacking a comprehensive examination of the model's spatial understanding abilities.

To address this, they proposed HLR-AVSceneQA, a comprehensive benchmark designed around a "Hear-Localize-Reason" hierarchical structure. This three-layer structure corresponds to three progressively advanced capabilities:

• Hear: Can the model correctly identify the content of the sound it hears?
• Localize: Can the model locate the sound source from both egocentric and allocentric perspectives?
• Reason: Can the model use the localized sound source to reason about its relationship with other elements in the visual scene?

The value of this hierarchical design is that it not only provides an overall score but also diagnoses exactly which layer the model struggles with—whether it cannot recognize the sound, can recognize it but cannot locate it, or can locate it but cannot perform relational reasoning.

HLR-LLM: A Dual-Pathway Audio Architecture

To address the shortcomings of existing models, the research team proposed HLR-LLM. Its core idea is to add a binaural spatial audio branch on top of the existing monaural semantic audio pathway.

The key to this design is "no sacrifice": the semantic audio pathway is responsible for answering "what sound is this," while the spatial audio branch is responsible for answering "where is the sound." The two pathways work together, enabling the model to possess both semantic understanding and spatial awareness.

For training, the team adopted a curriculum learning strategy and introduced chain-of-thought supervision in the final stage, specifically training for multi-step spatial reasoning tasks. This means the model must not only output answers but also learn a step-by-step thinking process of "first localize, then reason."

Binaural spatial audio refers to sound recorded with two microphones simulating the spacing between human ears, where the two channels preserve cues such as interaural time difference (ITD), interaural level difference (ILD), and head-related transfer function (HRTF). Humans rely on these subtle differences to judge sound source direction, making binaural signals a key input for spatial localization. Connecting a binaural branch to a large model means the model can regress the angle information of sound sources from inter-channel differences, rather than merely extracting "what sound is this" from the waveform. This branch inevitably introduces additional encoding and alignment costs, so the engineering challenge lies in how to align spatial features with visual and semantic features in a unified representation space, rather than simple concatenation.

Egocentric and allocentric are two reference frames in spatial cognition. The former describes positions from the observer's own perspective, such as "the sound is to my front-left"; the latter uses the scene or external reference points as the basis, such as "the sound source is to the right of the piano in the frame." The two are not perceptually equivalent, and are easily confused especially when the viewpoint changes dynamically. The model needs to master both reference frames to complete cross-perspective reasoning from "my left" to "which object in the frame is producing sound." The benchmark examines these two types of perspectives separately, which can determine whether the model has truly built a scene-level spatial representation or has only learned a directional shortcut tied to camera pose.

Experimental Results: Spatial Shortcomings of Existing Models and HLR-LLM's Improvements

The experimental results revealed two important findings:

Existing AV-LLMs still have significant limitations in spatial localization and relational reasoning. Even powerful audio-visual foundation models do not perform well when facing questions like "where is the sound coming from" and "what object in the scene is this sound related to." This shows that improvements in semantic understanding do not automatically bring spatial understanding.

HLR-LLM achieved significant improvements in spatial audio-visual scene understanding without compromising semantic audio-visual perception. This result validates the effectiveness of the dual-pathway architecture—adding a spatial branch does not cause the model to "get distracted" and lose its original semantic understanding capabilities.

The Significance of This Work

From a broader perspective, this work touches on a core issue in multimodal AI: perception has multiple dimensions, and semantics is only one of them. Humans naturally process both "what" and "where" simultaneously when understanding their environment, while current AI systems develop unevenly across these two dimensions.

The introduction of the HLR-AVSceneQA benchmark also provides a quantifiable evaluation tool for future research. With hierarchical diagnostic capabilities, researchers can more precisely identify specific model weaknesses, rather than vaguely concluding that "spatial understanding is lacking."

Of course, there is still a gap between benchmark testing and real-world applications. The acquisition quality of spatial audio on consumer devices, generalization ability across different acoustic environments, and the computational overhead of real-time inference are all issues that this type of technology must face on the path to practical use. But at least, this work provides a clear technical path for adding a "sense of space" to audio-visual large models.