AuDiR: Diagnosing the Root Causes of Audio Reasoning Failures with Rule and Perception Interventions

AuDiR uses controlled interventions to split audio reasoning failures into perception, knowledge, and application causes.
AuDiR proposes a diagnostic framework based on controlled input interventions to separate audio reasoning failures into perception failures, missing rule knowledge, and rule application failures. Each item includes audio, a multiple-choice question, a task rule, and an acoustic observation description, with progressive conditions and an audio-removed control. On Gemini-3.8-flash, accuracy rises from 29.3% to 31.3% with the rule and to 98.5% with the observation text, highlighting a perception bottleneck; the no-audio condition (23.9%) is indistinguishable from chance.
Why Accuracy Alone Masks the Real Problems in Audio Reasoning
In audio reasoning tasks, a model can arrive at a wrong answer for very different reasons: it may not have heard the sound correctly (perception failure), it may not know the relevant rule (missing knowledge), or it may know the rule but fail to apply it (application failure). Final accuracy alone conflates these failure modes, making it hard for developers to improve models in a targeted way. AuDiR (Audio Diagnostic Reasoning) proposes a diagnostic framework based on controlled input interventions that attempts to separate these three types of failure.

How AuDiR Designs Its Intervention Experiments
Each AuDiR test item has four parts: an audio clip, a multiple-choice question, a task-general rule, and a text passage describing the acoustic observations needed to answer the question. By progressively supplying the rule and the observation description, the experiment sets up three incremental conditions, plus one condition in which the audio is removed while the question and rule are kept. The condition under which the model first answers correctly defines an operational response label.
The key to this design is "controlling variables": if the model answers correctly as soon as it receives the rule, the failure likely stems from missing rule knowledge; if it answers correctly only after receiving the acoustic observation text, the failure more likely lies in perception. Note that this label describes only an operational-level response pattern and is not equivalent to identifying a single internal causal mechanism.
The AuDiR benchmark currently contains 201 items across 18 families from 9 sources, with reference answers and perception extraction recorded separately.
Multiple-choice questions and the random-guess baseline
The 25.0% random-guess level mentioned in the paper corresponds to the four-option multiple-choice format. In evaluation, the random-guess baseline is an important reference point: if a model's accuracy without key inputs (such as audio) does not differ significantly from random guessing, it means the model gains no useful signal from the remaining information and can only guess blindly. Conversely, significantly above-chance performance suggests the model may be exploiting shortcuts or prior biases. So when interpreting AuDiR's no-audio condition results, whether the confidence interval covers 25.0% is the key statistical basis for judging whether the model can "fill in" the answer.
Confidence intervals and statistical significance
Confidence intervals express the range of uncertainty around an estimate. When the sample size is limited (e.g., 201 items), a point accuracy estimate may deviate from the true level due to sampling fluctuation. AuDiR reports that the 23.9% accuracy in the no-audio condition has a confidence interval that includes 25.0%, meaning this result is statistically indistinguishable from random guessing, so we cannot claim the model has reasoning ability beyond guessing. This kind of interval estimate supports the conclusion that "audio input is necessary" better than a point estimate alone, and it also reminds readers that when comparing accuracy differences across conditions, they should consider the margin of error rather than just the magnitude of the numbers.
Preliminary Evaluation: Performance of Gemini-3.8-flash
A preliminary evaluation on Gemini-3.8-flash shows 29.3% accuracy with no auxiliary information; 31.3% after providing the rule; and 98.5% when the acoustic observation text is also provided. With audio removed but the question and rule kept, accuracy is 23.9%, whose confidence interval includes the 25.0% random-guess level.
These data send two signals. First, providing acoustic observations to the model in text form yields a far larger improvement than providing only the rule—indicating that for this model, the perception bottleneck is more prominent than the rule-knowledge bottleneck. Second, accuracy in the no-audio condition is no different from random guessing, showing that the model cannot "fill in" the correct answer from the question and rule alone, which indirectly confirms the necessity of audio input.
The Link Between Knowledge Probes and the Gains from Providing Rules
AuDiR also designs a family-level knowledge probe and finds a correlation between incomplete rule recall and larger gains from providing the rule. In other words, the worse the model grasps a given type of rule, the more pronounced the improvement when the rule is supplied. This correlation provides additional consistency evidence for the diagnostic framework, but the authors also note that it is only a family-level statistical association and cannot be used to directly infer causal pathways for individual items.
What family-level analysis means
The AuDiR benchmark groups test items into "families," where items within the same family share similar task rules or acoustic phenomena. The family-level knowledge probe assesses the model's grasp of a given type of rule at this aggregated level rather than item by item. The advantage of this design is that it reduces per-item noise and improves statistical stability, but it also carries the risk of ecological fallacy: family-level associations (worse rule recall, larger gains from supplying the rule) cannot be directly generalized to individual items, because individual differences may be masked by aggregation. The authors explicitly distinguish these two levels precisely to avoid overinterpreting causality.
Identifying shortcuts (shortcut learning)
Shortcut learning refers to a model making correct judgments not by genuinely performing the reasoning the task requires, but by exploiting surface cues in the data or questions (such as option length, keyword co-occurrence, or answer position distribution). In audio reasoning, a model may "guess" the answer through semantic matching between the text prompt and the options while completely ignoring the audio content. Such shortcuts create a false impression of success in accuracy, causing evaluations to overestimate a model's true ability. AuDiR probes for this behavior through its audio-removed condition, but the authors acknowledge that shortcuts may exist in more subtle forms and that more controlled experiments are needed to rule them out.
Limitations and Open Questions
The authors of AuDiR acknowledge several limitations of the current work. Validation is still incomplete; models may exploit shortcuts rather than genuinely performing acoustic reasoning; and response patterns exhibit non-monotonicity, meaning that accuracy does not necessarily rise monotonically as more information is provided. These factors limit the reliability of stronger failure-attribution conclusions.
From a methodological standpoint, the value of AuDiR lies not in providing the ultimate answer to "why the model is wrong," but in offering an operational, reproducible diagnostic procedure that moves failure analysis in audio reasoning from a single accuracy number toward multidimensional attribution. Future work needs more thorough validation to support stronger causal claims.
Implications for Audio Reasoning Evaluation
AuDiR reminds us that evaluating audio reasoning models should not stop at accuracy leaderboards. Through rule and perception intervention experiments, researchers can pinpoint model weaknesses at a finer granularity: is it unable to hear clearly, does it not know, or can it not apply what it knows? This diagnostic approach is instructive for building more reliable audio understanding systems and offers a paradigm that future benchmark design can draw on.
One-line Summary
AuDiR uses controlled intervention experiments to break audio reasoning failures into perception, knowledge, and application categories, revealing the real bottlenecks that accuracy masks.
Paragraph Summary
AuDiR proposes a diagnostic framework based on controlled input interventions that aims to separate the failure modes of audio reasoning models into perception failures, missing rule knowledge, and rule application failures, rather than lumping them together under final accuracy. Each test item includes audio, a multiple-choice question, a task-general rule, and an acoustic observation description; by progressively supplying the rule and observation text, it sets up incremental conditions plus a control condition with audio removed, using the stage at which the model first answers correctly as an operational response label. A preliminary evaluation on Gemini-3.8-flash shows 29.3% accuracy with no auxiliary information, only 31.3% after providing the rule, but a jump to 98.5% after providing the acoustic observation text, indicating that the perception bottleneck is far more prominent than the rule-knowledge bottleneck; in the no-audio condition, accuracy is 23.9% with a confidence interval covering the 25.0% chance level, showing that the model cannot infer the answer from the rule and question alone. A family-level knowledge probe further finds that incomplete rule recall correlates with larger gains from providing the rule. The authors also acknowledge limitations such as incomplete validation, possible shortcut learning, and non-monotonic responses, which limit stronger causal attribution.
Related articles

RePlan-VLA: Progress-Aware Replanning Breakthrough for Long-Horizon Robotic Manipulation
The RePlan-VLA framework converts VLA models' short-horizon capabilities into reliable long-horizon execution through progress-aware adaptive replanning, improving success rates by 21.9% and 10.4% on RoboCasa365 and CALVIN, and validating real-world deployment on a Franka Research 3 robot.

Geometric Bridge Flow Matching: A New Breakthrough in Blind Rigid Motion Correction
Geometric bridge flow matching enables blind rigid motion correction in head CBCT, achieving 37.1 dB PSNR and 0.27 mm RPE on simulated data in ~9.1 minutes, outperforming existing methods.

Google Releases Gemini 4 Hydrogen: 2.5x Faster Without Sacrificing Intelligence
Google releases the lightweight Gemini 4 Hydrogen: 2.5x faster task completion, AA Intelligence Index of 54 beating its flagship sibling, 7.3s single-step on Agent Arena faster than Claude and GPT rivals, four thinking levels with pay-as-you-need pricing, now available across AI Studio and more.