AuDiR
A diagnostic framework for audio reasoning based on controlled input interventions that separates three failure modes—perception failure, knowledge absence, and application failure—by progressively supplying rules and acoustic observation text.
Timeline (last 90 days)
AuDiR (Audio Diagnostic Reasoning) is an audio reasoning diagnostic framework based on controlled input interventions that separates three failure modes—perception failure, knowledge absence, and application failure—by progressively providing rules and acoustic observation text.
Each AuDiR test item consists of four parts: an audio clip, a multiple-choice question, a task-general rule, and a text describing the acoustic observation needed to answer the question.
AuDiR sets up three progressive intervention conditions plus a condition in which the audio is removed but the question and rule are kept; whichever condition a model first answers correctly under is assigned an operational response label.
The AuDiR benchmark currently contains 201 items, 18 families, from 9 sources.
The 25.0% random-guess level in AuDiR corresponds to the four-option multiple-choice format.
Preliminary evaluation on Gemini-3.8-flash shows: when no auxiliary information is provided, accuracy is 29.3%.
On Gemini-3.8-flash, accuracy rises to 31.3% when the rule is provided, and reaches 98.5% when the acoustic observation text is also provided.
AuDiR finds a family-level statistical association between incomplete rule recall and larger gains from providing rules, i.e., the worse a model masters a certain type of rule, the more pronounced the improvement after the rule is supplied.
The authors of AuDiR point out that family-level statistical associations cannot be used to directly infer the causal path of individual items, and that there is a risk of ecological fallacy.
AuDiR probes the recognition shortcuts (shortcut learning) that models may exploit through the condition in which audio is removed, but the authors acknowledge that shortcuts may exist in more subtle forms.
3 more timeline events
All Facts (13)
AuDiR (Audio Diagnostic Reasoning) is an audio reasoning diagnostic framework based on controlled input interventions that separates three failure modes—perception failure, knowledge absence, and application failure—by progressively providing rules and acoustic observation text.
50%UnverifiedEach AuDiR test item consists of four parts: an audio clip, a multiple-choice question, a task-general rule, and a text describing the acoustic observation needed to answer the question.
50%UnverifiedAuDiR sets up three progressive intervention conditions plus a condition in which the audio is removed but the question and rule are kept; whichever condition a model first answers correctly under is assigned an operational response label.
50%UnverifiedThe AuDiR benchmark currently contains 201 items, 18 families, from 9 sources.
50%UnverifiedThe 25.0% random-guess level in AuDiR corresponds to the four-option multiple-choice format.
50%UnverifiedPreliminary evaluation on Gemini-3.8-flash shows: when no auxiliary information is provided, accuracy is 29.3%.
50%UnverifiedOn Gemini-3.8-flash, accuracy rises to 31.3% when the rule is provided, and reaches 98.5% when the acoustic observation text is also provided.
50%UnverifiedAuDiR finds a family-level statistical association between incomplete rule recall and larger gains from providing rules, i.e., the worse a model masters a certain type of rule, the more pronounced the improvement after the rule is supplied.
50%UnverifiedThe authors of AuDiR point out that family-level statistical associations cannot be used to directly infer the causal path of individual items, and that there is a risk of ecological fallacy.
50%UnverifiedAuDiR probes the recognition shortcuts (shortcut learning) that models may exploit through the condition in which audio is removed, but the authors acknowledge that shortcuts may exist in more subtle forms.
50%UnverifiedThe authors of AuDiR candidly acknowledge limitations of the current work, including incomplete validation, the possibility that models exploit recognition shortcuts, and non-monotonic response patterns.
50%UnverifiedAuDiR's operational response labels describe only response patterns at the operational level and are not equivalent to identifying a unique internal causal mechanism.
50%UnverifiedWhen interpreting the results of the no-audio condition, whether the confidence interval covers 25.0% is the key statistical basis for judging whether a model can arrive at an answer from the question and rule alone.
50%