[KongchangAI]
ConceptAudio Diagnostic Reasoning

AuDiR

A diagnostic framework for audio reasoning based on controlled input interventions that separates three failure modes—perception failure, knowledge absence, and application failure—by progressively supplying rules and acoustic observation text.

Timeline (last 90 days)

Nov 10

AuDiR (Audio Diagnostic Reasoning) is an audio reasoning diagnostic framework based on controlled input interventions that separates three failure modes—perception failure, knowledge absence, and application failure—by progressively providing rules and acoustic observation text.

Unverified50%
Nov 10

Each AuDiR test item consists of four parts: an audio clip, a multiple-choice question, a task-general rule, and a text describing the acoustic observation needed to answer the question.

Unverified50%
Nov 10

AuDiR sets up three progressive intervention conditions plus a condition in which the audio is removed but the question and rule are kept; whichever condition a model first answers correctly under is assigned an operational response label.

Unverified50%
Nov 10

The AuDiR benchmark currently contains 201 items, 18 families, from 9 sources.

Unverified50%
Nov 10

The 25.0% random-guess level in AuDiR corresponds to the four-option multiple-choice format.

Unverified50%
Nov 10

Preliminary evaluation on Gemini-3.8-flash shows: when no auxiliary information is provided, accuracy is 29.3%.

Unverified50%
Nov 10

On Gemini-3.8-flash, accuracy rises to 31.3% when the rule is provided, and reaches 98.5% when the acoustic observation text is also provided.

Unverified50%
Nov 10

AuDiR finds a family-level statistical association between incomplete rule recall and larger gains from providing rules, i.e., the worse a model masters a certain type of rule, the more pronounced the improvement after the rule is supplied.

Unverified50%
Nov 10

The authors of AuDiR point out that family-level statistical associations cannot be used to directly infer the causal path of individual items, and that there is a risk of ecological fallacy.

Unverified50%
Nov 10

AuDiR probes the recognition shortcuts (shortcut learning) that models may exploit through the condition in which audio is removed, but the authors acknowledge that shortcuts may exist in more subtle forms.

Unverified50%

3 more timeline events

All Facts (13)

Unverified

AuDiR (Audio Diagnostic Reasoning) is an audio reasoning diagnostic framework based on controlled input interventions that separates three failure modes—perception failure, knowledge absence, and application failure—by progressively providing rules and acoustic observation text.

50%
Unverified

Each AuDiR test item consists of four parts: an audio clip, a multiple-choice question, a task-general rule, and a text describing the acoustic observation needed to answer the question.

50%
Unverified

AuDiR sets up three progressive intervention conditions plus a condition in which the audio is removed but the question and rule are kept; whichever condition a model first answers correctly under is assigned an operational response label.

50%
Unverified

The AuDiR benchmark currently contains 201 items, 18 families, from 9 sources.

50%
Unverified

The 25.0% random-guess level in AuDiR corresponds to the four-option multiple-choice format.

50%
Unverified

Preliminary evaluation on Gemini-3.8-flash shows: when no auxiliary information is provided, accuracy is 29.3%.

50%
Unverified

On Gemini-3.8-flash, accuracy rises to 31.3% when the rule is provided, and reaches 98.5% when the acoustic observation text is also provided.

50%
Unverified

AuDiR finds a family-level statistical association between incomplete rule recall and larger gains from providing rules, i.e., the worse a model masters a certain type of rule, the more pronounced the improvement after the rule is supplied.

50%
Unverified

The authors of AuDiR point out that family-level statistical associations cannot be used to directly infer the causal path of individual items, and that there is a risk of ecological fallacy.

50%
Unverified

AuDiR probes the recognition shortcuts (shortcut learning) that models may exploit through the condition in which audio is removed, but the authors acknowledge that shortcuts may exist in more subtle forms.

50%
Unverified

The authors of AuDiR candidly acknowledge limitations of the current work, including incomplete validation, the possibility that models exploit recognition shortcuts, and non-monotonic response patterns.

50%
Unverified

AuDiR's operational response labels describe only response patterns at the operational level and are not equivalent to identifying a unique internal causal mechanism.

50%
Unverified

When interpreting the results of the no-audio condition, whether the confidence interval covers 25.0% is the key statistical basis for judging whether a model can arrive at an answer from the question and rule alone.

50%

Source Articles