MTG-AD-Eval
An evaluation framework built on audio description expert guidelines that tests, through end-to-end rendered audio tracks, whether inline descriptions are embedded within silent segments and avoid overlap, providing fine-grained reference-free metrics.
Timeline (last 90 days)
The Google team released the MTG-AD dataset and the MTG-AD-Eval evaluation framework for the inline audio description (inline AD) task.
The MTG-AD-Eval framework and code have been released on GitHub (google/mtg-ad).
The core step of MTG-AD-Eval is end-to-end rendering: the description scripts generated by the model are synthesized into audio tracks and placed back onto the original video timeline to check whether they fit into natural silent segments and whether they cause disruptive overlap.
MTG-AD-Eval provides fine-grained, reference-free, actionable evaluation metrics built on audio description expert guidelines.
The study validated the automated reference-free metrics using quality ratings from sighted raters and 7 BLV participants, finding a significant correlation between the two.
All Facts (5)
The core step of MTG-AD-Eval is end-to-end rendering: the description scripts generated by the model are synthesized into audio tracks and placed back onto the original video timeline to check whether they fit into natural silent segments and whether they cause disruptive overlap.
50%UnverifiedMTG-AD-Eval provides fine-grained, reference-free, actionable evaluation metrics built on audio description expert guidelines.
50%UnverifiedThe study validated the automated reference-free metrics using quality ratings from sighted raters and 7 BLV participants, finding a significant correlation between the two.
50%UnverifiedThe Google team released the MTG-AD dataset and the MTG-AD-Eval evaluation framework for the inline audio description (inline AD) task.
50%UnverifiedThe MTG-AD-Eval framework and code have been released on GitHub (google/mtg-ad).
50%