KeyASRBench
A bilingual benchmark for evaluating keyword retention in ASR. It proposes the Keyword Error Rate (KER) metric, based on occurrence counts and whole-phrase freezing annotation, to measure how well speech recognition systems retain business-critical phrases in their entirety.
Timeline (last 90 days)
KeyASRBench is a bilingual benchmark for measuring ASR systems' ability to preserve keywords (person names, product names, technical terms, version numbers, relational phrases) as whole phrases.
KeyASRBench proposes an occurrence-based Keyword Error Rate (KER), computed on frozen whole-phrase annotations, treating each keyword occurrence as an independent evaluation unit and counting it as preserved only if the entire phrase is completely correct.
KeyASRBench's challenge set is 10 hours, constructed conditioned on models, and contains 1,501 Chinese and English audio clips covering 4,723 keyword occurrences.
KeyASRBench's evaluation targets include five locally inferred checkpoints and two official API systems, ensuring that results do not depend on a single deployment environment.
In the KeyASRBench evaluation, the language-specific MER/WER range is 5.31% to 12.91%, while the KER range for the same set of systems is 13.61% to 33.81%.
KeyASRBench sets up a disjoint 30-minute model-blind control set to test whether the selection of the challenge set introduces bias.
On the 30-minute control set, KeyASRBench's KER range is 10.65% to 27.31%, lower than the challenge set but still significant in absolute terms, demonstrating that keyword errors are prevalent in real transcriptions.
KeyASRBench currently covers Chinese and English and a fixed keyword set, and its control set is 30 minutes in size; its value lies in proposing a reproducible KER framework rather than providing a universally applicable industry threshold.
All Facts (8)
KeyASRBench is a bilingual benchmark for measuring ASR systems' ability to preserve keywords (person names, product names, technical terms, version numbers, relational phrases) as whole phrases.
50%UnverifiedKeyASRBench proposes an occurrence-based Keyword Error Rate (KER), computed on frozen whole-phrase annotations, treating each keyword occurrence as an independent evaluation unit and counting it as preserved only if the entire phrase is completely correct.
50%UnverifiedKeyASRBench's challenge set is 10 hours, constructed conditioned on models, and contains 1,501 Chinese and English audio clips covering 4,723 keyword occurrences.
50%UnverifiedKeyASRBench's evaluation targets include five locally inferred checkpoints and two official API systems, ensuring that results do not depend on a single deployment environment.
50%UnverifiedIn the KeyASRBench evaluation, the language-specific MER/WER range is 5.31% to 12.91%, while the KER range for the same set of systems is 13.61% to 33.81%.
50%UnverifiedKeyASRBench sets up a disjoint 30-minute model-blind control set to test whether the selection of the challenge set introduces bias.
50%UnverifiedOn the 30-minute control set, KeyASRBench's KER range is 10.65% to 27.31%, lower than the challenge set but still significant in absolute terms, demonstrating that keyword errors are prevalent in real transcriptions.
50%UnverifiedKeyASRBench currently covers Chinese and English and a fixed keyword set, and its control set is 30 minutes in size; its value lies in proposing a reproducible KER framework rather than providing a universally applicable industry threshold.
50%