KeyASRBench: A New Benchmark for ASR Keyword Retention

KeyASRBench proposes the KER metric, revealing systematic keyword retention weaknesses under low WER.
KeyASRBench is a Chinese-English bilingual benchmark designed to answer whether production ASR systems truly preserve business keywords. Traditional WER/MER metrics cannot capture the "all-or-nothing" nature of keywords, so it proposes an occurrence-weighted keyword error rate (KER) based on whole-phrase annotations. The challenge set contains 1,501 audio clips and 4,723 keyword occurrences; five local checkpoints and two API systems show MER/WER of 5.31%-12.91% but KER as high as 13.61%-33.81%. A 30-minute model-blind control set yields KER of 10.65%-27.31%, proving keyword errors are not a filtering bias. This gap explains why industry still needs hotword lists and contextual biasing, and suggests evaluation should add keyword-level metrics.
Public speech recognition benchmarks keep setting new records for low error rates, yet ASR systems in production still commonly rely on hotword lists, contextual biasing, retrieval augmentation, and post-processing correction. This contrast raises a key question: do transcriptions actually preserve the keywords that matter to the business—names, product names, technical terms, version numbers, and relational phrases? KeyASRBench is a bilingual benchmark designed to answer exactly this question.

From WER to KER: A Shift in Metric Perspective
Traditional ASR evaluation centers on word error rate (WER) or mixed error rate (MER), measuring the edit distance between the overall transcription and the reference text. Such metrics are sensitive to uniformly distributed recognition errors but cannot capture the "all-or-nothing" nature of keywords: if a product model number is off by a single character, the entire phrase loses its retrieval value, yet it contributes only a tiny increment to WER.
KeyASRBench proposes an occurrence-based keyword error rate (KER), computed over frozen whole-phrase annotations. "Frozen" means that the boundaries and content of keyword phrases are fixed before evaluation, preventing models from re-segmenting based on context to evade errors. KER treats each keyword occurrence as an independent evaluation unit and counts whether it is fully preserved, thereby directly quantifying "whole-phrase retention" capability.
Word error rate (WER) is the most fundamental evaluation metric in the ASR field, computed as the sum of substitutions, deletions, and insertions divided by the number of words in the reference text. It assumes every word carries equal error weight, thus providing a reasonable summary of overall recognition quality. But in keyword scenarios, this uniform weighting dilutes the severity of key-phrase errors: for a product name consisting of five words, even a single wrong word zeroes out the retrieval value of the entire phrase, while WER only increases by 1/5 of an error. Mixed error rate (MER) additionally incorporates formatting differences such as punctuation and capitalization into the statistics on top of WER, but it likewise cannot distinguish keywords from non-keywords. KER is designed precisely to fill this blind spot: it treats each keyword occurrence as an independent unit, counting it as retained only when the entire phrase is completely correct and otherwise as an error, thereby treating "partially correct" as "completely failed," which is closer to the exact-match requirements of downstream tasks.
Dataset and Evaluation Design
KeyASRBench's challenge set is 10 hours long, conditionally constructed per model, and contains 1,501 Chinese and English audio clips covering 4,723 keyword occurrences. The evaluation targets include five locally inferred checkpoints and two official API systems, ensuring results do not depend on a single deployment environment.
At the metric level, the language-specific MER/WER ranges from 5.31% to 12.91%, while the KER range for the same set of systems is 13.61% to 33.81%. The gap between the two ranges shows that systems with low overall error rates do not necessarily preserve keywords reliably—KER reveals structural weaknesses masked by averaged metrics.
The research team also set up a disjoint 30-minute model-blind control set to test whether challenge set selection introduced bias. KER on the control set ranges from 10.65% to 27.31%; although somewhat lower than the challenge set, the absolute values remain significant, proving that the errors on the selected phrases are not an artifact of challenge set filtering but are pervasive in real transcriptions.
Contextual biasing is a common technique in production ASR systems. Its core is to introduce an external word list or phrase list during decoding, raising the recognition probability of these specific word sequences by adjusting language model scores or using methods such as shallow fusion. A hotword list is a simplified form of contextual biasing, typically containing only a keyword list without distinguishing weights. Retrieval augmentation, by contrast, retrieves relevant text fragments from an external knowledge base after or during recognition to assist the language model in rescoring or error correction. These techniques are necessary precisely because general-purpose ASR models are prone to errors on keywords rare in training data. KeyASRBench's KER metric provides exactly the quantitative tool for evaluating the actual gains of these techniques: computing KER before and after introducing biasing makes the improvement in keyword retention immediately visible, whereas the change in WER is often negligible.
Implications for Production Systems
The systematic gap between KER and WER explains why industry still invests heavily in hotwords and contextual biasing in the era of low WER. For scenarios requiring exact matching—product model numbers in customer service tickets, drug names in medical records, party names in legal documents—KER reflects the true risk to downstream tasks better than WER.
This benchmark also suggests that evaluation methods need updating: reporting only overall error rates masks failure modes at the keyword level, whereas whole-phrase annotation and occurrence-weighted design provide a more business-relevant, actionable metric for measuring "usability."
Limitations and Future Directions
As a pilot, KeyASRBench currently covers Chinese and English and a fixed keyword set, with a 30-minute control set. Its value lies in proposing a reproducible KER framework rather than providing industry-universal thresholds. Future work can extend the framework to more languages and domains and explore the mapping between keyword weights and business impact.
One-line Summary
KeyASRBench proposes the KER metric, revealing systematic keyword retention weaknesses under low WER.
Paragraph Summary
KeyASRBench is a Chinese-English bilingual benchmark designed to answer whether production ASR systems truly preserve business keywords. Traditional WER/MER metrics cannot capture the "all-or-nothing" nature of keywords, so the benchmark proposes an occurrence-weighted keyword error rate (KER) based on whole-phrase annotations. The challenge set contains 1,501 audio clips and 4,723 keyword occurrences; five local checkpoints and two API systems show MER/WER of 5.31%-12.91% but KER as high as 13.61%-33.81%. The 30-minute model-blind control set yields KER of 10.65%-27.31%, proving keyword errors are not a filtering bias. This gap explains why industry still needs hotword lists, contextual biasing, and other post-processing methods, and suggests evaluation should add actionable metrics at the keyword level.
Related articles

Zed Editor Releases v1.29.2: Observations on a Routine Version Iteration
Zed editor releases patch v1.29.2, version bump by @JosephTLyons, commit verified via GitHub signature. This article covers the release's basic info, what a patch version means, and practical upgrade advice for users.

Anthropic IPO Questioned: Hacker News Debates Timing of Going Public
A Hacker News post titled "Just say no to Anthropic's IPO" sparks debate on the timing of Anthropic's public listing and AI capitalization, touching on valuation bubbles and mission independence.

Hands-On with the Genie 4 World Model: Directing Worlds in Real Time Like a Puppeteer
Google DeepMind's Genie 4 world model in depth: Special Guest X's hands-on demo of real-time scene direction via prompts, custom interactive apps in Project Genie, and prompts-as-seeds reproducibility.