[KongchangAI]
Benchmark

KeyASRBench

A bilingual benchmark for evaluating keyword retention in ASR. It proposes the Keyword Error Rate (KER) metric, based on occurrence counts and whole-phrase freezing annotation, to measure how well speech recognition systems retain business-critical phrases in their entirety.

Timeline (last 90 days)

Nov 10

KeyASRBench is a bilingual benchmark for measuring ASR systems' ability to preserve keywords (person names, product names, technical terms, version numbers, relational phrases) as whole phrases.

Unverified50%
Nov 10

KeyASRBench proposes an occurrence-based Keyword Error Rate (KER), computed on frozen whole-phrase annotations, treating each keyword occurrence as an independent evaluation unit and counting it as preserved only if the entire phrase is completely correct.

Unverified50%
Nov 10

KeyASRBench's challenge set is 10 hours, constructed conditioned on models, and contains 1,501 Chinese and English audio clips covering 4,723 keyword occurrences.

Unverified50%
Nov 10

KeyASRBench's evaluation targets include five locally inferred checkpoints and two official API systems, ensuring that results do not depend on a single deployment environment.

Unverified50%
Nov 10

In the KeyASRBench evaluation, the language-specific MER/WER range is 5.31% to 12.91%, while the KER range for the same set of systems is 13.61% to 33.81%.

Unverified50%
Nov 10

KeyASRBench sets up a disjoint 30-minute model-blind control set to test whether the selection of the challenge set introduces bias.

Unverified50%
Nov 10

On the 30-minute control set, KeyASRBench's KER range is 10.65% to 27.31%, lower than the challenge set but still significant in absolute terms, demonstrating that keyword errors are prevalent in real transcriptions.

Unverified50%
Nov 10

KeyASRBench currently covers Chinese and English and a fixed keyword set, and its control set is 30 minutes in size; its value lies in proposing a reproducible KER framework rather than providing a universally applicable industry threshold.

Unverified50%

All Facts (8)

Unverified

KeyASRBench is a bilingual benchmark for measuring ASR systems' ability to preserve keywords (person names, product names, technical terms, version numbers, relational phrases) as whole phrases.

50%
Unverified

KeyASRBench proposes an occurrence-based Keyword Error Rate (KER), computed on frozen whole-phrase annotations, treating each keyword occurrence as an independent evaluation unit and counting it as preserved only if the entire phrase is completely correct.

50%
Unverified

KeyASRBench's challenge set is 10 hours, constructed conditioned on models, and contains 1,501 Chinese and English audio clips covering 4,723 keyword occurrences.

50%
Unverified

KeyASRBench's evaluation targets include five locally inferred checkpoints and two official API systems, ensuring that results do not depend on a single deployment environment.

50%
Unverified

In the KeyASRBench evaluation, the language-specific MER/WER range is 5.31% to 12.91%, while the KER range for the same set of systems is 13.61% to 33.81%.

50%
Unverified

KeyASRBench sets up a disjoint 30-minute model-blind control set to test whether the selection of the challenge set introduces bias.

50%
Unverified

On the 30-minute control set, KeyASRBench's KER range is 10.65% to 27.31%, lower than the challenge set but still significant in absolute terms, demonstrating that keyword errors are prevalent in real transcriptions.

50%
Unverified

KeyASRBench currently covers Chinese and English and a fixed keyword set, and its control set is 30 minutes in size; its value lies in proposing a reproducible KER framework rather than providing a universally applicable industry threshold.

50%

Source Articles