The AI Annotation Challenge in Quran Recitation Error Checking: What Actually Counts as a Mistake?

An Overlooked NLP Niche

Using automatic speech recognition (ASR) to check Quran recitation sounds like a straightforward task — transcribe speech, then compare against the source text. But in practice, it's far more complex than it appears. This arXiv study exposes a core challenge: when a machine checks Quran recitation from ASR transcripts, how do you distinguish a "genuine mistake" from a repetition, a self-correction, an opening formula, or an acceptable spelling variation?

This isn't just a technical problem — it's fundamentally a problem of annotation convention. A reciter mispronouncing a word is an error, but what if they say the word again (repetition), correct themselves mid-recitation (repair), or use a standard opening invocation (opening formula)? None of these should be flagged as errors. The research team manually annotated 100 real recordings, producing 348 scoring units and 162 localization events across ten combination label types.

Baseline Evaluation: Localization Is Easy, Classification Is Hard

The researchers built an executable evaluator that scores both labels and word positions simultaneously. This dual evaluation is critical — you need to determine both "is there an event here?" and "what category does it belong to?"

The data reveals a striking gap between the two dimensions. A plain text diff approach achieved a label-aware F1 of 0.525, while localization F1 reached 0.826. An adapted production-grade cleaning and alignment component scored a label F1 of 0.518 and a localization F1 of 0.786, with both achieving an exact-span F1 of 0.505.

The message is clear: identifying where something might be wrong is relatively easy, but accurately determining what type of problem it is proves much harder. Localization F1 consistently sits above 0.78, while label F1 hovers around 0.5 — a gap that speaks for itself.

The Hidden Trap in Annotation Interfaces

One particularly illuminating finding: after correcting word coordinates in the repair adapter, all five previously annotated repetition events were successfully recovered. This shows that baseline failures aren't always due to the algorithm itself — sometimes the problem lies in the annotation interface or data coordinates.

The authors issue a methodological warning: before interpreting baseline failures, you must first audit the annotation interface. Without first ruling out technical defects in the annotation pipeline, you risk misattributing interface bugs to model inadequacy — a mistake that could lead to entirely wrong conclusions. This lesson applies to any machine learning project that relies on human-annotated data.

A Preliminary Experiment with AI Coding Agents

The study also ran a preliminary pilot: three coding agents and eight models each completed eight independent 20-minute single-pass tasks. The variance in results was enormous — label-aware F1 ranged from 0.143 all the way to 0.892.

Seven of the runs significantly outperformed all baselines, delivering impressive results. But one run fell below even the most naive diff method due to a single missing normalization step. This contrast vividly illustrates the decisive role of preprocessing in tasks like this: one overlooked normalization step is enough to cause a powerful model to completely fall apart.

The Real Bottleneck Is "Convention," Not "Detection"

The study's most thought-provoking conclusion emerges at the event detection level. Across six runs, 970 out of 972 gold event instances received overlapping predictions — meaning the models could almost always locate where an event occurred.

So where does the difficulty lie? The authors put it plainly: the remaining challenge is not detection, but convention — specifically, determining span extent and resolving label types whose boundaries are defined by adjudication rules rather than anything directly visible in the text.

Seven events (out of 162 total) stumped all six same-day runs, and five of them involved the same orthographic rule that even the strongest runs consistently missed. This suggests that judging certain error types depends on human-defined rule boundaries that text pattern recognition alone cannot resolve.

Limitations and Contributions

The authors are candid about the study's boundaries: none of the runs completed annotation before building the system, so this pilot only measures the "algorithmic half" of the task. In other words, the process of formulating the annotation conventions themselves — and the human judgment that entails — has not yet been covered by this automated pipeline.

The value of this research lies not in delivering a high-scoring model, but in cleanly decomposing the structure of a specialized domain evaluation task: detection, localization, label classification, and convention adjudication form a ladder of escalating difficulty. For NLP applications in highly convention-dependent domains like religious or legal texts, this analytical framework offers broadly applicable methodological guidance — technology can handle detection and localization, but defining "what counts as a mistake" ultimately still requires human convention to answer.