Dấu or Die
A Kaggle benchmark constructed by Vietnamese developer Tuan Nguyen, designed to evaluate the ability of large language models to restore Vietnamese tone marks (dấu). It tests LLM contextual understanding and reasoning on a low-resource tonal language.
Timeline (last 90 days)
Vietnamese developer Tuan Nguyen built a Kaggle benchmark called 'Dấu or Die' to test whether large language models can restore removed Vietnamese tone marks.
The benchmark contains 37 handwritten sentences divided into five categories: everyday, ambiguous, idiom, code_mixed, and proper_noun.
The evaluation pipeline uses Unicode NFD normalization to automatically strip tone marks and map đ to d, with fully deterministic scoring and no LLM used as a judge.
The key metric, the headline score, refers to the proportion of sentences fully and correctly restored; if even one tone mark is wrong the entire sentence is counted as incorrect, and the test is built on the kaggle-benchmarks library.
For production-grade diacritic restoration, a small model is a sensible choice, provided the right small model is selected.
The exact-match metric cannot distinguish between output that is close but semantically correct and output that is completely wrong; the author plans to accept a set of valid answers for each sentence in a future version.
All Facts (6)
Vietnamese developer Tuan Nguyen built a Kaggle benchmark called 'Dấu or Die' to test whether large language models can restore removed Vietnamese tone marks.
50%UnverifiedThe benchmark contains 37 handwritten sentences divided into five categories: everyday, ambiguous, idiom, code_mixed, and proper_noun.
50%UnverifiedThe evaluation pipeline uses Unicode NFD normalization to automatically strip tone marks and map đ to d, with fully deterministic scoring and no LLM used as a judge.
50%UnverifiedThe key metric, the headline score, refers to the proportion of sentences fully and correctly restored; if even one tone mark is wrong the entire sentence is counted as incorrect, and the test is built on the kaggle-benchmarks library.
50%UnverifiedFor production-grade diacritic restoration, a small model is a sensible choice, provided the right small model is selected.
50%UnverifiedThe exact-match metric cannot distinguish between output that is close but semantically correct and output that is completely wrong; the author plans to accept a set of valid answers for each sentence in a future version.
50%