Dấu or Die Benchmark: Can LLMs Restore Vietnamese Tone Marks?

Vietnamese developer Tuan Nguyen did something fascinating: he built a Kaggle benchmark called Dấu or Die that specifically tests whether large language models (LLMs) can restore missing Vietnamese tone marks. What looks like a trivial task actually reveals real challenges in low-resource language processing.

Why Test Tone Restoration?

Vietnamese has six tones, plus a set of vowel diacritics (ă, â, ê, ô, ơ, ư, đ). In practice, however, many Vietnamese people skip these marks when typing — what's known as "không dấu" (no tones): phones without a Vietnamese keyboard, URLs and filenames, usernames, and legacy databases or SMS systems that only accept ASCII all force text to go "bare."

The problem is that removing tones causes many words to collapse into the same character string. For example, ban could be bàn (table), bán (sell), bạn (friend), or bận (busy). Native speakers instantly disambiguate from context, and the author wanted to know: do LLMs genuinely understand context, or do they just default to the most common word?

His favorite test sentence is:

Cai ban nay ban bao nhieu tien vay ban? → Cái bàn này bán bao nhiêu tiền vậy bạn? ("Hey friend, how much does this table cost?")

The same three letters appear three times, and each one must be restored to a different word. This is the ultimate test of contextual reasoning.

Vietnamese is an isolating tonal language where tones not only distinguish meaning but also carry grammatical information. The six tones are: level tone (thanh ngang, unmarked), falling tone (huyền, ), questioning tone (hỏi, ?), broken tone (ngã, ~), rising tone (sắc, ´), and heavy tone (nặng, .), each corresponding to a dedicated diacritic set in the standard Vietnamese writing system (Chữ Quốc ngữ). Vietnamese also has seven basic vowels, six of which can generate new phonemes through diacritics (e.g., a→ă/â, e→ê, o→ô/ơ, u→ư), while đandd` are two distinct consonants. With tones and vowel diacritics stacked together, a single syllable can have over a dozen different written forms, each with a completely different meaning. This makes "toneless Vietnamese" (tiếng Việt không dấu) informationally dense with ambiguity in a way that goes far beyond simply removing accent marks — it effectively compresses multiple parallel lexical systems into a single character stream.

How the Benchmark Works

The author hand-wrote 37 sentences across five categories, covering linguistic phenomena of varying difficulty:

• everyday: Ordinary everyday sentences, serving as a baseline
• ambiguous: Sentences built around easily confused words (bàn/bán/bạn/bận, cười/cưới, ốm/ôm, thấy/thầy, cá/cà/cả, etc.)
• idiom: Proverbs and set phrases with fixed wording, testing cultural memory
• code_mixed: Vietnamese mixed with English developer slang (e.g., "Em deploy code lên server rồi…"), where English words must remain unchanged
• proper_noun: Place names and personal names, such as Hà Nội, Đà Nẵng, Nguyễn Du, Truyện Kiều

The evaluation process is rigorous: tones are automatically stripped using Unicode NFD normalization, with đ mapped to d; models return structured output {restored_text: str}; scoring is entirely deterministic, with no LLM-as-judge.

The key metric is called the "headline score" — the proportion of sentences restored completely correctly. A single wrong tone marks the whole sentence wrong, because one incorrect tone is enough to change the meaning. Per-category word-level accuracy is also reported. The entire test is built on the kaggle-benchmarks library.

The Unicode NFD (Canonical Decomposition) normalization used in evaluation is the standard technique for stripping tones: NFD decomposes characters with diacritics into a base character plus combining diacritic sequences (e.g., ắ → a + breve + acute), after which filtering out all code points belonging to the "combining diacritic" Unicode category precisely restores pure ASCII base characters while leaving English and numbers intact. This approach is more robust than regex substitution, handling various precomposed Vietnamese characters without needing to maintain a manual mapping table. The separate đ→d mapping is needed because đ remains a single code point after NFD decomposition and must be handled explicitly. Using deterministic string matching rather than an LLM judge for scoring avoids the systematic bias of "using AI to evaluate AI," though at the cost of being unable to distinguish between "close but semantically correct" and "completely wrong" outputs — which is the author's motivation for introducing multiple reference answers in future versions.

Results from Seven Models

The author selected seven models from five different labs, mixing "fast and cheap" tiers with larger models, wanting to see whether model scale or lab training philosophy matters more for this kind of low-resource language task.

Rank	Model	Sentence Score	Correct Sentences
1	Gemini 3.7 Flash	1.00	37/37
2	GLM-5	0.92	34/37
2	Gemini 2.5 Pro	0.92	34/37
4	GPT-5.4 mini	0.89	33/37
5	Claude Haiku 5.5	0.84	31/37
6	Qwen 3 Coder 480B	0.68	25/37
–	Gemma 4 26B	Error	–

It's worth noting that Gemma 4 26B, Google's open-weights model, terminated with a runtime error before producing any scores. The author admits this was a bug in his own evaluation framework — the scoring code assumes at least one sentence is returned in the correct structured format, and if the model fails on every sample, the aggregation crashes. He emphasizes this is "not a verdict on Gemma."

Four Noteworthy Findings

The newer "Flash" tier beat the older "Pro" tier. Gemini 3.7 Flash restored all sentences correctly, while Gemini 2.5 Pro got 3 wrong. For this task, generation recency matters more than tier size.

Bigger isn't better if the model specializes elsewhere. Qwen 3 Coder has 480B parameters — the largest model in the test — yet came in last (25/37). A model fine-tuned for code seems to have lost the fine-grained multilingual nuance that Vietnamese tones demand.

The task looks easy, but the spread is wide. Any native speaker can restore these sentences effortlessly, yet there's a 12-sentence gap between the best and worst model. One wrong tone means the whole sentence is wrong — and that matters to readers too: "bán nhà" (sell a house) and "bàn nhà" (table house) are absolutely not "close enough."

Paying more at the top barely buys improvement. Kaggle's score-vs-cost chart shows GPT-5.4 mini achieved a 0.89 score for roughly $0.02 on the entire benchmark; GLM-5 got just one more sentence right (0.92) but cost over 10× more, around $0.30. For a production "add tones back" feature, a small model is the smart choice — provided you pick the right small model.

A Surprise: The Benchmark Found Its Own Ambiguity

An interesting side story: the benchmark exposed its own limitations. In an initial development run, Gemini 3.7 Flash restored Sep bao hop online… as "Sếp báo họp online" (the boss "announces" a meeting) rather than the author's reference answer "Sếp bảo họp online" (the boss "says" there's a meeting). Both are perfectly valid Vietnamese. When the author reran at temperature 0 for the leaderboard, the model chose "bảo" and scored a perfect 1.00.

This yields two lessons: first, removing tones sometimes destroys information that no amount of context can recover, so exact match is a somewhat harsh metric and future versions should accept a set of valid answers per sentence; second, even at temperature 0, model outputs aren't completely stable, and this instability appears precisely on genuinely ambiguous items — a single leaderboard number obscures this uncertainty.

The ambiguity between "bảo" (say, tell) and "báo" (announce, report) is a classic case of "context insufficient to disambiguate" in Vietnamese. In the toneless form bao, both words have the same syntactic function (both can serve as verbs taking objects) and highly similar semantics, making it impossible to determine which is more "correct" from a single-sentence context alone. This kind of true ambiguity is fundamentally different from a model error: a model error means choosing wrong when there's a unique correct answer, while true ambiguity means multiple linguistically valid answers objectively exist. The fact that the model produces different outputs at temperature 0 is precisely because the token probability distribution for such items is extremely close, and minor differences in random seed or underlying sampling are enough to flip the result. This also demonstrates that temperature does not equal "determinism" — it only reduces randomness without eliminating branching instability when probabilities are near-equal.

What's Next

The author outlined directions for future measurement: fix the evaluation framework so format failures score as 0 rather than crashing, then rerun open-source models like Gemma; support multiple valid answers per sentence (bảo/báo) and run repeated trials to measure stability; add teen slang that mixes missing tones (like "ko", "dc", "j z"); test longer passages; cover North-South dialect vocabulary differences; and even a reverse test — can models detect "wrong tones" introduced by typos?

This project, independently completed by Tuan Nguyen, uses a tiny but elegant benchmark to reveal many universal challenges in evaluating low-resource language tasks: generation recency matters more than scale, specialization sacrifices generality, cost-benefit ratios don't scale linearly, and exact-match metrics carry inherent tension. For practitioners working in multilingual NLP, it's a methodological case study well worth borrowing from.