OpenTWBench: Taiwan Knowledge Benchmark Reveals How Evaluation Harness Shapes Model Scores

Large language model evaluation scores are often treated as a direct reflection of model capability. But a newly released open evaluation suite, OpenTWBench, offers an unsettling observation: on exactly the same questions, merely changing the answer extraction method can send accuracy from 20.8% to 93.9%. This means a substantial portion of "model capability" differences actually stems from the evaluation harness itself, not the model.

Two Benchmarks: Academic Exams and Contemporary Taiwanese Daily Life

OpenTWBench is built around two complementary benchmarks.

The first is tw-exam-bench, sourced from fifteen years of national examination questions released as open data by Taiwan's Ministry of Examination, covering 148 subjects and a total of 255,854 four-choice multiple-choice questions. This data is naturally high-quality: among 31,312 sets of duplicate questions, the answer key agreement rate reaches 99.91%, and the mere 27 discrepancies were each verified and attributed.

A second round of manual review removed 2,180 questions whose stems depended on images that were not carried in the text. All final scores are computed on the remaining 253,674 questions. This processing detail matters — it shows the benchmark builders recognized that flaws in question presentation can directly contaminate evaluation results.

The second is tw-formosa-bench, generated from community-curated corpora, containing 1,696 difficulty-graded questions about contemporary Taiwanese daily life. Its positioning is deliberately distinct from exam questions: this is the Taiwan that licensing exams would never ask about. One discipline governed the generation process — no model participated in writing reference answers, and the question set was filtered against a frozen pool of seven models, keeping that pool's scores at the passing margin (46.9%–74.6%, median around 54%). Leave-one-out experiments showed this filtering preserved model rankings (near-zero bias).

But this carefully designed difficulty window lasted only briefly. Within six weeks, frontier models cleared the entire range: the public leaderboard now has 108 models, 22 of which exceed the benchmark's ceiling, with the best reaching 91.6%. This itself is a dynamic worth recording — the "difficulty lifespan" of static benchmarks is shrinking.

How Much of a Score Comes from the Harness, Not the Model

The most valuable part of OpenTWBench is its quantification of evaluation harness effects.

Answer extraction method. On the same questions, switching answer extraction from pattern matching to explicit \boxed{} answers raised accuracy from 20.8% to 93.9%. This is the most jarring set of numbers in the paper — same model, same questions, only the extraction logic differs, and the score differs by more than fourfold.

Placeholders in the system prompt. Merely adding a placeholder character to the system prompt could vault a model from last place (58.2%) to first place (83.9%) on the same benchmark.

Thinking budget. Doubling the thinking token budget from 8k to 16k yielded only a 0.2% improvement while changing the distribution of failed questions — that is, more computation didn't make the model "smarter," it merely rearranged its mistakes.

Jury degradation. A verification panel of three judges ultimately proved to be "one judge in disguise": two of its members approved 92%–99% of the presented content. This directly undermines the reliability assumption behind multi-judge verification.

Answer extraction and the \boxed{} format: In LLM evaluation pipelines, models typically answer in natural language, and the evaluation script must "extract" the final answer to compare against the reference. Early pattern matching (regex for numbers, option letters, etc.) relied on the stability of model output format. Once a model changes its phrasing style — hiding the answer inside reasoning, using Chinese numerals, or repeating multiple candidates at the end — extraction fails, and correctly answered questions get marked wrong. \boxed{} is the LaTeX command for boxing a formula, widely used in math reasoning benchmarks as an explicit delimiter for the "final answer." Requiring models to put answers in \boxed{} establishes a strong convention between output format and extraction logic, nearly eliminating ambiguity. The 20.8%→93.9% gap OpenTWBench demonstrates is essentially not the model getting stronger, but extraction failure rates being eliminated. This warns evaluators: any claimed "accuracy" must be reported together with the extraction specification, otherwise it is neither reproducible nor comparable.

Thinking budget and test-time compute: For models with explicit reasoning capabilities (such as those outputting chain-of-thought or internal reasoning tokens), thinking budget refers to the upper limit of reasoning tokens they are allowed to consume. Expanding the budget falls under "test-time compute scaling," which intuitively should bring capability gains. But OpenTWBench observed that 8k→16k brought only a 0.2% improvement, and the distribution of failed questions changed. This indicates that extra reasoning compute doesn't necessarily translate into more correct answers — it may simply be longer wandering down wrong reasoning paths, or turning originally lucky guesses into errors. If thinking budget is not fixed during evaluation, scores for the same model are not comparable — the budget effectively becomes a hidden hyperparameter.

Why These Findings Deserve Serious Attention

Much of the industry's model comparisons, leaderboard rankings, and "capability improvement" narratives rest on an implicit assumption: that the evaluation harness is transparent and neutral. OpenTWBench breaks this assumption with same-question controlled experiments.

The scope of harness effects has been underestimated. When a change in extraction method can produce a 73-percentage-point difference, when a single character in the system prompt can reshuffle the top and bottom of a leaderboard, then cross-institution, cross-version, and cross-time score comparisons require exceptional caution. Without unified extraction specifications and prompt protocols, conclusions like "Model A scores 5 points higher than Model B" may simply be artifacts of harness differences.

Another implication concerns benchmark lifecycle. The difficulty range tw-formosa-bench carefully calibrated against a frozen model pool was breached within six weeks. This shows that benchmark difficulty is not a constant property — it drifts with model capability. Designers must either accept rapid benchmark saturation or establish continuous update mechanisms.

LLM-as-a-judge and jury degradation: Because open-ended responses are hard to score by exact match, the industry widely adopts the "LLM-as-a-judge" method, often using multiple judges with majority or averaging to improve reliability. The implicit assumption is that judges are independent and can catch each other's errors. The "jury degradation" OpenTWBench observed — three judges actually being one judge in disguise, with two members' approval rates reaching 92%–99% — directly challenges this assumption. When the same model or the same prompt template is reused to generate multiple judges, their judgments are highly correlated; the jury provides not independent redundancy but repetition of the same bias. This requires evaluation designers to report the sources of judge independence (whether models, prompts, temperatures, and scoring dimensions are genuinely differentiated), rather than judging reliability by count alone.

Benchmark saturation and difficulty lifespan: Static benchmarks are progressively breached by new models over time, with scores eventually approaching the ceiling and losing discriminative power — a phenomenon called benchmark saturation. OpenTWBench's value lies in providing a quantified case of "difficulty lifespan": the passing-margin difficulty window tw-formosa-bench calibrated against a frozen model pool (46.9%–74.6%) was broken by frontier models within six weeks, with 22 of 108 models exceeding the original ceiling. This shows that benchmark design cannot assume constant difficulty, and should plan update mechanisms in advance or shift toward dynamic adversarial evaluation (such as continuously generating new questions or adversarial question pools).

Open Assets

The OpenTWBench team has released all datasets, per-question difficulty data, filtering manifests, and a public leaderboard (https://opentwbench.ai). The release of per-question difficulty data is especially critical: it allows external researchers to independently reproduce harness effect experiments, and turns "how much of a score comes from the harness" from a case observation into a systematically testable object.

For evaluation practitioners, the real value of this toolkit may not lie in Taiwan knowledge itself, but in putting evaluation methodology problems on the table — scores are never just the model's story.

One-line Summary

OpenTWBench uses same-question controls to prove that harness differences can send accuracy from 20.8% to 93.9% — scores are never just the model's story.

Paragraph Summary

OpenTWBench is an open-source evaluation suite comprising tw-exam-bench (over 250,000 four-choice questions based on fifteen years of Taiwan national exams) and tw-formosa-bench (1,696 difficulty-graded questions generated by the community, covering contemporary Taiwanese daily life). Its core contribution isn't Taiwan knowledge itself, but quantifying the harness's impact on scores: on identical questions, merely changing answer extraction (pattern matching to explicit \boxed{}) raised accuracy from 20.8% to 93.9%; adding one placeholder character to the system prompt vaulted a model from last to first; doubling the thinking budget yielded only a 0.2% gain; a three-judge panel degraded to a single judge, with two members' approval rates at 92%–99%. Moreover, a carefully calibrated difficulty range was breached within six weeks, revealing the "difficulty lifespan" problem of static benchmarks. The authors call for unified extraction specifications and prompt protocols, lest cross-model, cross-version, and cross-time score comparisons be mere artifacts of harness differences.