Can Translated Benchmarks Replace Native-Language Evaluation? A Study on MMLU Proxy Validity

The Common Practice of Translated Benchmarks and Its Hidden Concerns

In large model evaluation for non-English languages, it is common practice to directly use machine-translated English benchmarks (such as MMLU). The underlying assumption is that the translated test still measures the same language and knowledge abilities. But a study titled "Gold vs. Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?" systematically questions this assumption, framing the issue as one of proxy validity.

The study compared DeepL-translated MMLU with six native-language benchmarks, covering five Nordic and Baltic languages across seven paired comparisons, with 29 models evaluated per pair. The core question is not merely whether rankings agree, but whether, after controlling for model scale and language proficiency, translated scores still provide incremental predictive value for native-language performance.

Proxy validity is borrowed from validity theory in psychometrics. When a measurement instrument does not directly measure the target construct but infers it through an observable surrogate indicator, the proxy validity of that surrogate must be assessed. In the evaluation context, the translated benchmark is the "proxy," and the ability measured by the native benchmark is the "target construct." The core question of proxy validity is whether there is a sufficiently strong association between proxy scores and target scores that still holds after controlling for confounders. Rank correlation alone is insufficient to establish proxy validity, because rank correlation may be driven by shared factors such as model scale and language proficiency; therefore, the study adopts incremental explanatory power (ΔR²) as a stricter criterion, testing whether translated scores still explain additional variance in native-language performance after controlling for covariates.

Proxy Validity Varies by Benchmark

The conclusion is not simply that translated benchmarks are "usable" or "unusable." After applying a unified robustness standard to the seven pairs, the study obtained four robust effects, two marginal effects, and one null result.

The strongest evidence appeared for curricula-overlapping benchmarks such as Skolprov, with an incremental explanatory power of ΔR²=.128 (95% CI [.038,.253]), indicating that translated MMLU scores significantly predict performance on this native benchmark. Conversely, for the Sweden-specific factual knowledge benchmark SwedishFacts, translated MMLU provided no significant incremental value (p=.242).

This contrast points to a clear pattern: proxy performance is better when the source content of the translated benchmark overlaps with the native evaluation target, whereas locally anchored knowledge is difficult to capture reliably.

ΔR² (incremental coefficient of determination) is a key metric in hierarchical regression: how much the proportion of variance explained by the regression model increases when translated MMLU scores are added on top of a baseline model that already includes control variables (such as model scale and language proficiency). A larger ΔR² indicates that translated scores carry more predictive information independent of the control variables, and thus stronger proxy validity. The 95% confidence intervals reported in the study (e.g., [.038,.253]) are used to judge whether this increment robustly deviates from zero; if the interval excludes zero, the effect is statistically significant. Furthermore, the classification of "four robust, two marginal, and one null result" results from applying a unified robustness standard to the seven pairs; this pre-specified uniform criterion helps avoid selective reporting across different pairs, thereby reducing the risk of false positives from multiple comparisons.

Language Ability or Local Knowledge?

The study also reveals a confounding factor: SwedishFacts is the only benchmark using an open-ended response format, so the respective effects of local content and response format cannot be fully disentangled. This means the conclusion that "local knowledge is harder to proxy with translated benchmarks" may partly stem from format differences rather than purely from knowledge locality.

From a methodological perspective, this study reminds evaluators that translated benchmarks may measure more of general language ability and curricula knowledge than culture- or region-specific factual knowledge. Claiming that a model "has mastered Swedish knowledge" based on translated MMLU lacks evidential support.

Open-ended response format refers to an evaluation format in which the model freely generates text rather than selecting an answer from options. Compared with multiple-choice formats, open-ended formats typically require an additional scoring step (such as human or model grading) and may introduce additional sources of variance such as rater agreement, answer length, and stylistic expression. Precisely because of this, when a benchmark possesses both an open-ended format and local content, the contributions of the two to results are statistically difficult to separate—this is the confound identified by the study. To clarify this confound, an ideal design would be to construct a 2×2 benchmark matrix crossing format and content, or to compare two response formats on the same content, thereby isolating format effects from content-locality effects.

Implications for Practitioners

For teams requiring non-English evaluation, this study provides actionable criteria: if the evaluation goal is general subject ability, translated benchmarks can serve as low-cost proxies; if the goal is localized knowledge, native-created benchmarks should be prioritized. Designing cross-lingual evaluation requires first answering "what to measure" before deciding "what to measure with."