Trained Only in English, Yet It Took First Place in Japanese: Darwin ZTC Tops S1MB

What Is S1MB, and Why Did It Add a Japanese Track?

S1MB (System One Mosaic Benchmark) is a benchmark suite built specifically to evaluate "decision models." These aren't text-generating models — they output judgments directly. Give one a reasoning scenario, and it returns a conclusion rather than an explanation. S1MB works by ranking multiple specialized benchmarks individually, converting each benchmark's ranking into a score, and averaging them into a Borda total score, preventing any single task from dominating the final result.

Recently, the leaderboard's operators added a Japanese track, japanese-v1, comprising 27 Japanese benchmarks, and tested existing models against it directly. The leaderboard is publicly accessible, and anyone can verify the data in a browser.

The Borda count is a voting aggregation method originating from 18th-century French mathematician Jean-Charles de Borda. The core idea: have each voter rank the candidates, assign points by rank (e.g., with N candidates, first place gets N points, second gets N-1, and so on), then sum all voters' points to produce a ranking.

S1MB borrows this logic: treat each specialized benchmark as a "voter," with the model's rank in that benchmark serving as its ordering, then convert to points and average. This offers two advantages — first, it prevents any single benchmark from dominating the overall leaderboard due to a large question count or high variance; second, even if a model performs mediocrely on one item, as long as it consistently ranks near the top across many benchmarks, its total score will still be high. This also explains why Darwin ZTC took only 10 first-place finishes out of 27 benchmarks yet still sits firmly at the top: its edge comes from its overall "rank distribution," not from a single standout performance.

The Result: A Model Trained in English Also Takes First on the Japanese Track

Darwin-27B-ZTC-v5 took first place on the Japanese track with a Borda score of 84.57, leading second-place Perplexity (80.86) by 3.71 points, with torchcast (80.49) and Jev 1.13 (75.43) close behind. In terms of average task score, Darwin scored 60.05, with second place at 58.43.

More crucial is coverage: across 27 benchmarks, it took 10 first-place finishes and 16 top-three finishes. This isn't a case of one standout item pulling up the total score — it's consistent strength across the board.

It's also the top performer on the English track (160 models), with a score of 90.80. In other words, the same model leads on both language tracks, despite never having received any Japanese-specific training.

Cross-Lingual Transfer: An Underrated Capability Signal

Darwin ZTC's training identity is an English decision engine, with no Japanese decision training set. But the abilities that make a model a good "judge" — spotting traps, moral judgment, probability calibration — transferred to Japanese almost entirely intact.

This has direct value for product teams deploying multilingual systems. Routing, scoring, and safety gating modules need to run in every language the product serves. If judgment capability can transfer across languages, the marginal cost of entering a new language market drops significantly, since there's no need to retrain a dedicated judge model for each language.

Conversely, a model with strong judgment isn't memorizing surface patterns of a language — it's learning a more abstract, more transferable skill. This may also explain why it holds its ground on both the English and Japanese tracks.

Cross-lingual transfer refers to a model's ability to apply capabilities learned in one language to another without additional training. It has been extensively studied in generative models (e.g., zero-shot cross-lingual question answering), but on "decision models" — which output judgments rather than text — public quantitative evidence has been lacking until now.

Whether transfer succeeds depends on whether the model has learned "surface language patterns" or the "underlying task structure." Surface patterns are tightly bound to specific vocabulary and grammar, failing the moment the language changes; but abilities like spotting logical traps, calibrating probabilities, and making moral trade-offs operate at a level of abstraction above language itself, making them easier to transfer. The S1MB Japanese track happens to provide exactly such a testing ground: its 27 benchmarks span knowledge, reasoning, cultural common sense, and other dimensions. If the transfer were merely coincidental, it would be hard to consistently rank near the top across such broad coverage.

Head-to-Head: Win-Loss Distribution Across 27 Benchmarks

Comparing Darwin ZTC against the top three on the Japanese track one by one, it wins on the majority of benchmarks: against Perplexity it's 15 wins, 9 losses, 3 ties; against torchcast, 15 wins, 11 losses, 1 tie; against Jev 1.13, 15 wins, 10 losses, 2 ties.

A few representative victories:

• JMMLU (Japanese knowledge exam): 88.4, tied for best.
• Common-sense moral judgment: 88.0, first.
• SOBACO (Japanese cultural common sense): 72.5, first.
• JaNLI (Japanese natural language inference with trap sentences): 83.7.
• JSICK semantic relatedness: 56.6.
• Operator-authored math and formal logic questions: first in all three question types.

Interestingly, benchmarks like SOBACO that involve native Japanese cultural common sense are theoretically among the least friendly to non-Japanese-trained models, yet it still took first. Of course, the roughly 12 losses also show this transfer isn't lossless — on some benchmarks it still loses to more specialized opponents.

JMMLU is the Japanese version of MMLU (Massive Multitask Language Understanding). MMLU consists of multiple-choice questions across many disciplines and is widely used as a benchmark for measuring a model's breadth of knowledge; JMMLU retains the same subject structure, but both questions and options are presented in Japanese, so it tests not only knowledge itself but also whether a model can correctly understand and answer in a Japanese context.

JaNLI corresponds to the NLI (Natural Language Inference) task: given a premise and a hypothesis, determine whether they are entailed, contradictory, or neutral. Such tasks are often designed with "trap sentences" — e.g., negation, quantifier ambiguity, or word-order changes — specifically to test whether a model truly understands semantics rather than relying on keyword matching.

SOBACO, meanwhile, focuses on native Japanese cultural common sense, such as festivals, customs, and social norms. For a non-Japanese-trained model, such knowledge is hard to acquire indirectly from general corpora, so taking first place on this benchmark demonstrates the depth of transfer more than winning on purely logic-based benchmarks.

Frequently Asked Questions

Was the model fine-tuned on Japanese decision data? No. It was trained as an English decision engine, and its Japanese results come from transfer capability.

How is the Borda score calculated? Each benchmark's ranking is converted into points and then averaged, so no single task can dominate the final result.

Where can I verify this? The public leaderboard lets you view the Japanese and English track data directly.

Why is dual-language leadership worth noting? Routing, scoring, and safety gating need to work in every language a product supports, and a judge model that transfers will lower the cost of multilingualization.

One-line Summary

English-trained decision model Darwin ZTC won the S1MB Japanese track without Japanese training, validating cross-lingual transfer capability.

Paragraph Summary

S1MB is a benchmark suite for decision models that converts rankings across multiple specialized benchmarks into averaged scores (Borda total) to prevent any single task from dominating. Its newly added Japanese track includes 27 benchmarks. Testing shows that the purely English-trained Darwin-27B-ZTC-v5 took first place on the Japanese track with a Borda score of 84.57, leading second-place Perplexity by 3.71 points; it achieved 10 first-place finishes and 16 top-three finishes across the 27 items, and also ranked first among 160 models on the English track. The article argues that "judge" abilities such as spotting traps, moral judgment, and probability calibration can transfer across languages, which means a significant drop in deployment costs for product teams needing multilingual routing, scoring, and safety gating.