Battle for the Top Open-Source Model: A Deep Audit of Two Leaderboards

Statistical Fog Beneath the Consensus at the Top

When two major leaderboards both rank MiMo-V2.6-Pro as the top open-weight model, the conclusion seems set in stone. But a line-by-line audit reveals it is far from simple. On October 19, both BenchLM and Artificial Analysis showed MiMo-V2.6-Pro at the top—the former with a score of 73.7, the latter with an intelligence index of 46.3. The problem lies in the details: BenchLM marks this row as Estimated and provides a conditional interval from 63.92 to 83.37. That interval spans 19.45 points, while rows marked as Supported in the same table have intervals only 8.2 to 10.0 points wide.

More critically, the intervals of the top nine models all overlap. All 36 pairs overlap without exception; even adding the tenth row, all 45 pairs overlap. BenchLM itself notes on the page that MiMo-V2.6-Pro "has a higher point estimate," but "the conditional interval does not establish ranking confidence."

So the more accurate statement is: on weak evidence, MiMo-V2.6-Pro has the highest point estimate among open-weight rows. That is much weaker than "best open-source model," but it is the only conclusion the page data supports. Note that conditional intervals and 90% confidence intervals are constructed differently, so directly comparing their widths is not entirely equivalent.

Conditional intervals vs. confidence intervals: A conditional interval is typically constructed after a model has already been selected as a "top candidate" or under some filtering condition; it does not cover the uncertainty of whether the model is truly better than others. A 90% confidence interval, by contrast, attempts to cover the true value of the parameter and accounts for the sampling distribution. Their widths are not directly comparable: conditional intervals tend to be narrower because they ignore the multiple-comparison problem introduced by model selection. In a leaderboard setting, even if two models' conditional intervals overlap, it does not mean they are not significantly different; conversely, non-overlapping conditional intervals do not directly imply high ranking confidence. When auditing, check whether the interval is constructed for the target of "ranking" rather than merely for "score."

After First Place, the Ordering Starts to Drift

Matching the top ten open-source rows from BenchLM with Artificial Analysis by model name yields nine successful matches ("dots3-note Preview" is missing from the latter's data). Ranking only these nine models separately gives a Kendall's tau of 0.61, with 29 concordant pairs and 7 discordant pairs. This means the overall trend is similar, but the middle section is not stable.

The most typical example is DeepSeek^Ad V4.1 Flash 1017, which ranks sixth on BenchLM but jumps to third on Artificial Analysis. If we use "top five" as the cutoff, Ling 3.1 Flash makes the cut on BenchLM but not on Artificial Analysis; DeepSeek V4.1 Flash 1017 is the opposite.

Artificial Analysis's own top is also quite tight: the top five open-weight models have indices of 46.3, 45.1, 44.8, 43.7, and 43.6, with adjacent gaps of only 1.2, 0.3, 1.0, and 0.1 index points—the entire top five spans just 2.7 points. Each row shows only a single number, making it impossible to tell whether a 0.1 gap is real.

Open-Source Labels Diverge More Than Rankings

More noteworthy than ranking noise is the membership problem. "Open weights" is a label each aggregator assigns on its own, and these labels contradict each other.

Kimi K3's absence. On BenchLM, Kimi K3 scores 69.92, ranking 18th overall (out of 226 models), yet it is missing from the open-source list because its weight access is marked as "pending" and there is "no verified weight release." By point estimate, it should rank second on the open-source list, slightly above Qwen3.8 Max's 69.7. However, on the Hugging Face Hub, the moonshotai/Kimi-K3 repository is public and ungated, containing 96 safetensors shards under a custom license called kimi-k3, with over 1 million downloads. Artificial Analysis marks the same model as isOpenWeights: true with an intelligence index of 43.6.

Qwen3.8 Max's contradiction. BenchLM ranks it as the second open-weight model, but Artificial Analysis marks it as isOpenWeights: false. Anonymous requests to Qwen/Qwen3.8-Max and Qwen/Qwen3.8-Max-0902 on the Hub API both return HTTP 401 (the Hub returns this code even for nonexistent repositories), so the repository's existence cannot be confirmed. The two leaderboards may point to different snapshots—which is itself a problem: identical display names without checkpoint identifiers are not stable keys.

MiMo-V2.6-Pro's checkpoint uncertainty. The Hub hosts XiaomiMiMo/MiMo-V2.6-Pro-RL (MIT license, public, 132 safetensors files), but a repository named MiMo-V2.6-Pro cannot be resolved anonymously. Neither leaderboard page states whether the evaluated row corresponds to the -RL checkpoint.

Hub API's 401 response: The Hugging Face Hub returns HTTP 401 (Unauthorized) for anonymous requests to nonexistent, private, or authentication-required repositories, rather than 404 (Not Found). This design avoids leaking whether a private repository exists, but it also means a 401 alone cannot distinguish "repository does not exist" from "repository exists but requires permission." To confirm existence, use an authenticated request with a token or cross-check via the Hub's search/model-list API. The 401 for Qwen/Qwen3.8-Max in this article only shows that anonymous resolution failed; it does not directly imply the repository does not exist. Likewise, the inability to anonymously resolve MiMo-V2.6-Pro does not mean its weights are unreleased. When auditing, distinguish "anonymously invisible" from "confirmed nonexistent."

Independent Evidence or Circular Verification?

BenchLM's methodology page mentions that BenchAlign, before combining, "calibrates benchmark protocols, human preference signals, and external composite indices." If those external composite indices include another leaderboard, then their agreement has a circular component, and the tau value of 0.61 would overstate the degree of independent confirmation. The page does not list the specific names of the external indices, so this cannot be determined.

An agreement that cannot be decomposed has weaker evidential strength than a disagreement that can be explained. The two leaderboards also have different input types: Artificial Analysis publishes an index based on a fixed evaluation set; BenchLM tracks 640 benchmarks, scores subsets across six pillars, excludes vendor-reported numbers from rankings, and fills gaps with learned estimates. Different inputs producing similar outputs is somewhat reassuring; but different inputs producing different open-source labels is a signal that you need to verify the labels yourself.

Kendall's tau and rank correlation: Kendall's tau measures agreement between two rankings, ranging from -1 (perfectly opposite) to 1 (perfectly concordant), with 0 meaning unrelated. It is computed from "concordant pairs" and "discordant pairs": if model A ranks ahead of B on both leaderboards, that is a concordant pair; if A is ahead on one leaderboard and B is ahead on the other, that is a discordant pair. tau = (concordant pairs - discordant pairs) / total pairs. With small samples, tau fluctuates substantially: for nine models there are 36 pairs, and tau=0.61 has a standard error of about 0.12, giving a 95% confidence interval of roughly 0.37 to 0.85—so the conclusion of "moderate-to-strong agreement" is not precise. Moreover, tau reflects only overall monotonic association and is insensitive to local swaps at the top or in the middle; interpret it together with the specific rows that move.

A Five-Step Audit Before Citing Rankings

1. Read the evidence label. A label of "Estimated" or "conditional interval" means the ranking rests on weak evidence—do not write "best" on that basis.
2. Compare intervals, not point values. Run an interval-overlap check on the top rows; if all overlap, state "no discernible difference."
3. Resolve the checkpoint. Find the exact Hub repository or API model ID the row points to. Display names like "Max" or "Pro" do not identify a checkpoint.
4. Verify the open-source label. Confirm the weight files exist, whether the repository is gated, and what the license actually permits. A custom license is not equivalent to MIT or Apache-2.0.
5. Cross-check another source. Compute the rank correlation, then examine specific disagreements—because the rows that move are the interesting ones.

Limitations of This Audit

• This is a single snapshot from October 19; both pages are continuously updated.
• Artificial Analysis's values come from row data embedded in its leaderboard page, not a documented API. Parsing that JSON is fragile, and no index version number is visible on the page.
• Name matching across sites was done manually; nine of BenchLM's ten rows were matched. The rankings in the table are positions within these nine rows, not rankings on the full list.
• Nine samples is small for rank correlation; tau=0.61 should be treated as a rough indicator.
• No models were run firsthand; all scores are as published by the aggregators.

FAQ

What is the best open-weight model in October 2026? It depends on the leaderboard and the definition of "open source." MiMo-V2.6-Pro has the highest point estimate on both leaderboards (BenchLM 73.7, Artificial Analysis 46.3), but BenchLM marks that row as Estimated, with an interval 19.45 points wide—so its lead over subsequent rows is not statistically established.

Why isn't Kimi K3 on BenchLM's open-source list? BenchLM records its weight access as pending and states there is no verified weight release, even though the Hugging Face repository is public. Artificial Analysis marks the same model as open weights.

What does "Estimated" mean on a leaderboard row? On BenchLM, it means the row relies on weaker evidence than "Supported" rows, which satisfy the site's independent-evidence rules. The interval shown for Estimated rows is a conditional interval, which BenchLM states cannot establish ranking confidence.

How can I check whether a model is truly open-weight? Resolve its exact repository on the Hugging Face Hub, confirm the weight files exist and the repository is ungated, and read the license text. The hub_status function above can automate the first three checks.

One-line Summary

Both leaderboards rank MiMo-V2.6-Pro first among open-source models, but weak statistical evidence, ranking drift, and contradictory open-source labels mean a five-step audit is needed before citing.

Paragraph Summary

The article audits the October 19 rankings of open-weight models from BenchLM and Artificial Analysis. Although both rank MiMo-V2.6-Pro first, BenchLM marks that row as Estimated with a conditional interval 19.45 points wide, and the top nine intervals all overlap—ranking confidence is not established. Kendall's tau for the nine matched models is only 0.61, the middle rankings are unstable, and DeepSeek V4.1 Flash 1017 rises from sixth to third. More seriously, the open-source labels diverge: Kimi K3 is public on Hugging Face yet excluded by BenchLM, Qwen3.8 Max is marked open by one leaderboard and closed by the other, and MiMo-V2.6-Pro's checkpoint correspondence is unclear. The article also notes that agreement between leaderboards may involve circular verification through shared external indices, and offers a five-step audit before citing rankings.