Chinese Open-Source Models Trail US Closed-Source by Only 7 Months: The AI Race Landscape Is Shifting

The Truth About the AI Race, Revealed by an Open-Source Leaderboard

While people are still debating whether the AI gap between China and the US is two years or five years, an open-source leaderboard built by an independent developer offers a surprising answer: 7 months.

The leaderboard, called whatmodel.sh, comes from developer Jake Manger, who shared the finding on Reddit. Its data foundation is solid—22,113 benchmark results collected from 1,011 sources, covering 987 benchmarks. The text model section alone contains 18,755 results across 617 benchmarks for 1,211 models.

More critically, the data sources are distinguished: 14,436 results come from independent evaluations (ACL Anthology, arXiv papers, Epoch AI, LMArena, EvalPlus, EleutherAI harness, etc.), while 7,677 come from lab self-reports. This level of transparency and traceability makes the conclusions more convincing than typical leaderboards.

Open-weight models and open-source models are two concepts often confused in the AI field. Open-weight models refer to models whose training parameters (weights) are publicly downloadable—users can deploy, fine-tune, or even distill them on their own—but the training data, training code, and full training pipeline are usually not disclosed. Fully open-source models (such as some early versions of BLOOM) release both data and code. Most models released by Chinese companies like DeepSeek^Ad and Qwen fall into the former category. This "semi-open" model allows the community to rapidly innovate on top of them, but it also means outsiders cannot fully reproduce the model's capabilities. Understanding this distinction helps more accurately assess what "open-source models catching up to closed-source" really means—what's being caught up is the capability frontier, not full transparency and reproducibility.

What a 7-Month Gap Means

This number is worth noting because it points to a trend: open-weight models are rapidly approaching the capability frontier of closed-source models.

Chinese companies like DeepSeek, Alibaba's Qwen, and Zhipu continue to release open-weight models, while flagship models from top US labs (OpenAI, Anthropic, Google) remain largely closed-source. A 7-month gap, in an era where AI capabilities iterate on a weekly basis, is nearly equivalent to one version generation apart.

From a practical application standpoint, this means that if enterprises choose to deploy Chinese open-source models, their capabilities are roughly equivalent to the best closed-source models from about six months ago. For a large number of non-frontier scenarios—customer service, content generation, code assistance—this gap may already be small enough not to affect real-world results.

You may not have noticed that the leaderboard strictly distinguishes "independent evaluations" from "lab self-reports." Independent evaluations account for about 77%, which grounds the "7 months" conclusion in a relatively neutral basis rather than relying on vendors' own benchmark selections.

Benchmarks are standardized test sets for measuring AI model capabilities—for example, MMLU tests multidisciplinary knowledge, HumanEval tests code generation, and GSM8K tests mathematical reasoning. Each benchmark typically contains hundreds to tens of thousands of questions, and higher scores indicate stronger capability in that dimension. But benchmarks have clear limitations: questions may be contaminated by model training data, leading to "leaderboard gaming"; different organizations may selectively report favorable benchmark results. whatmodel.sh aggregates 22,113 results across 987 benchmarks and mitigates this problem by distinguishing independent evaluations from lab self-reports. However, there is still a gap between benchmark scores and real-world application experience, especially in complex scenarios like multi-turn dialogue and tool calling, where a single score cannot fully reflect a model's actual performance.

Methodological Insights from an Open-Source AI Leaderboard

The value of whatmodel.sh lies not just in its conclusions, but in the verifiable approach to AI evaluation it demonstrates.

The dataset uses a CC BY-SA 4.0 license, the scoring code uses an MIT license, and everything is open-sourced on GitHub. Every number can be traced back to its original source, and every source's license is recorded. This approach directly addresses the current trust crisis in AI evaluation—when vendors can selectively report benchmarks, the value of independent third-party aggregation becomes clear.

The interactive version allows users to click on any model to view its scores and sources, and this transparency is practically valuable for researchers and enterprise technology selection teams. The developer also appeals on GitHub for users to star the project if they find it useful.

CC BY-SA 4.0 is a type of Creative Commons license that requires users to attribute and share derivative works under the same terms, suitable for datasets and other creative content. The MIT license is an extremely permissive software license that allows users to freely use, modify, and distribute code, requiring only that the copyright notice be retained. Together, they mean that whatmodel.sh's data can be freely used and redistributed by researchers, but must remain open-source; the code has virtually no usage restrictions. This licensing strategy is especially important in the AI evaluation field because it lowers the barrier for third-party verification and secondary analysis, helping to build community trust. In contrast, many vendors' evaluation reports do not disclose raw data, making conclusions difficult to verify.

Potential Impact on the AI Industry Landscape

If the 7-month gap is real and continues to narrow, its impact on the industry is multi-layered.

For the open-source community, this is a signal that sustained investment pays off. After the Llama series, Chinese open-source models have become an important force in the global open-source AI ecosystem. For enterprise users, the customizability and cost advantages of open-source models become more attractive when the gap narrows to under six months.

For closed-source vendors, the commercial moat created by a technology generation gap is narrowing. When open-source models trail by only 7 months, closed-source models must offer something beyond raw capability—better toolchains, security guarantees, enterprise-grade support—to maintain a premium.

Of course, a single leaderboard cannot represent everything. Gaps may vary greatly across different task types and evaluation dimensions. But this data at least provides a new perspective: the landscape of the AI race is more dynamic and tighter than many people imagine.

The full data and interactive charts can be viewed at whatmodel.sh, and the project source code is open-sourced on GitHub.

Version generation gap in AI typically refers to the iteration cycle in which model capabilities make a significant leap. Taking OpenAI's GPT series as an example, the interval from GPT-3 to GPT-4 was about two and a half years, but it went through multiple smaller iterations in between. When we say open-source models trail closed-source models by "7 months," it actually means that the current best capability of open-source models is roughly equivalent to the level of closed-source models released 7 months ago. Considering that top labs typically launch major upgrades every 6-12 months, a 7-month gap means open-source models may be only one major version behind. However, this catch-up is not linear: closed-source models may suddenly widen the gap through advances in reasoning, multimodality, and safety alignment, while the open-source community relies on global collaboration and rapid fine-tuning to narrow it. Therefore, 7 months is a dynamic snapshot, not a fixed endpoint.