Open-Source Models Outperform Closed-Source Flagships? GPT-5.6 Faces Open-Source Challenge

Recently, a discussion about comparing the capabilities of open-source and closed-source large models has been quietly heating up in developer communities. A Bilibili creator posted a video raising a sharp question: how did the closed-source flagship GPT-5.6 get pushed behind on a leaderboard that an open-source model posted itself? This episode of "GitHub Open-Source Project Analysis" revolves around three things: Meta's open-source model, DeepSeek's version update, and Beam's long-context roadmap. Let's take a look at these discussions to see just how far open-source models have come.

Meta's Open-Source Model: From "Assistant" to "Taking Over the Project"

According to the video, Meta open-sourced a model called "8161" (refer to official information for the exact model name), and the official demo was impressive. In the demo, the model can find bugs on its own, add features, and complete an entire piece of software from scratch while humans just watch from the sidelines. Even more striking, it can take over editing software, review footage frame by frame, pick shots and trim them itself, and directly produce a finished video. This kind of end-to-end automation "from requirement to finished product" has gone beyond the scope of traditional code completion.

However, the video also points out that this is not an entirely new model but an update. Its benchmark targets are closed-source flagships like Opus and GPT-5.6. In the evaluation data on the model's own card, it beat these two competitors with a score of 77.0. But the key issue is: this score is a "table the model posted itself," and there is currently no third-party reproduction. It's like an athlete keeping score themselves—it may not necessarily be fake, but its credibility inevitably takes a hit.

Benchmark self-evaluation and third-party reproduction: In model evaluation, benchmark scores released by vendors themselves are often called "self-evaluation" or "self-reported results." Because details such as test sets, prompts, and sampling parameters may not be fully disclosed, other teams find it difficult to precisely reproduce the same results, so the credibility of self-reported scores is limited. The industry gives more recognition to independent third-party evaluations (such as LMSYS Chatbot Arena and Open LLM Leaderboard) or reproducible academic benchmarks. The "table the model posted itself" mentioned in the article refers precisely to this kind of evaluation method lacking external verification; its scores can only serve as a reference and should not be used directly for cross-model comparisons.

DeepSeek Update: Scores Rose, but the Exam Paper Changed Too

The second focus is DeepSeek^Ad. The video mentions that DeepSeek brought V4.1 Flash up to the 2017 version (this refers to the version; please refer to official sources for the specific number) and debuted the Reflection feature. In another comparative test, someone used the same prompt to have four models each build a small 3D game, and only DeepSeek succeeded in creating a scene with thousands of glowing windows. The official cases are more specific, including 3D modeling of the Forbidden City and a web-based racing game, both built from the component level.

But the video also cautions that scores cannot be fully trusted. DeepSeek's score on one item jumped from 31.2 to 42.3, and the "exam paper" was changed in between, so the two exams cannot be directly compared. This actually points out a common dilemma in current model evaluation: benchmarks are updated frequently, and score differences between different versions are hard to attribute entirely to improvements in model capability. For ordinary developers, the reference value is limited, and more attention should be paid to how models perform in actual tasks.

Benchmark version changes and score comparability: Benchmarks are not static. As questions leak, models overfit, or task difficulty is adjusted, test sets often get updated versions. Once the version changes, old and new scores are no longer directly comparable because the "exam paper" itself has changed. If the test set was changed in between, the score jump from 31.2 to 42.3 mentioned in the article cannot simply be attributed to improved model capability. When referring to such data, developers should confirm whether the test version is consistent and give priority to horizontal comparisons under the same version or actual task performance.

Beam's Roadmap: Long Context and the Reality of Local Deployment

The third noteworthy item is Beam. The video says it takes a different path: it can read an entire codebase at once, handle long tasks without splitting them up, and without repeatedly explaining the background. For scenarios that require understanding large codebases, this is indeed a practical feature. But the reality is that the only one that can fit on an ordinary computer is Nemotron Bolt (24GB VRAM), and Q&A recommends starting at 48GB. Configurations such as Meta 8x1G 60E, Beam 5010E, and DeepSeek 7630E all require higher-end hardware. Meta even needs two top-spec Mac Studios, each with 512GB of memory and 1TB of storage, while the other two also require multi-GPU parallel execution.

This means that although open-source models are challenging closed-source flagships in terms of capability, the barrier remains. For most individual developers, the only one they can truly get hands-on with is a relatively lightweight model like Bolt; the other three are more like preparations for teams with machines and projects.

Long context and VRAM constraints in local deployment: Long-context models need to keep the key-value cache (KV Cache) for a large number of tokens in VRAM, and the longer the context, the greater the VRAM usage. Therefore, "reading an entire codebase at once" as mentioned in the article places higher demands on hardware. 24GB of VRAM can only support shorter contexts or quantized lightweight models; to process a complete codebase, 48GB or even more VRAM is usually required, or multi-GPU parallelism and model parallelism techniques must be adopted. This is also why only relatively lightweight models like Nemotron Bolt can run on ordinary computers, while larger configurations require top-spec workstations or servers.

Open Source vs. Closed Source: The Gap Is Narrowing, but the Evaluation System Needs Improvement

Taken together, the information in the video can be summarized as follows: this week's change is that models are moving from the "directional protocol" line (which may refer here to the traditional API calling model) to "take on a project and do it themselves." Open-source models are indeed approaching closed-source flagships in capabilities such as automation and long context, but the barrier is still there—the door has only opened a crack.

One additional point: some statements in the video are relatively colloquial and involve specific version numbers and scores, so readers should verify official information when referring to them. In addition, the leaderboard data posted by the models themselves lacks third-party verification, so comparisons should be made cautiously. The competitive landscape between open source and closed source is still evolving rapidly, and the evaluation system also needs more transparent and reproducible benchmarks to support it.

For developers, the focus might be on: which capabilities are truly usable, and which configurations are realistically accessible. After all, no matter how high the score is, if it can't run, it's just a number on paper.

One-line Summary

Open-source model capabilities are approaching closed-source flagships, but self-reported scores and hardware barriers remain controversial.

Paragraph Summary

This article reviews three recent developments in open-source large models: Meta's open-source model demonstrated project-level automation from bug-finding to producing a finished video, with self-reported scores surpassing closed-source flagships but lacking third-party reproduction; DeepSeek's update stands out in 3D generation, but its score jump involves a benchmark change and should not be directly compared; Beam focuses on long context and can read an entire codebase at once, but local deployment has high hardware requirements, and ordinary developers can only try lightweight versions. The article points out that open-source models are shifting from assistant tools to project-level automation, and the capability gap is narrowing, but the evaluation system is opaque and hardware barriers remain, so developers should pay more attention to actual usability and accessible configurations.