[KongchangAI]
· 3 min read· 1,603 words

Alibaba's Qwen 4 Beta Beats Claude Opus 5.5 at One-Sixth the Cost

Alibaba's Qwen 4 Beta Beats Claude Opus 5.5 at One-Sixth the Cost

Unreleased Qwen 4 Flash beats Opus 5.5 on creativity in a private game-dev test, but credibility and weights remain unverified.

An early unreleased version of Tongyi Qianwen Qwen 4 outperformed Claude Opus 5.5 on the independent Game Field Bench test, but the edge comes not from coding—the two tie on pure code—but from the "creativity" dimension of inventing game content (54 vs. 41). The test covers 12 never-released real game projects with roughly 60,000 tasks per model, mostly human-scored. Costs show Qwen at $0.40 per run versus Opus at $2.60, but the pricing depends on an unexplained paid channel. The 27B-parameter small version scores about 45, slightly above Opus's medium setting, but is slow and its weights are unreleased. Later, Gemini 4 Carbon and Fable 5.2 overtook on the "reverse evaluation" dimension—the very area where Opus served as judge.

An unreleased model leads in game development testing

According to analysis by a YouTube channel, an early unreleased copy of Tongyi Qianwen Qwen 4 outperformed Claude Opus 5.5 on a test called Game Field Bench. Run by an independent tester, the model added new items, mechanics, and bug fixes across 12 never-before-released real game projects—games that no model had ever seen during training.

The leaderboard lists the model as Qwen 4 Flash, which held the top spot for 44 hours before being surpassed by two new models, one of them from Claude. The result sparked community discussion about whether unreleased models are worth paying attention to—the "Qwen 4" label used by the tester is not an official Alibaba name; the codename shown on the leaderboard is "Codename Hawk."

The tester describes that the model must complete 30 to 100 modifications

The nature of the gap: creativity, not code

The key to Qwen 4 Flash's lead over Opus 5.5 is not coding ability. Game Field Bench's total score is the average of seven dimensions, and the two are evenly matched on the pure programming portion. The real gap shows up in the "creativity" dimension: when asked to invent new game content on its own, Qwen scored 54 and Opus scored 41, with these ideas scored manually item by item on a 1-to-10 scale.

Qwen leads slightly on visual presentation and has an even clearer advantage in sound effects. This means that if you use Opus to generate creative content, the gap is real; but if you only use it to fix code, at least in this test there is no difference.

One detail worth noting: Opus 5.5 itself served as a judge for three parts of the test—writing mechanics, fixing bugs, and "reverse evaluation" (where the model scores other AIs' modifications and real developers' update notes). Even so, Opus led Qwen by at most two points across these three parts. If Opus were favoring its own answers, the lead should have been larger.

The test's credibility and limitations

Game Field Bench's design reduces room for "leaderboard gaming": the 12 private projects cover three each of roguelike, online RPG, single-player RPG, and survival games, and models must dig into the projects themselves, play the games, or write bots to discover what content is missing. Each model must complete roughly 60,000 tasks, most scored by humans—a considerable investment of manual labor for an independent test.

But the tester also leaves information undisclosed: who the raters are, how an unreleased model was run through a paid connection that Alibaba does not sell publicly, and how Opus 5.5's cost of $2.60 per run was converted from a Claude subscription. Qwen's $0.40 bill is likewise based on that unexplained connection. So "one-sixth the cost" should be understood as the tester's per-run bill, not the actual price users would pay.

The comparison data is worth following closely

Reverse Evaluation is a test design in which the model being evaluated acts as a judge: the model must score modifications generated by other AIs as well as update notes submitted by real developers. This approach is often used to supplement human scoring at a scale it cannot cover, but it also introduces a clear risk of bias—a model may favor outputs similar to its own style or give higher scores to proposals from similar models. In this article, Opus 5.5 serving simultaneously as a contestant and as a judge for three parts is a typical example of this design controversy. Notably, the later Gemini 4 Carbon and Fable 5.2 overtook the field precisely on this dimension, with scores 2 to 4 times higher, showing that this dimension's weighting and scoring mechanism heavily influence the final ranking. Readers interpreting total scores should distinguish "reverse evaluation" from "actual production capability."

The small version deserves more attention

For users who want to run models locally, the small Qwen 4 on the leaderboard is more meaningful. This version has 27 billion parameters, a scale that can run on home hardware. It scores about 45, slightly above Opus 5.5's medium setting and about 3 points below the large model mentioned at the start. For this parameter scale, that is a strong result.

But it is slow: each task takes about 3 hours, twice the runtime of that Opus run. Every fifth of the work, its working memory fills up and it must summarize what it has completed before continuing, and these pauses add to the time cost.

More interestingly, the small model once rose to 47 points, just 0.2 points behind Opus. Then the tester pushed a fix labeled "Audio Corrected," and its sound score dropped by more than a third, bringing the total back down to its current level. The same fix pushed the large Qwen 4 Flash to the top of the leaderboard. A single fix shifting scores by more than two points within five hours suggests that fresh scores on this test are more like a first draft than a final conclusion.

The outside world cannot verify before model weights are released

Working memory filling up refers to the phenomenon in which a model's context window gradually fills during long tasks. When a model performs dozens to hundreds of consecutive modifications, the code, error messages, and operation history from each step accumulate in the context; once it approaches the window limit, the model either forgets earlier content or must trigger summarization and compression. The small Qwen 4 in this article needing to summarize completed work after roughly every fifth of the task is exactly this mechanism at work. Compressed summaries lose detail, causing later modifications to conflict with earlier decisions, and each pause adds real elapsed time—which explains why a single task takes about 3 hours. Context management capability thus becomes a key bottleneck for small local models performing long-cycle development tasks, not just a matter of parameter scale or inference speed.

Open weights and official release still pending

The small Qwen 4 cannot currently be downloaded because its files are not yet public. The leaderboard lists the Qwen 4 version as 35GB but does not explain the compression method. Until Alibaba releases the weight files, the outside world cannot independently verify it. For reference, the currently downloadable Qwen 3.8 Flash Next—which Alibaba calls an experimental preview that forms the design basis of Qwen 4—scores about 40 on the same test, on par with DeepSeekAd V4.1 Flash, at a cost of two cents per run, but at 128GB it is workstation-class.

Alibaba's official disclosures about Qwen 4 are extremely limited: in September it said Qwen 4 was in training; on October 29 an attendee reported that Alibaba Cloud's technical lead mentioned "Qwen 4 releases next week," but that week came and went without it. A day earlier, Qwen's own coding tool reserved a slot for Qwen 4 with support for a million-token context, but the update notes said it did not actually call the real Qwen 4 for testing. The official price list also has no price for Qwen 4.

On the poster released afterward, the missing name was precisely Qwen, with the caption "The becoming begins" referencing an ancient story of a fish transforming into a giant bird—but with no date and no specs.

A single demo shows token usage halved

Another headline—that Qwen 4 uses only half the tokens DeepSeek needs on web tasks—comes from a single demo by a single creator, and no token counts were shown.

Open weights means that a model's parameter files are publicly released after training, so anyone can download them, run them locally, or even fine-tune them, as opposed to the closed-source model of offering only API access. The significance of open weights lies in reproducibility and verifiability: the outside world can independently run the same model and check leaderboard scores without relying on a vendor's or a single tester's claims. This article's emphasis that the small Qwen 4's 35GB file is not yet public and its compression method unexplained points precisely to this verification gap. By contrast, Qwen 3.8 Flash Next has been called by Alibaba an experimental preview forming the design basis of Qwen 4 and is downloadable, so although it scores lower, it is currently the only reference point that third parties can independently verify. Until the weights are released, all rankings concerning Qwen 4 can only be treated as signals awaiting verification.

The leaderboard shifts again within days

On Monday evening, Google's Gemini 4 Carbon and Claude's Fable 5.2 both surpassed Qwen 4 Flash. Gemini scored 57.5 and Fable about 54. But they overtook on the "reverse evaluation" item—where their scores were 2 to 4 times those of other models, with Gemini highest. In other words, their lead comes from judging other AIs' game modifications, not from making better games. On the remaining six dimensions, Qwen 4 Flash is on par with Fable and slightly ahead of Gemini.

For practical choices: if you pay to have Opus write code, Fable 5.2 Medium is the better choice—slightly better at coding than both, available now, and about one-third of Opus's cost on the tester's bill. If you need creativity, Qwen 4 Flash is still the strongest and has the lowest test cost among all models that beat Opus, except you cannot buy it yet. If you want an open-source model you can run yourself, Qwen 3.8 Flash Next is currently the closest option, runnable on a single large workstation, about 7 points below Opus.

Alibaba's official data will eventually be released, and comparing it with this leaderboard on the same day will be more valuable than any rumor.

#Qwen 4#Claude Opus 5.5#Game Field Bench#Alibaba Qwen#AI model benchmark#open weights
Share:

Related articles