Qwen 4 Max Preview: The Truth Behind the Benchmark Controversy and a Developer's Guide

Alibaba's Qwen 4 Max Preview has no official benchmarks, sparking conflicting third-party evaluations that expose flaws in judging AI models.
Alibaba released Qwen 4 Max Preview with a 1M token context and tool calling, but no official benchmark scores, and the preview expires November 25. Third-party results conflict: Model Dial scores 78.8/100, ranking 14th, with knowledge ~90, coding ~80, but code quality only 59; GameField claims it's worse than older Qwen. Part of the divergence stems from daily-updated test builds like Qoduck, meaning evaluators may test different models. The article also gleans MoE architecture clues from Qwen 3.8 Flash Next (125B total parameters, 6B activated per word) and notes Max lacks open weights. It advises developers to test with real problems, connect via standard interfaces for easy switching, and avoid product dependencies for now.
Alibaba has released Qwen 4 Max Preview, its largest AI model ever, but the company published no benchmark scores, while third-party evaluations diverge wildly—some call it excellent, others say it's worse than older versions. This controversy reveals deeper problems in how we judge AI models.
Controversy at Launch: A Model Without a Report Card
On November 11, Alibaba launched Qwen 4 Max Preview. Every word in the name deserves scrutiny: Qwen 4 is the family, Max is the largest member, and Preview means it's only a trial version. It supports text, image, and video understanding, can think before answering, and can call tools to perform actions rather than just converse.
The most eye-catching number is the 1 million token context window—enough to hold a small library in a single conversation. But there's a key limitation: this preview has an expiration date, ending November 25, when the model will be retired. Alibaba promises a formal release afterward, making this essentially a 15-day test drive.

The question follows: how good is this car? Usually, a launch comes with a benchmark report card, but this time Alibaba's report card is blank—no benchmark tables, no scores, only a promise of "deeper knowledge and stronger coding." A promise is not a measurement.
Context Window refers to the maximum number of tokens a model can process in a single pass, including input and output. A 1 million token context window means the model can read approximately 750,000 English words or a large codebase, long document, or multi-turn conversation history at once. The challenge of expanding context windows lies in the computational and memory costs of attention mechanisms growing quadratically with length, as well as the model's ability to retain key information over long contexts (i.e., "effective context"). In practice, models may not utilize all information within the window evenly, so long-context capabilities require dedicated evaluation.
Third-Party Evaluations: Same Model, Drastically Different Conclusions
The outside world began testing on its own. Independent evaluator Model Dial ran private coding and knowledge tests, scoring the preview at 78.8/100, ranking 14th among 54 models. That's a good score, but not dominant.
More interesting is the score pattern: knowledge tasks around 90, coding around 80, but code quality and polish only 59. This means it's a strong thinker, but its code may be rough.
Then came a confusing twist. GameField, a benchmark testing models' ability to build games, reported that the preview scored lower than earlier Qwen models, calling the whole situation "confusing." Meanwhile, other developers posted web demos they loved. Same model, same week, completely opposite conclusions.

How can both be true? Because each test measures something different. It's like judging a new restaurant: one critic tastes the food, another evaluates the decor, a third only counts serving speed. All three are honest, but they're not measuring the same thing—different skills, different settings, small samples.
The "Duck" Incident: Test Ambiguity
There's an even stranger reason for divergent results, involving a "duck." Days before launch, a coding tool called QDAR offered a mysterious model named Qoduck, with no maker information, no details. People speculated it was Qwen 4. On launch day, Qwen confirmed it: Qoduck was a test build of Qwen, and this build was updated daily.
Imagine two people reviewing the same app: one uses Monday's version, the other Wednesday's. They might not be evaluating the same thing at all. This doesn't prove why GameField scored low, but no one outside Alibaba knows whether everyone tested the same model.
Under the Hood: Glimpsing the Architecture from Flash Next
The scale or design of Qwen 4 Max can't be directly seen—Alibaba didn't share it. But there's a clue: Qwen 3.8 Flash Next, a smaller model released in August, was officially described as previewing the design Qwen 4 would adopt—a Mixture of Experts (MoE) architecture.

This architecture is like a team of experts hidden inside a single model: total parameters around 125 billion, but only about 6 billion activated per word. Like a hospital with hundreds of doctors where you only need to see two. This keeps the model fast and cheaper to run. It's the family recipe, not the final product—Max's actual configuration remains unknown.
Mixture of Experts (MoE) is a technique that activates only part of a model's parameters during inference. Traditional dense models call all parameters for every input, while MoE splits the model into multiple "expert" subnetworks, with a routing network deciding which experts each word or token should be sent to. This allows total parameter count to be very large while actual computation scales only with the number of activated experts. For example, Qwen 3.8 Flash Next has about 125 billion total parameters but activates only about 6 billion per word, greatly reducing inference costs while maintaining capability. MoE's challenges lie in routing balance and training stability, but it has become one of the mainstream directions for scaling current large models.
Questions from the Open-Source World: Where's the Downloadable Model?
In September, Alibaba announced four tiers for Qwen 4, including Max, Plus, Flash, and a 27-billion-parameter open-source model. So far, only Max exists, and only in preview form. No open weights—no downloadable model files.
Having weights means you can run the model on your own machine, customize it, and keep it forever. A subscription-based preview can't do that. There's also no simple pay-per-use pricing plan.
Open Weights refers to publicly releasing the parameter files after a model is trained, so anyone can download, run locally, fine-tune, or even redistribute it. This contrasts with closed-source models that offer only API access. The value of open weights lies in keeping user data local, enabling domain-specific customization, and protecting against service shutdowns or price increases. But open weights do not equal open source, because training data, code, and training processes are often not disclosed. For the Qwen 4 series, only Max is currently available as a preview API, with no open weights, meaning developers cannot truly own or control the model.
What Developers Should Do Right Now

For designers or infrastructure developers, the advice is straightforward: try it out, but throw a real problem at it, not a demo. Don't build product dependencies—scores are missing, pricing is vague, and the model expires in a few days.
A smart approach is to connect to the model through a standard interface, so switching later is just a one-line code change.
Back to the original question: is Qwen 4 Max excellent or worse? No one can honestly answer right now. And that's the real lesson: benchmarks are just one test, demos are just one moment, and neither can show how it will handle your actual work. The better model is the one that reliably solves your real problems, every time.
Related articles

How the Grok Team Hits 11,000 PRs a Month: Replacing Manual Review with Deterministic Constraints
Inside the Grok team's formula for 11,000 monthly PRs at a 0.5% rollback rate: Lint rules, type systems, and formal verification as deterministic constraints.

Is Spacetime Made of Something? Six Possible Answers from Quantum Gravity
Is spacetime made of something? This article surveys the main candidate answers from quantum gravity: spacetime may be fundamental, geometry quantized, causal relations prior to distance, spacetime an emergent state of matter, geometry derived from string theory, or no spacetime atoms needed at all.

After GTA 6, Where Does the Game Industry Go?
The Vergecast talks with Game File's Stephen Totilo about GTA 6's impact on gaming: billion-dollar budgets, console market struggles, cloud gaming tests, and the future of microtransactions—and whether it marks the end of an era.