[KongchangAI]
· 2 min read· 1,454 words

Qwen 4 Max Launches With Zero Benchmarks: What Did Testers Find?

Qwen 4 Max Launches With Zero Benchmarks: What Did Testers Find?

Qwen 4 Max's zero-benchmark preview launch reveals a split profile: strong visuals, shaky code.

Alibaba's Qwen 4 Max launched November 11 as a preview with no benchmarks or model card, available only through Alibaba Cloud's subscription token packages with million-token context and multimodal input. Community testers pieced together a capability profile: developer AI Joey produced a CPU design that passed checks from a single prompt in five hours, but it had 19 safety violations and can't be manufactured; one-shot visual builds drew high praise, with token efficiency about a quarter of DeepSeek's; but code-fix testing scored only 49 versus 75 for a competitor, and reproducibility was shaky. The anonymous model CoDuck was confirmed as a test version. The preview retires November 25—suitable for low-cost frontend and prototyping experiments, but wait for the official release for production-grade agent tasks.

Alibaba's Qwen 4 Max officially launched on November 11, but the company published no benchmark results, no model card—not even a single score. That unusual silence pushed community testers to roll up their sleeves over the past two days, trying to piece together a real picture of the model's capabilities through actual tasks. Some found pleasant surprises, others found disappointment—and one detail about CPU design directly changed how we interpret all the test results.

What Launched: Preview Version, Million-Token Context, Subscription Access

The official announcement promised stronger coding and agent capabilities, and has already racked up over a million views. But what's actually available right now is just a preview: a one-million-token context window, support for text, images, and video, with reasoning enabled by default at the highest setting.

This version can't be downloaded, and there's no pay-per-token API. The only entry point is Alibaba Cloud's token package, billed monthly. Pricing is considerably lower than expected—the lightweight tier is $6 per month, with a $3 promotional rate for new users through November 30, including 20,000 credits. According to NeuroHive, during the preview period Qwen 4 Max consumes credits at as little as one-fifth the normal rate, dropping to under one-tenth during Beijing nighttime hours.

There's another easily overlooked clue: if you used Alibaba's coding app Coder last week, you may have already seen it. On November 7, an anonymous model called CoDuck appeared in Coder. NeuroHive noticed its token package and tokenizer matched Qwen's, and it vanished the instant the preview went live. Coder later confirmed: during those four anonymous days, Duck was Qwen's test version.

An anonymous model called CoDuck appeared there on November 7th.

What a million-token context actually means

A million tokens is roughly equivalent to 700,000 to 800,000 English words, or several full-length novels—enough to fit the core files of a mid-sized code repository in one go. The core value of long context is letting the model maintain a global reference to large amounts of material within a single inference pass: understanding cross-file dependencies, or reading an entire design document before generating code. But long context doesn't equal "effective use of long context": a model's ability to retrieve and reason over key information in extremely long inputs (often called the "needle in a haystack" test) tends to degrade as length increases, and token consumption and cost rise in tandem. The token efficiency advantages testers observed here need to be understood in the context of specific task types—they can't simply be equated with strong long-context capability.

Five-Hour CPU Design: Real but Unmanufacturable

The most eye-catching test came from developer AI Joey. The day before the public preview, he gave an early Qwen 4 checkpoint a single prompt and then walked away. Five hours later, the model delivered a CPU design—it wrote the design itself, ran tests, proved correctness, and produced the file formats a chip foundry would actually request.

Going further, he paired Qwen 4 with another model, SWE2 Flash, and after 10.6 hours of collaboration got a complete circuit board design: a four-layer board with 144 LEDs, with Qwen handling design and the other model handling routing.

But most headlines skipped a key line from the same post: it can't be manufactured yet—there are 19 safety violations, no pin definitions, no memory. So it's a real design that passed real checks, but it can't go to the factory tomorrow. AI Joey's own framing was restrained—"just trying to push the model."

He calls it what it is, just trying to push the model.

What a CPU design deliverable actually is

The article mentions the model "produced the file formats a chip foundry would actually request," which typically means GDSII (layout data) or the netlists and timing constraints used for tape-out. A complete CPU design flow includes: RTL code (describing logic in Verilog/VHDL), logic synthesis, place and route, timing analysis, and DRC/LVS physical verification. What AI Joey meant by "passed real checks" most likely refers to the model independently running simulations and formal verification to prove logical functional correctness. But "19 safety violations" and "no pin definitions, no memory" mean the design is far from closed at the physical implementation and interface levels—pin definitions determine how the chip connects to external circuits, and the memory subsystem is a prerequisite for a CPU to run programs. So this is a prototype that's verifiable at the functional level but unmanufacturable at the engineering level, with substantial manual work remaining before tape-out.

One-Shot Visual Builds: Where It Shines

Tester Ajay runs the same set of one-shot prompts against every new model—no tools, no retries. He asked Qwen 4 Max for a V8 engine cutaway and got moving pistons, firing order, and a control panel he never asked for. What surprised him most was that the model added engine sound effects in the code—the whole build, including thinking, consumed about 85,000 tokens.

His highest-rated prompt, Lighthouse Island, got the best result of any model, with a mechanical hummingbird nailed in one shot for under 56,000 tokens. Another tester, Lokdar, compared Qwen and DeepSeekAd using the same test framework, and Qwen's token usage was about a quarter of the latter's. He also had Qwen build the most interactive website for a hardware synthesizer, running until the model declared itself done.

Qwen also takes criticism well. In a tumor simulator task, AI Joey had Opus review the code and Qwen fix it. Three rounds later, the code had a built-in self-test module with 18 checks.

Three rounds later, it had a built-in self-test with 18 checks.

Divergence in a Benchmark Vacuum

The official zero-benchmark approach has stirred community suspicion. When Alibaba released Qwen 3.8 Max in September, it came with a trophy for first place on the Code Arena web development leaderboard; this time, silence. Lumina Bench asked directly: "No benchmarks—I'd really like to know why."

Independent test results are inconsistent. Lumina Bench's own 3D helicopter task was rated "not bad, but not yet at the level of current models." Chinese tester NFT Chen compared code-fix performance against Kimi K3.1 Pro on a 100-point scale: Kimi scored 75, Qwen 4 Max scored 49. To be fair, this was a single-person test, and the same tester also posted an animation of a pelican riding a bicycle—wheels rotating fully, pelican sitting steady.

This aligns with Lokdar's conclusion after four days of head-to-head comparison: Qwen 4 looks prettier when it delivers, provided it delivers. One prompt Qwen tried three times, failing the first two, while DeepSeek passed on the first try. In his crab test, Qwen won on animation and token cost, DeepSeek won on sand and lighting. There's also a claim shared by Kiminismus and explicitly labeled as rumor—"Qwen 4 is disappointing." Unverified, but in the absence of official numbers, rumors fill the vacuum.

The overall shape is roughly this: one-shot visual builds and token cost look strong; code fixes and result reproducibility look shaky. That's probably exactly what the word "preview" means.

Why benchmarks matter, and why they might be missing

A benchmark is a method of scoring model capability using standardized task sets—SWE-bench for code fixes, competitive leaderboards like Code Arena for web development. Its value lies in providing a comparable, reproducible capability reference that lowers users' selection costs. Vendors might skip benchmarks for several reasons: the preview isn't fully tuned yet and scores wouldn't be representative; concern that comparisons with competitors would look unfavorable; or a deliberate choice to let community testing generate buzz. But the cost of zero benchmarks is an information vacuum—rumors, single-point tests, and subjective impressions fill the gap, producing enormous variance in evaluations. The 49-vs-75 code-fix comparison here comes from a single person's single test, with an extremely small sample size, and can't be treated as conclusive; likewise, praise for visual tasks depends on specific prompts and aesthetic judgment. When interpreting any community test, pay attention to task type, number of attempts, and whether retries were allowed.

Who Should Try It, Who Should Wait

Three limitations are worth knowing upfront. The package is limited to interactive use within the coding tool—no scripts, no batch processing. It isn't fast—Ajay measured about 47 tokens per second, and one river scene took 30 minutes. Most critically, the preview ends on November 25, after which the model retires.

If you work on frontend, 3D scenes, or rapid prototyping, $3 is a low-cost way to find out "what one prompt can get you." If you need an agent to fix production codebases, early testers recommend waiting for the official release. Alibaba's post ended with a challenge: take a real problem and test it. The testers did—and got a CPU with 19 violations, an engine you can hear, and a 49 out of 100.

#Qwen 4 Max#Alibaba Qwen#benchmark#CPU design AI#LLM preview#code fix benchmark#CoDuck
Share:

Related articles