Qwen3 235B A22B in Practice: A 2.4-Trillion-Parameter Open Model Surpasses Claude Opus

Chinese Trillion-Parameter Models Launch in Rapid Succession

The iteration pace of Chinese large models lately has been dizzying. Shortly after the release of Kimi K3, Alibaba's Tongyi Qianwen team dropped a bombshell of its own—the most powerful version of Qwen3 (with a staggering parameter count of 2.4 trillion, i.e., 2.4T). More importantly, the team has officially confirmed it will be released as an open weight model.

It's worth clarifying a frequently confused distinction here: there is a subtle but crucial difference between open weight and what is typically meant by fully open source. Open weight means the model's parameter files are available for public download and local deployment—users can run inference and even fine-tune it—but the training code, complete dataset, and training pipeline are not necessarily made public. Meta's LLaMA series is a representative example of this model: its weight files are publicly downloadable, but usage must comply with a proprietary commercial license agreement, and key aspects such as training data composition and the RLHF alignment process remain opaque. True open source (as defined by OSI) requires that code, data, and training processes all be transparent and reproducible—anyone can view, modify, and redistribute without restrictive clauses.

Notably, the OSI (Open Source Initiative) formally released the Open Source AI Definition v1.0 in 2024, explicitly requiring that genuinely open source AI must include data information sufficient to reproduce training, complete training code, and parameter weights—all three are indispensable. This definition directly excludes mainstream "open source" models like the LLaMA series and Mistral, and has led the more precise term "open weight" to gradually become industry consensus.

The deeper implications of this distinction extend far beyond legal text. From a regulatory standpoint, the EU AI Act imposes lighter oversight on open weight models than on closed-source ones, but this has sparked debate: are open weight models that lack training data transparency really safer than closed-source models? Open source advocates argue that public weights at least allow safety researchers to conduct red-team testing and adversarial attack analysis on the model; critics point out that once an open weight model is released, it cannot be recalled, and if alignment flaws exist, the cost of fixing them is far higher than the server-side hot updates available to closed-source models. From a security audit perspective, open weight models allow researchers to reverse-engineer the parameters to uncover potential biases or adversarial vulnerabilities, but the lack of training data transparency means the origins of model behavior cannot be understood at the root. Meta's LLaMA series and Mistral both fall under the open weight model rather than strict open source. This distinction has substantive implications for commercial usage licensing, the feasibility of security auditing, and academic reproducibility—but for most developers, open weight is practical enough: data stays on-premises, system prompts can be customized, latency is more controllable, and it can be deployed locally, fine-tuned, and integrated into private systems. This is the core value that the closed-source API model cannot provide, and it's why "open weight" enjoys far greater acceptance in the commercial ecosystem than the criticism of its limitations from academia would suggest.

Worth noting is that behind the 2.4T parameter scale, a Mixture of Experts (MoE) architecture is very likely at play. The idea behind this architecture traces back to early academic exploration in 1991, but the modern MoE revival can be traced to the 2017 paper Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer by Google Brain, which for the first time integrated a sparse MoE layer into a large sequence model, achieving order-of-magnitude improvements in training efficiency. What truly brought MoE into the public spotlight was Mixtral 8x7B, released by Mistral AI in early 2024—the first mainstream open weight MoE model, which caused a stir at the time with its combination of 46.7B total parameters and 12.9B active parameters.

The MoE architecture took thirty years to mature from academic concept to engineering practice. Google's 2022 Switch Transformer was the first to scale MoE to trillion-parameter levels, validating the architecture's feasibility; 2023's GLaM and ST-MoE addressed the core pain point of early MoE training instability—using auxiliary loss functions to constrain router load balancing, preventing the problem of a few experts being over-activated while most experts degrade into dead neurons. It was this series of engineering breakthroughs that made the 2.4T scale a deliverable product today rather than a number on paper.

Its core structure is: in the Transformer's feed-forward network layer, the single FFN is replaced with N parallel "expert networks," equipped with a learnable gating router (Router), where each token during inference activates only K of the experts (typically K=2 or K=8). Unlike traditional dense models, MoE does not activate all parameters during inference; instead, the "router" dynamically selects a few "expert subnetworks" to process each token. The core trade-off of MoE is: it requires more GPU memory during training than a dense model with equivalent active parameters (because all expert weights must be loaded), but the FLOPs consumption during inference can be greatly reduced. Taking Qwen3 2.4T as an example, if its dense active parameters are on the order of ~200B (analogous to DeepSeek-V3's ratio of 671B total / 37B active), then the actual compute consumption during inference is comparable to a mid-sized dense model, yet the model's knowledge capacity and expressive power approach the full parameter scale. The engineering challenges of MoE center on two points: router load balancing (preventing all tokens from flowing to the same expert), and cross-device communication overhead in distributed inference—this is also the biggest challenge in productionizing MoE inference services. GPT-4, Mixtral, and DeepSeek-V3 all adopt similar approaches, and it is precisely this architecture that makes ultra-large-scale models feasible from an engineering standpoint.

Although the complete model weights, detailed specifications, and official benchmarks have not yet been formally released, the "open source" point has been clearly committed to by the team. In an era dominated by closed-source large models topping the leaderboards, everyone can see what a trillion-scale open source model means.

Even more noteworthy is the bold declaration in the official release announcement: the model is "one of the most powerful models today, second only to Fable 5." Such statements are rare in the industry, and one YouTube reviewer decided to use his own benchmark to verify whether this claim lives up to its billing.

The Preview Version Is Already Available for Use

For users eager to try it out, the good news is there's no need to wait for the weights to be released. The preview version is already live and can be called directly through the official Token plan as well as tools like Coder. In other words, if you've already subscribed to a Token plan, you can experience this cutting-edge model for free right now.

The reviewer did exactly that—he plugged the model into Claude Code and conducted a comprehensive evaluation using his self-built "KingBench" benchmark. What distinguishes KingBench from mainstream academic benchmarks (like MMLU, HumanEval, GSM8K) is its "observability" design philosophy. A common feature of mainstream benchmarks is that scoring can be fully automated and results are binary—this brings about a "Goodhart's Law" effect: when a scoring metric becomes the optimization target, models learn to score high on these specific formats without necessarily genuinely improving their capabilities. This problem is especially prominent in the AI evaluation field: when HumanEval was released, it represented the frontier of code capability evaluation; but as training data contamination on this benchmark worsened across major models, HumanEval's discriminative power has dropped sharply—some models score over 95% on HumanEval, yet their actual code generation abilities differ enormously.

This phenomenon is called "Benchmark Contamination" by researchers, and its essence is the erosion of the boundary between training and test sets. When the MMLU benchmark was released, it represented the frontier of knowledge reasoning evaluation, but as major models incorporated it into their training targets, some models can now exceed 90% on MMLU while performing mediocrely on real-world tasks. It is against this backdrop that new evaluation frameworks like KingBench and LiveCodeBench, built with "anti-contamination" as a core design principle, have begun to attract attention.

KingBench's counter-strategy is to convert evaluation results into visual artifacts that humans can intuitively judge—whether a 3D glasses case can be clicked open, whether a panda looks like it's "eating," whether an archery game has a good feel. Humans can make these judgments in seconds, yet they cannot be "gamed" by models, because the output itself is the final judge. The output of each question is a visualized interactive artifact, rather than filling in answers or code pass-rate statistics. Whether the front end renders correctly, animation smoothness, and game playability all rely on human subjective judgment rather than automated scoring, making it closer to real user needs and an effective discriminator between top-tier and near-top-tier models. This evaluation philosophy aligns with the direction of frameworks like Stanford's HELM, Scale AI's SEAL, and LiveCodeBench, which uses real-time new questions from competition platforms to avoid data contamination: reducing reliance on fixed-format answers and increasing the weight given to evaluating open-ended output quality. This test suite covers front-end and animation tasks, Three.js 3D tasks, SVG drawing, math puzzles, long-horizon autonomous agent tasks, and one extremely difficult 3D modeling problem, with a maximum of 10 points per question and a total of 80 points.

Breaking Down the Eight Hardcore Tests One by One

Front-End and 3D Interaction Capability

The first question is the elevator simulator: it requires building a simulation system that can generate passengers at different floors, with three elevators each carrying only one person; passengers who miss the ride must wait for the next trip, and the system displays each person's target floor prompt when hovering. It ultimately scored 8 points—the elevator logic was correct and the animation was smooth. While it fell short of Opus 4.8's perfect score, it tied with GPT 5.6 and GLM 5.2.

The second question, the Three.js contact lens case, is a challenge almost every model fails—it requires a lid with prominent L/R markings that can be clicked open. Three.js is a JavaScript 3D graphics library based on WebGL, and its challenge lies in the synchronization of multiple layers of abstraction: the model must simultaneously understand Euler angle and quaternion rotation representations, object hierarchy relationships (the propagation of parent-child node transformations), the interaction between the camera frustum and lighting model, and the timing relationship between the browser event loop and render frames.

The reason Three.js can be a powerful tool for evaluating top-tier models is that it sits at the intersection of multiple knowledge domains: at the mathematical level it involves linear algebra (matrix transformations, coordinate system conversions), quaternion rotations (avoiding the gimbal lock problem), and raycasting (used for mouse click detection); at the software engineering level it requires understanding the parent-child transformation propagation mechanism of the Scene Graph; and at the WebGL rendering pipeline level it involves the workings of vertex shaders and fragment shaders. Especially worth mentioning is the "Gimbal Lock" problem—when Euler angles are used to describe 3D rotation, certain angle combinations cause two rotation axes to coincide, losing one degree of freedom and producing animation jitter. The core value of quaternions lies precisely in fundamentally avoiding this problem at the mathematical level, and correctly choosing the rotation representation is one of the key judgment points for whether Three.js code runs smoothly. When a model can, without visual feedback, purely through code reasoning, derive a 3D interactive scene that renders correctly in the browser, this proves it possesses a systematic understanding spanning multiple abstraction layers, rather than fragmented API memorization. Any deviation in understanding at any dimension causes the 3D scene to collapse visually, and the error is often not an exception but "rendering something weird," which is hard to catch via test cases—this is precisely why the vast majority of models have historically failed this question. This model likewise scored 8 points, becoming the second-strongest model ever on this question (only behind the perfect-scoring Fable 5, even surpassing Opus 4.8, which only got 7 points). The third question, the folding table slider animation, also scored 8 points, tying with Opus 4.8.

Drawing and Game Physics

The fourth question required drawing a panda eating a hamburger using SVG. The challenge of SVG (Scalable Vector Graphics) is entirely different from Three.js—it requires the model to build a precise 2D coordinate system in its mind, use the Bézier curves of path commands to describe organic shapes (like the curvature of a panda's ears), and ensure that the layering order and proportional relationships of multiple shapes are internally consistent at the pure code level. This is a "no pixel buffer" drawing challenge: relying entirely on code to describe geometric shapes and spatial relationships, with no room for pixel-level error tolerance. It demands extremely high spatial imagination and coordinate reasoning from the model, and is an effective discriminator between "being able to generate runnable code" and "truly understanding space and graphics." SVG's path command system (combinations of M/L/C/Q commands) is essentially a domain-specific language for describing geometry. The model must not only remember the syntax but also "pre-render" the final image in mental space without any canvas feedback—this ability differs fundamentally from pure text reasoning and is a true reflection of a large model's spatial understanding ability. The result had coordinated proportions, a highly recognizable hamburger, and authentically captured the "eating" posture, scoring 8 points—tied for the highest with Kimi K3, Grok 4.5, and GLM 5.2.

The fifth question, the archery shooting game, required implementing aiming and shooting, a hit leaderboard, and other features. This model delivered a perfect score of 10—the game was playable, the physics feel was good, and the leaderboard updated normally, a result previously achieved only by Opus 4.8 and Grok 4.5.

Reasoning and Long-Horizon Agent Tasks

The sixth question is a math puzzle about counting permutations of ordered pairs, with the correct answer being 2460—many models fail this completely. This model precisely calculated 2460, banking a perfect score and joining the ranks of strong reasoning models like Opus 4.8, Fable 5, Kimi K3, and GPT 5.6, proving it's not just a "front-end pretty face."

The seventh question is the long-horizon autonomous task that most tests comprehensive capability: generate a game facts dataset from scratch, fine-tune a Gemma 2B model, then build a local web interface that generates a fun panda fact each time it refreshes. This represents the most challenging task type in the current AI Agent field—a multi-step tool calling chain. It differs fundamentally from a single code generation task: the model must maintain a "working memory" across multiple steps, sequentially completing cross-domain operations like data generation, environment configuration, model fine-tuning, and service deployment.

The ReAct (Reasoning + Acting) framework is the mainstream paradigm for such tasks, formally proposed by Princeton University and Google in a 2022 paper. Its core idea is to have the language model complete tool-augmented tasks by alternately outputting "thinking steps" and "action instructions," making the decision process traceable and debuggable. In engineering practice, the bottleneck of the ReAct framework often lies not in single-step reasoning ability but in error recovery ability—when a tool call returns an unexpected error, can the model correctly interpret the error message and adjust its subsequent plan, rather than falling into a loop of repeatedly attempting the same failed operation (i.e., a "hallucination loop")? This requires the model to coordinate three capabilities: state tracking (remembering that the fine-tuning script was successfully generated but not yet executed), error recovery (being able to re-plan the path when pip dependency installation fails), and tool interface adaptation (understanding the input format conventions of different tools). OpenAI, Anthropic, and Google DeepMind all list "reliable multi-step task completion capability" as a core research goal for 2024-2025; benchmarks like TAU-Bench and SWE-bench, specifically aimed at real software engineering tasks, show that even the strongest models rarely exceed a 50% completion rate on complex tasks requiring dozens of steps—this is precisely what makes a perfect score on the seventh question so valuable. This model completed the entire process fully autonomously, earning another perfect score—in actual engineering this is equivalent to a junior engineer independently completing a full ML project iteration cycle, and it was the third perfect score in a row.

The eighth question, the 3D watch, was the hardest of the entire set, requiring the second/minute/hour hands to run smoothly and supporting date, day of week, and dual time zone display. It ultimately scored only 3 points—the watch could render and the hands moved, but it didn't reach the requirement of being fully usable. However, for reference—the highest score on this question was only Fable 5's 4 points, and many models scored 0 outright, so 3 points is actually a decent attempt.

Ranked Second, With Marketing That Rarely Delivers

In the end, the model finished with 65/80 points (81.25%), ranking second on the reviewer's entire leaderboard. It trailed only Fable 5's 82.5%, but clearly surpassed Claude Opus 4.8's 80%, while significantly leading Kimi K3 (77.5%), GLM 5.2 (75%), and GPT 5.6 (71.25%).

In other words, the official declaration of "second only to Fable 5" was delivered precisely, down to the letter, in this independent benchmark. For users accustomed to vendor overhyping, "marketing matching reality" is itself a rare thing.

User Experience: Not Without Shortcomings

That said, the charts don't tell the whole story. When the reviewer actually plugged it into Claude Code, he found the model responded quickly, followed instructions well, and its output quality did reach the level of top-tier closed-source models—but it performed slightly weaker on sustained tasks in long sessions—sometimes files that should have been created weren't, and it occasionally "dropped the ball" at the tool-chain handoff level.

This phenomenon is closely related to two structural bottlenecks that all top-tier models currently face in agentic scenarios. The first is the "attention decay" problem of the context window, also known as the "Lost in the Middle" phenomenon—formally named by a Stanford University research paper of the same name in 2023. The research found that when Transformer models process long context, they exhibit a systematic attention discount for information located in the middle of the context: the model more easily remembers content at the beginning and end while ignoring the middle. The underlying mechanism of this characteristic relates to positional encoding schemes—although modern schemes like RoPE (Rotary Position Embedding) and ALiBi have improved the extrapolation ability for ultra-long context, the U-shaped distribution characteristic of attention (strong at both ends, weak in the middle) still exists. This stems from the softmax normalization characteristic of the attention mechanism itself rather than a flaw in a specific positional encoding, meaning this limitation cannot be completely eliminated by a simple engineering patch. This characteristic produces serious engineering consequences in agent scenarios: in a long task containing 50 rounds of tool calls, the temporary file path generated in round 20 has likely "faded" from the model's effective attention range by round 45. Current mainstream mitigation strategies include: forcing the model to maintain a structured "working state summary" in the system prompt, introducing external key-value storage as an explicit memory module, and using sliding window compression techniques to summarize early context.

The second is the tool calling protocol adaptation problem: different frameworks (Claude Code, LangChain, AutoGen, etc.) have subtle differences in the JSON format for tool calls and the semantic conventions of error return codes. If the model mainly saw the call logs of a certain framework during pre-training and fine-tuning, it may generate calling instructions with format deviations under other frameworks. These two problems are different in nature: the former requires improving the model's long-range attention mechanism or introducing an external memory module; the latter can be quickly fixed through a small amount of instruction fine-tuning (SFT) targeted at a specific framework. This also explains why the same model performs so differently across different agent frameworks—it takes longer real-world usage to distinguish between the two.

The reviewer particularly emphasized that this is only a preliminary observation. He plans to continue using it as his daily primary model for a period of time before giving a more complete conclusion in a follow-up video.

Conclusion: A Milestone for the Open Source Camp

All things considered, this ultra-large-scale version of Qwen3 is a highly competitive model: ranked second, edging out Claude Opus 4.8, and achieving perfect scores on the three tasks of game development, math reasoning, and agents. And its greatest value lies in accessibility—the preview version is already included in the Token plan, allowing users to experience a frontier-level model at a cost far lower than that of closed-source labs.

Combined with the model weights soon to be opened, this is undoubtedly a heavy blow for the entire open source ecosystem. A model with 2.4 trillion parameters, adopting an MoE architecture, and reaching this level of performance choosing to open its weights means researchers can study its behavior locally, enterprises can deploy it in private environments without sending data to third-party APIs, and the developer community can build specialized fine-tuned versions on top of it—not something you see every day. From Kimi K3 to Qwen3, the birth of two consecutive trillion-scale models in a short period once again confirms that Chinese large models are accelerating their rise.

Key Takeaways

Key Takeaways

Key Takeaways

Key Takeaways