Qwen's Six-Generation Evolution: A Complete Breakdown from Architecture Innovations to the Reasoning Race

Alibaba's Qwen has iterated through six major versions in the two years since its first release in August 2023. If one thread runs through this entire evolutionary arc, it's this: before Qwen2.5, the focus was on model architecture; after Qwen2.5, the focus shifted to post-training and reasoning. This article breaks down the key improvements of each generation along a timeline to help you follow that thread.

The Turning Point: From Architectural Innovation to the Reasoning Race

To understand Qwen's evolution, you first need to grasp a critical inflection point — Qwen2.5. Before it, Qwen's iterations centered on the model architecture itself: MoE (Mixture of Experts), GQA (Grouped Query Attention), long-context extrapolation, and other structural innovations came in rapid succession.

After Qwen2.5, as OpenAI o1 and DeepSeek ignited a wave of competition around reasoning capabilities, the industry's attention shifted from "how to design a model" to "how to train a model." Post-training pipelines grew increasingly complex — this phase refers to the fine-tuning process that follows large-scale text pretraining, targeting human preferences and specific capabilities, and typically includes supervised fine-tuning (SFT), preference optimization (e.g., RLHF, DPO), and reinforcement learning. Dataset sizes kept expanding, and reasoning-focused reinforcement learning became a battleground for all major players.

The main thread is clear: before 2.5, compete on structure; after 2.5, compete on reasoning. Grasp this, and the logic behind all six generations becomes immediately apparent.

Qwen1: Getting Started on LLaMA's Shoulders

The first-generation Qwen, released in August 2023, was fundamentally built on the same technical foundation as LLaMA. On top of the standard Transformer architecture, it adopted several design choices that had already been validated at the time:

• Rotary Position Embedding (RoPE): A positional encoding scheme proposed by Su Jianlin that encodes positional information as rotation matrices within attention computations. Compared to absolute position encodings, it naturally supports relative positional relationships and can be extended via extrapolation techniques to sequences longer than those seen during training. The initial context window was only 2,048 tokens, later extended to 8,192 via NTK interpolation (adjusting the rotational frequency base for smooth scaling).
• Pre-RMS Norm: Normalization placed before the layer, improving training stability.
• SwiGLU activation: Replacing traditional activations to enhance expressiveness.
• Removing most bias terms from linear layers: Simplifying parameters.

On the training side, Flash Attention was used to accelerate computation, along with the AdamW optimizer and BF16 mixed-precision training.

As you can see, the first-generation Qwen wasn't particularly novel — it was more of an integration and engineering implementation of mature industry practices. But it laid a solid foundation for the rapid iterations that followed.

Qwen1.5 and Qwen2: The Golden Age of Architectural Innovation

Qwen1.5: First Introduction of GQA and MoE

Qwen1.5 in February 2024 made two important moves.

First, it introduced GQA (Grouped Query Attention) for the first time. In standard multi-head attention (MHA), each query head has its own independent key-value pair, so memory usage grows linearly with the number of heads. Multi-query attention (MQA) lets all heads share key-value pairs, but at the cost of accuracy. GQA sits between the two — query heads are grouped, with key-value pairs shared within each group — striking a balance between memory efficiency and model expressiveness. Compressing the KV cache directly determines how many concurrent requests can be handled during inference.

Second, it launched the first MoE version. The core idea of MoE (Mixture of Experts) is to replace feed-forward network layers with multiple parallel "expert" sub-networks, where a router dynamically selects a small subset of experts for each forward pass — maintaining high parameter capacity while controlling actual compute costs. Qwen1.5 used a configuration of "4 shared experts + 60 fine-grained experts, with 4 activated per pass." The context window was also increased to 32K.

Qwen2: GQA Rolled Out Across the Full Lineup

Qwen2 in June 2024 extended GQA from select models to the entire product line, directly cutting KV cache usage by 40%. For long-context handling, it combined Yarn extrapolation + Dual Chunk Attention (DCA) to boost long-sequence capabilities. Yarn (Yet Another RoPE extensioN) improves RoPE's interpolation strategy, significantly enhancing long-context performance while preserving short-context quality. DCA splits long sequences into chunks and combines local with cross-chunk attention — together, the two approaches substantially extend the effective context length. Training data also grew from 3 trillion to 7 trillion tokens, and post-training combined offline DPO with online DPO. Offline DPO optimizes on static preference data pairs for stable training; online DPO dynamically generates samples from the current model and labels preferences, producing data distributions that better match the current training state.

These two generations represented Qwen's most architecturally prolific period, with GQA, MoE, DCA, and other now-familiar techniques all landing during this phase.

Qwen2.5: A Great Leap in Data and Reasoning

Qwen2.5, released in September 2024, was a genuine "great leap forward" — and the turning point in the main narrative described above.

First, pretraining data surged from 7 trillion to 18 trillion tokens, with a particular emphasis on strengthening code and math capabilities. This leap in data scale laid a solid foundation for the model's overall capabilities.

More critical was the upgrade to the post-training pipeline. Beyond offline DPO, the online phase introduced GRPO (Group Relative Policy Optimization) for the first time. GRPO is a lightweight PPO variant proposed by the DeepSeek team. Traditional PPO requires a separately trained value network (Critic) to estimate the baseline, which is computationally expensive. GRPO eliminates this independent value network, instead sampling a group of outputs for the same question and using the within-group average reward as the baseline to compute relative advantage values — dramatically reducing training resource consumption. This makes it especially well-suited for large-scale reinforcement learning on tasks with clear verification signals, such as mathematical reasoning and code generation. Meanwhile, the Yarn + DCA combination pushed context capacity all the way to 1 million tokens.

Starting with Qwen2.5, the competitive focus in model development clearly shifted from architecture to data quality and training pipelines — in lockstep with the industry-wide "slow thinking" wave that followed the release of OpenAI o1.

Qwen3: Simultaneous Upgrades to Architecture and Training

Qwen3 in April 2025 was a comprehensive upgrade, advancing on both architecture and training fronts at the same time.

Architecture

• Dense models introduced QK RMS Norm to stabilize attention computation.
• MoE models removed shared experts in favor of finer-grained expert partitioning, paired with a globally batched load-balancing loss — this change distributes expert utilization from a global perspective, preventing the "collapse" problem where some experts are perpetually overloaded while others sit idle.

The most eye-catching innovation was dynamic thinking: a single model can autonomously switch between "thinking" and "non-thinking" modes without requiring two separate deployments. Thinking mode is similar to OpenAI o1's chain-of-thought reasoning — the model generates an internal reasoning process before producing an answer, suited for complex tasks like math and logic. Non-thinking mode outputs answers directly, suited for latency-sensitive scenarios. On the training side, capabilities for both modes were injected in stages through cold-start initialization, GRPO reinforcement, and mode-switching learning — giving a single model the ability to allocate compute on demand. Ultra-large model sizes of 235B and 22B were also released.

Training

Data continued to double, reaching 36 trillion tokens. The post-training pipeline became quite long and complex: starting with chain-of-thought data for cold-start, then GRPO-based reasoning reinforcement, followed by learning to switch between thinking and non-thinking modes, and finally reinforcement learning for specific scenarios. Smaller models were distilled from larger ones using KL divergence (a metric that measures the difference between two probability distributions) to align the smaller model's output distribution as closely as possible with the larger model's.

Qwen3-2507: Separating Thinking and Non-Thinking

Qwen3-2507 in July 2025 takes a different approach — splitting thinking and non-thinking into two completely separate models rather than cramming them into a single model that switches back and forth. This shift reflects a practical deployment reality: dedicated models often outperform general-purpose hybrid solutions in both inference efficiency and deployment simplicity. The context window was further extended to 256K, and dedicated models for specific use cases were released separately: Qwen3-Coder for programming and QwenMT for translation.

Six-Generation Quick Reference and Further Reflections

Condensing six generations into one-line summaries:

• Qwen1: LLaMA foundation — RoPE + RMS Norm + SwiGLU
• Qwen1.5: First introduction of GQA and MoE
• Qwen2: Full-lineup GQA, 40% KV savings, 7T tokens
• Qwen2.5: 18T tokens, online GRPO, 1M context
• Qwen3: Fine-grained MoE, QK Norm, dynamic thinking, 36T tokens
• Qwen3-2507: Thinking and non-thinking separated, 256K context

Core thread: before 2.5, compete on structure; after 2.5, compete on reasoning.

Interestingly, most labs only publish their training pipelines publicly — the detailed composition of datasets is rarely disclosed. So what we can observe from the outside is mostly methodological evolution. The data engineering that truly determines model capability lies beneath the surface of the iceberg: data cleaning strategies, quality filtering ratios, how synthetic data is mixed with real data, and so on. These details are often harder to replicate than the algorithms themselves, and they represent an important lens through which to understand today's competitive landscape in large language models.

Key Takeaways