Qwen2.5-1.5B
A small language model released by Alibaba's Qwen team, with 1.5 billion parameters. It supports Chinese and English and is commonly used for local inference and quantization experiments.
Timeline (last 90 days)
The experiments use the recurrent model Ouro-1.4B and the standard Transformer Qwen2.5-1.5B, constructing minimal-pair questions that differ by only one number.
When the probe looks at only one prompt at a time (the conventional setting), even after training on 5,249 questions, Ouro's readout rate is only 0.36 and Qwen's only 0.19.
At the last prompt token, the numerical information does exist and is represented in a way shared across questions, with a readout accuracy of about 0.9.
Qwen2.5-1.5B scores 0.729 on GSM8K, 18.2 percentage points lower than Gemma 4 E2B; Qwen3-0.6B scores 0.599, 31.2 percentage points lower; SmolLM2-1.7B scores 0.461, 45.0 percentage points lower.
In the replaced model, the tied embedding/lm_head matrix remains bf16, so each token actually reads about 1.263 GB of weights (plus about 0.06 GB of KV cache); calculated by actual bytes moved, 25.5 tok/s corresponds to about 34 GB/s, roughly 10% of measured bandwidth.
For the Qwen2.5-1.5B model, the int4 theoretical roofline ceiling for batch-1 decoding is 411.8 tok/s, while the actual self-developed kernel reaches only 25.5 tok/s, a gap of about 16x.
The dataset covers four open-weight models — Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B — with 30 trajectories each.
All Facts (7)
The experiments use the recurrent model Ouro-1.4B and the standard Transformer Qwen2.5-1.5B, constructing minimal-pair questions that differ by only one number.
50%UnverifiedWhen the probe looks at only one prompt at a time (the conventional setting), even after training on 5,249 questions, Ouro's readout rate is only 0.36 and Qwen's only 0.19.
50%UnverifiedAt the last prompt token, the numerical information does exist and is represented in a way shared across questions, with a readout accuracy of about 0.9.
50%UnverifiedQwen2.5-1.5B scores 0.729 on GSM8K, 18.2 percentage points lower than Gemma 4 E2B; Qwen3-0.6B scores 0.599, 31.2 percentage points lower; SmolLM2-1.7B scores 0.461, 45.0 percentage points lower.
50%UnverifiedIn the replaced model, the tied embedding/lm_head matrix remains bf16, so each token actually reads about 1.263 GB of weights (plus about 0.06 GB of KV cache); calculated by actual bytes moved, 25.5 tok/s corresponds to about 34 GB/s, roughly 10% of measured bandwidth.
50%UnverifiedThe dataset covers four open-weight models — Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B — with 30 trajectories each.
50%UnverifiedFor the Qwen2.5-1.5B model, the int4 theoretical roofline ceiling for batch-1 decoding is 411.8 tok/s, while the actual self-developed kernel reaches only 25.5 tok/s, a gap of about 16x.
50%