Building GLM-5.3-Flash from Scratch: Training a 25M Parameter Model on CPU

AI Research with Just a CPU

The central claim of this tutorial is genuinely disruptive: you don't need an expensive GPU cluster. With nothing more than a regular CPU or MacBook, you can build and train a 25-million-parameter GLM-5.3-Flash model from scratch, running through the full pipeline of pre-training and reinforcement learning (RL). The author's point is clear — the model is small, but the architectural principles and experimental methodology you learn are fundamentally the same as what researchers at OpenAI and Anthropic do every day, just at a simpler scale.

The author raises a thought-provoking observation: the nature of AI research has shifted dramatically in just a few months. Tools like Claude Code and Codex can now independently write and run the vast majority of experimental code. A researcher's real value lies in "proposing high-level ideas" — deciding which experiments to run, how to design environments, and what research questions to ask. He cites Anthropic's latest job descriptions as covering 95% of AI researcher roles, and notes that writing code is no longer the core of these positions.

As he puts it: "Humans propose good questions and set direction; AI handles implementation, experimentation, and iteration." But this doesn't mean you can skip understanding algorithms — if you want to invent faster, better algorithms, you first need to understand how existing ones work. That's why this tutorial still walks through the code line by line.

Why Character-Level Vocabulary Instead of BPE

Real large models use BPE (Byte Pair Encoding) tokenizers with vocabularies typically ranging from 150,000 to 300,000 subword tokens. In a small model, this creates a fatal problem: the embedding layer and output head — two weight matrices with one dimension equal to vocabulary size — can consume 95% of all parameters, leaving only 5% for the actual model layers.

To sidestep this, the tutorial treats each ASCII character as an individual token, giving a vocabulary of just 260 tokens. This eliminates the need to train a tokenizer and ensures that parameters are actually spent on the model itself.

The author walks through the basic Transformer logic clearly: each token in the input is replaced by a corresponding vector embedding, which encodes the token's "meaning" (for example, the vector for "+" encodes information about the addition operation). The full prompt becomes a sequence of vectors fed into Transformer layers, and the final hidden state at the end is converted into a probability distribution over the entire vocabulary to predict the next token.

BPE (Byte Pair Encoding) is the standard tokenization method for most major language models. Starting from individual characters, it repeatedly merges the most frequently occurring adjacent byte pairs to produce a "subword vocabulary." High-frequency fragments like "ing," "tion," and "##ly" become single tokens, allowing the same text to be represented with fewer tokens — reducing sequence length and improving computational efficiency. GPT-4's tiktoken vocabulary has around 100,000 tokens; LLaMA around 32,000; the original GLM even more. The tradeoff is that BPE requires training a dedicated tokenizer, and the vocabulary matrix itself represents a massive parameter cost. For a model with only 25 million parameters, a 250,000-row embedding matrix would consume almost the entire parameter budget, leaving virtually nothing for the Transformer layers responsible for actual reasoning. A character-level vocabulary sacrifices compression efficiency in exchange for minimal parameter overhead and zero tokenizer dependency — a pragmatic and sensible tradeoff for a teaching context.

Architectural Highlights of GLM-5.3-Flash

Compared to the basic Transformer covered in earlier tutorials, GLM-5.3-Flash introduces several new structures: a hybrid linear and sparse attention mechanism, Mixture of Experts (MoE), DeepSeek^Ad-style manifold-constrained residual connections (hyperconnections), and weight tying (the input and output layers share the same vocabulary transformation matrix).

The tutorial version is drastically scaled down: the context window is just 192 tokens (the original GLM supports 1 million), and the number of layers and experts is far fewer than the original. The author openly admits that before reinforcement learning, this small model cannot successfully complete any task; after RL training, it can pass 16 out of 24 tasks — simple things like "write a Python function that multiplies the input by 2" — but the before-and-after difference is striking.

The most architecturally notable component is the sparse indexer. Instead of attending to the entire context window, a lighter, cheaper indexer first identifies the most relevant tokens (say, the top 1,000 or 2,000), and full attention is only computed over those. The indexer is itself a form of attention, but with far less compute — it scales much more slowly with sequence length than standard quadratic attention.

Linear attention (KDA, state-space model style) maintains a fixed-size "memory matrix" that all tokens write into sequentially. It excels at remembering recent tokens and runs extremely fast, but loses precise information over long distances. The sparse layers periodically restore that precision from far back. The author's experience is that this hybrid design is significantly faster to train and run, optimized specifically for inference.

For MoE, the original GLM has 288 experts with each token selecting 8, plus one shared expert that every token passes through (learning general information so other experts don't have to repeat it). The tutorial's small model simplifies this to between 2 and 8 experts.

Mixture of Experts (MoE) replaces feed-forward network (FFN) layers with multiple "expert" sub-networks. A router dynamically selects a small subset of experts for each token rather than activating all parameters, which allows total parameter count to scale dramatically while keeping per-forward-pass computation constant. Models like DeepSeek-V3/R1 use MoE to scale total parameters to hundreds of billions while keeping activated parameters around 37 billion, balancing model capacity with inference efficiency. The "shared expert" in GLM and DeepSeek architectures is a design detail: one expert always applies to all tokens, dedicated to learning cross-task general knowledge — preventing routing experts from repeatedly learning the same basic patterns and improving parameter utilization. Weight tying refers to sharing the same parameters between the input embedding matrix and the output projection matrix; GPT-2 and other early models already used this technique to reduce parameters by one vocabulary-size × hidden-dimension matrix, and it has been found to improve training stability.

Pre-training: Learning to Write Code by Imitation

Pre-training is fundamentally imitation learning. Given a prompt with a Python function, the model simply reproduces completions it has already seen in the data — it doesn't "think" about the answer. By predicting the next character, it gradually learns what each character means and becomes capable of generating them.

The standard pipeline is unchanged: probability distribution, cross-entropy, gradient update. The optimizer updates all parameters based on next-token correctness, making the right byte a little more likely each time. The author uses AdamW rather than the currently popular Muon, purely for simplicity.

Before training, the model outputs random characters; afterward, it gradually learns to generate correct code given a prompt. The author also walks through several real research experiments: the tradeoff between data repetition and data diversity (a small amount of repeated data learns faster initially, but diverse data overtakes it after 200 steps); whether to train on multiple data types "A then B then C" vs. interleaved (interleaved training prevents catastrophic forgetting, so the model doesn't forget A and B while learning C); and whether curriculum learning (simple before complex) actually works. The author honestly notes that some experimental differences may not be statistically significant, and that you must think carefully about what an experiment is actually measuring rather than over-interpreting results.

Reinforcement Learning: Teaching a Small Model to "Think"

Reinforcement learning is fundamentally different from pre-training. Instead of reproducing text, the model tries to generate answers and receives rewards for correct ones. The author puts it candidly: "It sort of learns to think, but I don't know exactly how — if you figure that out, you'll become the most famous AI researcher."

In this small model's post-training, the success rate rises from 4% to 45%; measured differently, the model goes from passing 0 out of 24 tasks to passing 16 (pass@1, meaning a single attempt per task).

The algorithm used is RLOO (in the same family as GRPO). The core idea: for the same problem, the model generates multiple answers (e.g., 16), each scored based on correctness, format validity, syntax, etc. The advantage of each answer is computed relative to the group average — answers better than average become more likely, worse ones become less likely. No separate judge model or human scoring is needed.

A few practical engineering details worth remembering: don't show the model the test cases, or it may engage in "reward hacking" — gaming the scoring system by reading the answers or even attempting to compromise the system. Reward structure can be designed flexibly: +1 for all tests passing, 0 for correct format but wrong numbers, -1 for invalid Python syntax. During RL updates, the author only updates the last block, final normalization layer, and output head — avoiding changes to earlier layers that might cause forgetting, while also making updates faster.

RLOO (REINFORCE Leave-One-Out) and GRPO (Group Relative Policy Optimization) both belong to the "group-relative advantage" family of algorithms — improvements over classic REINFORCE policy gradients. Traditional REINFORCE estimates gradients from a single sample, resulting in very high variance and unstable training. GRPO/RLOO's improvement: sample a group of answers for the same question (e.g., 16), use the average reward of the rest of the group as a baseline, and compute each answer's "advantage" = its own reward − group mean, drastically reducing variance. Compared to PPO, these methods don't require training a separate Critic (value function) network, making implementation much simpler. They've been widely adopted in post-training for language models including DeepSeek-R1 and Qwen. The reward signal itself can be rule-based (e.g., does the code pass unit tests? does the answer equal the reference answer?), without relying on another model for scoring — making the entire pipeline feasible even with limited compute.

Data Matters More Than Algorithms

A core judgment running through the entire tutorial: once you have a good enough algorithm, further optimization of the algorithm quickly hits diminishing returns, while optimizing data, environments, and training objectives yields far greater payoffs. The author uses DeepSeek as an example — they've stopped investing heavily in the GRPO algorithm because it's "good enough," and are directing more effort toward data and environment design.

The author's experiments also reveal the limitations of small models: because the model is small, RL only improves performance on similar tasks seen during training. One task showed no improvement and may have even regressed — the author candidly says it's "probably a code bug, but if it's not, that's even more interesting." He repeatedly emphasizes the importance of statistical significance: paper reviewers typically require testing across three random seeds, with all three yielding clear and consistent results. His own experiments, being too small in scale, did not produce statistically significant improvements.

On experimental design methodology, the author cites MIT's recommendation: start with small toy examples so you can truly understand what's happening — far better than jumping straight into large, complex experiments. Change only one variable at a time — the reward structure, group size, temperature (exploration vs. exploitation), penalty strength, and so on.

Advice for Aspiring AI Researchers

The author's career advice is refreshingly direct: if you want to get into OpenAI or Anthropic, you don't need a GPU. You need to learn how to build reinforcement learning environments, how to think about them, and how to verify whether a model is actually learning. Unless you specialize in scaling or pre-training, compute scale is not the key.

He recommends consistently posting your experiments on social media — "consistency beats intensity." Publishing steadily once a week beats posting five days in a row and then going silent. He also recommends keeping model training time to a few seconds, not wasting time waiting, because what you really need to practice is designing experiments and environments — not competing with big companies on compute, a battle you can never win.

All code is open-sourced on GitHub. Readers can have Codex or Claude Code clone the repo, explain the code, reproduce the charts, and even design more challenging experiments on top of it.