15-Year-Old Trains 200M MoE Language Model from Scratch: Full Architecture & Hands-On Breakdown

A 15-Year-Old's Ambitious Project

In an era where large models routinely pack hundreds of billions of parameters and training costs run into the millions of dollars, a 15-year-old developer named Abdelrhman Ebied shared his work on Reddit: Tiny-MoE — a Mixture-of-Experts (MoE) language model trained entirely from scratch. The project wasn't built to compete with mainstream models like Llama or Qwen. Its goal was far more focused: to genuinely understand how modern large language models work by implementing the entire training pipeline by hand.

The author openly acknowledges that the model is "nowhere near state-of-the-art," but it can generate English text, answer simple questions, and handle basic coding and math tasks. For an independent learning project built entirely on free Kaggle GPU resources, that's a solid result. Even more impressive is the technical stack — the architecture choices reflect a deep understanding of cutting-edge model design.

Technical Architecture: More Than a Toy Implementation

Looking at the technical details, Tiny-MoE is far more substantial than the "practice project" label might suggest. It integrates several ideas from the current frontier of model design.

Mixture-of-Experts (MoE) Architecture

The model has roughly 200M total parameters, but only about 90M are activated per token. This is the core advantage of the MoE architecture — a routing mechanism directs different "expert" networks to handle different types of inputs, expanding model capacity while keeping inference costs low. This "sparse activation" approach is the same key technique used in mainstream open-source models like Mixtral and DeepSeek.

The history of MoE goes back to Jacobs et al. in 1991, but it didn't truly explode in the LLM space until after 2022. The core idea is to replace a single feed-forward network (FFN) with multiple parallel "expert" networks, paired with a lightweight "router" that decides which experts should process each token. The Auxiliary Load Balancing Loss is a critical challenge in MoE implementations — without intervention, the router tends to route most tokens to a handful of experts, leaving the rest idle. This is known as expert collapse. The fact that a 15-year-old independently implemented the routing and load balancing logic demonstrates an understanding of modern LLMs well beyond the introductory level.

Multi-Head Latent Attention (MLA)

The project uses Multi-Head Latent Attention (MLA), the attention mechanism introduced by DeepSeek-V2 that has attracted significant attention. Standard Multi-Head Attention requires caching the Key and Value matrices for every layer during inference (the KV Cache), with memory usage scaling proportionally with sequence length and batch size — one of the primary bottlenecks in LLM inference costs. MLA addresses this by compressing KV matrices into a low-dimensional "latent vector" for caching, then recovering them via an up-projection matrix when needed. This is essentially a low-rank decomposition of the KV Cache. Compared to GQA (Grouped Query Attention), MLA applies a far more aggressive compression ratio, theoretically reducing KV Cache memory usage to just 5–13% of standard MHA. For compute-constrained environments, this is an eminently practical choice — it directly determines whether the model can run at all within the constraints of Kaggle.

Positional Encoding: RoPE + YaRN

For positional encoding, the author uses RoPE (Rotary Position Embedding) combined with YaRN for context extension. RoPE was proposed by Su Jianlin in 2021. Its core idea is to encode positional information as rotation matrices applied to Query and Key vectors, so that attention scores naturally capture relative position. RoPE is now standard in most mainstream models, but performance degrades sharply beyond the training context length. YaRN (Yet another RoPE extensioN method), proposed by Peng et al. in 2023, applies non-uniform scaling to RoPE's rotation frequencies along with an "attention temperature" adjustment, extending the model's context window by several to tens of times with minimal additional training — making it one of the lowest-cost context extension approaches available. This combination shows the author isn't just thinking about getting the model to run, but has also engineered for long-context handling.

Training Pipeline: A Complete Engineering Loop

It's worth emphasizing that Tiny-MoE is not built on the Hugging Face Transformers library — it's implemented in native PyTorch. That choice itself reflects the author's learning philosophy: don't just call pre-built libraries; implement every component by hand.

The model was trained from scratch on 8 billion (8B) tokens, with a carefully chosen dataset mix:

• FineWeb-Edu: Released by Hugging Face in 2024, this is a high-educational-value subset filtered from the 15-trillion-token FineWeb dataset. A dedicated classifier scores web content for educational value, retaining roughly 1.3 trillion tokens — known for its rigorous data cleaning.
• Cosmopedia v2: Represents the synthetic data approach — textbooks, stories, and Wikipedia-style articles generated in bulk by models like Mixtral-8x7B to compensate for the scarcity of real high-quality text.
• OpenWebMath: Focused on extracting math-related content from Common Crawl, with careful preservation of LaTeX formulas and denoising — a corpus dedicated to mathematical content.

The intent behind this data mix is clear: educational text improves language quality, math corpora strengthen reasoning, and together they embody the "small data, high quality" training philosophy championed by the Phi model series — fully aligned with the industry consensus that data quality matters more than data quantity.

The project also implements streaming and packed data pipelines, custom training and inference code, and text generation strategies covering Top-k, Top-p, repetition penalty, and n-gram blocking. These decoding strategies serve real engineering purposes: Top-k sampling limits each step to the k highest-probability tokens, while Top-p (nucleus sampling) dynamically selects the smallest set of tokens whose cumulative probability exceeds threshold p — the two are often combined to balance diversity and coherence. Repetition penalty and n-gram blocking address a classic failure mode of small models: getting stuck in repetitive loops. These are compensated for at the decoding stage through engineering. Together, these details form a complete engineering loop rather than a collection of disconnected code snippets.

Limitations and Honest Positioning

The author is clear-eyed about the model's limitations. He explicitly notes that Tiny-MoE still struggles with longer text generation — at the 200M scale, limits in parameter capacity and training data make it difficult to maintain long-range coherence.

But from a learning perspective, these limitations are exactly where the most valuable lessons hide. Understanding why a model breaks down on longer text is itself an important part of mastering how LLMs work. In his Reddit post, the author actively solicited feedback on code quality, project structure, model architecture, training process, and documentation — a rare display of openness and engineering maturity.

Why This Project Deserves Attention

The significance of Tiny-MoE lies not in benchmark scores, but in the trend and spirit it represents.

First, large model technology is being democratized. A few years ago, training a language model from scratch was something only top-tier research labs could do. Today, with open-source tooling, public datasets, and free compute like Kaggle, a high school student can independently walk through the entire process. The continued lowering of the barrier to entry is unlocking unprecedented creative potential.

Second, it sets an example for how to learn. Instead of repeatedly calling APIs at the application layer, the deeper approach is to sit down and implement things from the ground up. The author's choice to use native PyTorch and implement MoE and MLA by hand offers an extremely valuable reference path for anyone who wants to genuinely understand LLMs.

Third, the positive feedback loop of open community. By publishing the code on Reddit and asking for input, the author turned a personal project into an open-source asset that the community can examine and improve. The project is hosted on GitHub, and any interested developer is welcome to contribute.

For developers who also want to get started with LLM training, Tiny-MoE is a hands-on reference well worth studying in depth. It proves one thing: the most effective way to understand modern AI is often to build it yourself, from scratch.