MiniMind: An Open-Source Tutorial for Training a Large Language Model from Scratch in 2 Hours

Training Your Own Large Language Model from Scratch

In an era where large models routinely boast hundreds of billions of parameters and training costs run into millions of dollars, an open-source project called MiniMind is taking the opposite approach — its tagline is "Train a 64M-parameter large language model from scratch in just 2 hours." Since its release, the project has rapidly garnered over 55,000 Stars and 7,287 Forks on GitHub, gaining 472 new stars in a single day and becoming a widely discussed educational LLM project among developers worldwide.

The value of MiniMind isn't in "building a model that can rival GPT" — it's in transforming large language models from a "black box" into something that can be disassembled, reproduced, and understood. For the vast majority of developers, training a state-of-the-art large model is neither realistic nor necessary. But walking through the entire pipeline from data to inference is the best way to truly understand the essence of LLMs.

Why a 64M-Parameter Small Model Deserves Your Attention

Ultra-Low Training Barrier Makes It Accessible to Individual Developers

The project's core selling point is compressing training costs to a range affordable for individual developers. According to the author, the smallest model version has only about 26 million to 64 million parameters and can be fully trained — from random initialization to conversational capability — in as little as 2 hours on a single consumer-grade GPU (such as an NVIDIA 3090). Compared to mainstream models that require clusters and weeks of training, this is practically "toy-level" cost.

For reference, training a GPT-3-scale model (175 billion parameters) is estimated to require millions of dollars in compute and thousands of high-end GPUs working together for weeks. Even smaller open-source models like LLaMA-7B require 2,048 A100 GPUs running for approximately 21 days for full pretraining. MiniMind compresses the parameter count to 64M (roughly one-millionth of GPT-3), making single-GPU training possible. While this extreme simplification sacrifices model capability, it delivers unprecedented learning accessibility.

Behind this design philosophy is a clear educational principle: Get it running first, then understand it, then optimize it. When the time to train a model shrinks from "weeks" to "an afternoon," learners can conduct extensive trial-and-error experiments, building genuine intuition about hyperparameters, data quality, and training techniques.

Fully Open-Source: Building an LLM from Scratch with Pure PyTorch

What makes MiniMind particularly valuable is that it doesn't simply call high-level wrappers from Hugging Face — instead, it implements the core LLM architecture using native PyTorch code. PyTorch is a deep learning framework developed and maintained by Meta AI, widely favored by researchers and developers for its dynamic computation graphs and Pythonic programming style. It has become the dominant framework for training large language models in both academia and industry. Hugging Face's Transformers library provides numerous pre-packaged models and training tools on top of PyTorch — convenient to use, but hiding underlying implementation details to some extent. MiniMind's choice of pure PyTorch implementation means every critical line of code is transparent and visible to learners.

The project covers all key stages of a large language model's complete lifecycle:

• Tokenizer Training: Building a tokenizer from scratch rather than reusing an existing vocabulary. The tokenizer is the first step in how a large language model processes text, splitting raw text into the smallest units the model can handle — tokens. Modern LLMs widely adopt subword-based tokenization algorithms, the most common being BPE (Byte Pair Encoding). BPE starts from individual characters, repeatedly counts the most frequently occurring adjacent character pairs in the corpus, and merges them into new subword units until the vocabulary reaches a preset size. This method strikes a good balance between vocabulary size and text representation efficiency — common words are encoded as whole units while rare words are split into multiple subword segments, effectively solving the out-of-vocabulary problem. MiniMind's from-scratch tokenizer training allows learners to fully understand this often-overlooked but performance-critical component.

• Pretraining: Autoregressive language modeling on unlabeled text. Autoregressive language modeling is the core training paradigm of virtually all mainstream large language models today. The basic idea is: given all preceding words in a text sequence, the model must predict the probability distribution of the next word. The model generates tokens from left to right, using all previously generated tokens as context at each step. The GPT series, LLaMA, DeepSeek, and other models all employ this paradigm. This stage typically requires massive amounts of text data (trillions of tokens), through which the model learns grammar, semantics, world knowledge, and other capabilities.

• Supervised Fine-Tuning (SFT): Teaching the model to follow instructions and engage in conversation. Supervised fine-tuning is the critical step that transforms a pretrained language model into a practical conversational assistant. While the pretrained model has learned rich linguistic knowledge, it is essentially just a "text continuation engine." SFT fine-tunes the model on carefully constructed "instruction-response" data pairs, teaching it to understand and follow human instructions. The InstructGPT paper was the first to systematically demonstrate the power of SFT: even smaller models, after high-quality instruction fine-tuning, can produce output quality surpassing that of larger models without fine-tuning.

• LoRA Fine-Tuning: Low-cost adaptation for domain-specific tasks. LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method proposed by Microsoft in 2021. Its core idea is based on a key insight: the weight update matrices during fine-tuning of pretrained models typically have very low intrinsic rank. Instead of directly modifying the original model weights, LoRA inserts two small low-rank matrices alongside the Transformer's attention layers and trains only these two matrices. This can reduce the number of trainable parameters to one-thousandth or even one-ten-thousandth of the original model, dramatically decreasing memory usage while achieving performance close to full-parameter fine-tuning.

• DPO Reinforcement Learning: Alignment training based on human preferences. DPO (Direct Preference Optimization) is an alignment training method proposed by Stanford University in 2023. It serves as a more streamlined alternative to traditional RLHF (Reinforcement Learning from Human Feedback). Traditional RLHF requires separately training a reward model and a policy model, then optimizing with reinforcement learning algorithms like PPO — a complex and often unstable process. Through mathematical derivation, DPO merges reward modeling and policy optimization into a simple classification loss function, directly training the language model itself on preference data pairs, greatly simplifying the training pipeline.

• Model Distillation: Transferring knowledge from large models to small models. Knowledge distillation was first proposed by Hinton et al. in 2015. The core idea is to transfer knowledge from a large "teacher model" to a small "student model." The student model learns not only from the ground truth labels (hard labels) but also from the probability distributions output by the teacher model (soft labels). Soft labels contain rich "dark knowledge" about inter-class similarity relationships, helping the student model learn more information than hard labels alone. In the LLM domain, distillation techniques are widely used to compress the capabilities of very large models into smaller ones.

In other words, every stage you'd encounter in real industrial-grade LLM training — pretraining, alignment, fine-tuning, distillation — has a corresponding "minimum viable implementation" in MiniMind. This "small but complete" characteristic is precisely what sets it apart from other toy projects.

Educational Value: Taking Large Language Models Apart

Demystifying Large Models

Many developers' understanding of large language models stops at the API call level — "input a prompt, get a result." By fully exposing the training code, MiniMind lets learners see firsthand how a model starts from a pile of random weights, gradually learns to predict the next word, and eventually becomes capable of coherent conversation.

This "white-box" process is incredibly enlightening. When you train a model with your own hands that — while rough around the edges — can actually hold a conversation, you develop a far deeper understanding of concepts like the Transformer architecture, attention mechanisms, and loss curves than you'd ever get from reading papers alone. The Transformer is a neural network architecture proposed by the Google team in the 2017 paper Attention Is All You Need, which completely replaced the RNNs and LSTMs that had previously dominated NLP. Its core innovation — the self-attention mechanism — allows each position in a sequence to directly attend to all other positions, efficiently capturing long-range dependencies. In practice, multi-head attention splits the attention computation into multiple parallel "heads," enabling the model to simultaneously attend to information in different subspaces. Modern large language models build on this foundation with additional optimizations like RoPE (Rotary Position Embedding), GQA (Grouped Query Attention), and Flash Attention — and MiniMind's code provides intuitive, simplified implementations of these mechanisms.

The loss curve refers to the trend of the model's prediction error (typically measured by cross-entropy loss) over the course of training steps. A healthy loss curve should show a steady downward trend. By observing the loss curve, learners can intuitively understand whether the model is learning effectively, whether overfitting or underfitting is occurring, and whether hyperparameter settings like the learning rate are appropriate.

A Complete Learning Tutorial for Chinese-Speaking Developers

The project provides not only code but also comprehensive Chinese documentation and tutorials, which is a major reason for its rapid spread in the Chinese developer community. For Chinese-speaking developers, an open-source project that clearly explains "how a large model is built" in their native language offers a learning experience far more accessible than dense English papers and documentation.

A Realistic Perspective: Where Are the Limits of a Small Model?

It's important to be clear-eyed: a 64M-parameter model is worlds apart from mainstream large models like GPT-4 and DeepSeek in terms of actual capability. Models trained with MiniMind cannot handle complex reasoning, long contexts, or specialized knowledge Q&A — their outputs look "roughly right" rather than being "genuinely useful."

This capability gap isn't merely a simple numerical difference in parameter count — it also involves an important research finding in the large language model field: Scaling Laws. Research published by OpenAI in 2020 showed that language model performance follows predictable power-law relationships with model parameters, training data volume, and compute: as these three factors scale up proportionally, model loss decreases steadily. More importantly, when model scale exceeds certain critical thresholds, so-called "Emergent Abilities" appear — capabilities that are completely absent in smaller models (such as chain-of-thought reasoning and zero-shot learning) suddenly emerge. A 64M-parameter model is far below the scale thresholds where these abilities emerge, which fundamentally determines its capability ceiling.

But this isn't a flaw — it's the intended positioning. MiniMind is a learning tool, not a production tool. The question it answers is "How do large models work?" not "How do I build a commercial product?" Using it for getting started, teaching, and conducting experiments on fundamental principles is the ideal use case; expecting it to produce a usable conversational system will lead to disappointment.

Who Should Try MiniMind?

Overall, the following groups of people will benefit most from this project:

• AI Beginners: Students and engineers who want to understand the full LLM pipeline through hands-on practice
• University Teaching: Can serve as a lab project for deep learning and NLP courses
• Algorithm Researchers: Those who need a lightweight, rapidly iterable experimental platform to validate ideas
• Developers Curious About Large Models: Anyone who wants to demystify LLMs and "forge" a model with their own hands

Conclusion

As the arms race in large models intensifies, MiniMind offers a valuable "reverse perspective" — not pursuing bigger and stronger, but pursuing more transparent and more understandable. At minimal cost, it brings a technology that once seemed out of reach back within the grasp of ordinary developers.

More than 55,000 Stars already demonstrate the community's endorsement of this "democratizing education" philosophy. For anyone who truly wants to understand the underlying principles of large language models, spending an afternoon running through MiniMind may be more rewarding than reading ten survey papers.