nanoGPT
An open-source project developed by Andrej Karpathy that reproduces the core GPT-2 architecture in minimal code, serving as an introductory reference implementation for learning the Transformer training pipeline.
Core Facts
Timeline (last 90 days)
nanoGPT uses 124M as the default starting point.
Open-source projects such as nanoGPT and TinyLlama have already demonstrated the educational and application value of small models.
WeCo member Bing Chen Zhao's work applying automated research to nanoGPT was retweeted by Andrej Karpathy.
MIT's xv6 helps learners understand operating systems, while MinGPT and nanoGPT make the skeleton of GPT transparent, representing a learning paradigm of pedagogical re-implementation
OpenArch belongs to the same lineage of thinking as education-oriented projects like nanoGPT and minGPT.
The NanoGPT Speedrun concept originated from Andrej Karpathy's nanoGPT project.
nanoGPT reproduces the core architecture of GPT-2 with minimal code.
micrograd and nanoGPT have received over 70,000 and 30,000 stars on GitHub, respectively.
Andrej Karpathy, former head of Tesla AI and co-founder of OpenAI, developed nanoGPT, implementing GPT-2 training in approximately 600 lines of code, and released llm.c, implemented in pure C.
Teaching resources that are fully annotated with ELI5 explanations prioritize beginner-friendliness over engineering efficiency, and are complementary to concise implementations such as nanoGPT.
6 more timeline events
All Facts (19)
Karpathy's earlier nanoGPT implemented GPT-2 training in approximately 600 lines of code.
75%VerifiedNanoGPT implements a complete GPT training pipeline in fewer than 300 lines of code.
65%UnverifiedKarpathy的开源项目nanoGPT被全球开发者广泛用于学习大模型原理
85%UnverifiedKarpathy's microGPT project is a minimalist GPT training implementation whose core code is only a few hundred lines, aimed at helping developers understand the essence of the Transformer architecture.
60%UnverifiedOpen-source projects such as nanoGPT and TinyLlama have already demonstrated the educational and application value of small models.
50%UnverifiednanoGPT uses 124M as the default starting point.
50%UnverifiedMIT's xv6 helps learners understand operating systems, while MinGPT and nanoGPT make the skeleton of GPT transparent, representing a learning paradigm of pedagogical re-implementation
50%UnverifiedOpenArch belongs to the same lineage of thinking as education-oriented projects like nanoGPT and minGPT.
50%UnverifiedThe NanoGPT Speedrun concept originated from Andrej Karpathy's nanoGPT project.
50%UnverifiednanoGPT reproduces the core architecture of GPT-2 with minimal code.
50%UnverifiedAndrej Karpathy, former head of Tesla AI and co-founder of OpenAI, developed nanoGPT, implementing GPT-2 training in approximately 600 lines of code, and released llm.c, implemented in pure C.
50%UnverifiedTeaching resources that are fully annotated with ELI5 explanations prioritize beginner-friendliness over engineering efficiency, and are complementary to concise implementations such as nanoGPT.
50%UnverifiedTeaching resources with full annotations and ELI5 place more emphasis on beginner-friendliness rather than engineering efficiency, complementing projects like nanoGPT that pursue concise implementations.
50%UnverifiednanoGPT and minGPT are classic educational projects by Andrej Karpathy, showcasing concise and elegant minimal implementations.
50%UnverifiedThe approach of implementing large models from scratch is highly regarded in the industry because it forces learners to understand the mathematical principles and design decisions behind every line of code.
50%UnverifiedThe approach of implementing GPT from scratch is highly regarded in the industry because it forces learners to understand the mathematical principles and design decisions behind every line of code.
50%Unverified延迟解耦(Delayed Untying)的核心思想是：训练前2/3阶段将embed与lm_head绑定共享，到达某个节点后复制权重和优化器状态，然后让二者作为独立参数分别训练
50%UnverifiednanoGPT速通竞赛中参与者竞相在NVIDIA A100或H100等GPU上以最短wall-clock时间达到特定验证集loss
50%UnverifiedWeCo member Bing Chen Zhao's work applying automated research to nanoGPT was retweeted by Andrej Karpathy.
50%