GPT-2
A large-scale language model released by OpenAI in 2019 with 1.5 billion parameters, capable of generating coherent and fluent long-form text. Due to concerns about potential misuse for disinformation, it was released in stages. GPT-2 is an early and significant version in the GPT model series.
Core Facts
When OpenAI released GPT-2 in 2018, it staged the public release of the model weights on the grounds that its capabilities were too powerful and could be misused.
GPT-series models use the Byte Pair Encoding (BPE) algorithm to segment text into tokens.
The GPT series follows the decoder route, adhering to the autoregressive language model objective, and has revealed the phenomenon of emergent capabilities through scaling.
Timeline (last 90 days)
In the GPT-2 vocabulary, the emoji 🙂 (4 UTF-8 bytes) becomes 2 tokens; GPT-4o's larger vocabulary learned the entire sequence, so it is 1 token.
The researchers constructed a smoothed predictor based on neighboring token-pair counts and evaluated it on GPT-2's full vocabulary of 50,257 tokens.
Ethayarajh's 2019 research found that the contextual embeddings of BERT and GPT-2 occupy a narrow cone.
Early language models from the GPT-2 era tended to 'forget' earlier context in long texts, resulting in contradictory character behavior or plot discontinuities.
The research team compared seven BPE-based tokenizers within a single language: GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2.
Representative work in mechanistic interpretability includes the complete circuit reconstruction of the indirect object identification (IOI) task in GPT-2.
Top-p sampling has been widely adopted by models such as GPT-2 and outperforms fixed Top-k in open-domain generation tasks.
Weight tying sets the input embedding matrix and the output projection matrix as the same set of parameters; early models such as GPT-2 have already adopted this approach, which reduces parameters and helps training stability.
The indirect object identification (IOI) task was first systematically studied on GPT-2 by Wang et al. (2022).
0.8B parameters places it in the same ultra-small model tier as Meta's LLaMA 3.2 1B and Google's Gemma 2B, close to the lower bound of early GPT-2 (1.5B) in scale.
29 more timeline events
All Facts (20)
When OpenAI released GPT-2 in 2018, it staged the public release of the model weights on the grounds that its capabilities were too powerful and could be misused.
80%VerifiedGPT-series models use the Byte Pair Encoding (BPE) algorithm to segment text into tokens.
80%VerifiedThe GPT series follows the decoder route, adhering to the autoregressive language model objective, and has revealed the phenomenon of emergent capabilities through scaling.
80%VerifiedGPT series models use byte-level BPE (Byte-level BPE), which takes raw bytes as the basic unit to eliminate the out-of-vocabulary character problem.
75%VerifiedKarpathy's earlier nanoGPT implemented GPT-2 training in approximately 600 lines of code.
75%VerifiedDario Amodei previously served as VP of Research at OpenAI, leading the research work on key models such as GPT-2 and GPT-3.
70%VerifiedOpenAI has used a variant of BPE since GPT-2, and GPT-4's cl100k_base vocabulary contains approximately 100,000 tokens.
65%UnverifiedKarpathy的llm.c项目用纯C语言实现了GPT-2训练，完全不依赖PyTorch等框架
90%Partially VerifiedFew-shot learning capability was almost unusable in the era of GPT-2 (1.5 billion parameters) and suddenly became practical with GPT-3 (175 billion parameters).
75%Partially VerifiedGPT-2's largest version had approximately 1.5 billion parameters
65%UnverifiedTransformer Explainer is an interactive tool developed by a team at Georgia Tech that uses GPT-2 as a basis to demonstrate the computation flow of each Transformer layer.
60%UnverifiedIn the GPT-2 vocabulary, the emoji 🙂 (4 UTF-8 bytes) becomes 2 tokens; GPT-4o's larger vocabulary learned the entire sequence, so it is 1 token.
50%UnverifiedThe researchers constructed a smoothed predictor based on neighboring token-pair counts and evaluated it on GPT-2's full vocabulary of 50,257 tokens.
50%UnverifiedEthayarajh's 2019 research found that the contextual embeddings of BERT and GPT-2 occupy a narrow cone.
50%UnverifiedEarly language models from the GPT-2 era tended to 'forget' earlier context in long texts, resulting in contradictory character behavior or plot discontinuities.
50%UnverifiedThe research team compared seven BPE-based tokenizers within a single language: GPT-2, GPT-4, gpt-oss, Llama 3.1, Gemma 3, Qwen3, and Kimi K2.
50%UnverifiedRepresentative work in mechanistic interpretability includes the complete circuit reconstruction of the indirect object identification (IOI) task in GPT-2.
50%UnverifiedTop-p sampling has been widely adopted by models such as GPT-2 and outperforms fixed Top-k in open-domain generation tasks.
50%UnverifiedWeight tying sets the input embedding matrix and the output projection matrix as the same set of parameters; early models such as GPT-2 have already adopted this approach, which reduces parameters and helps training stability.
50%UnverifiedThe indirect object identification (IOI) task was first systematically studied on GPT-2 by Wang et al. (2022).
50%