Scaling Laws
A set of empirical relationships proposed by Kaplan and colleagues at OpenAI in 2020, describing power-law dependencies between model performance and parameter count, training data volume, and compute. These findings motivated the development of large-scale models such as GPT-3 and GPT-4.
Core Facts
Timeline (last 90 days)
Scaling Laws refer to the power-law relationship between model performance (such as loss function values) and model parameter count, data volume, and compute.
Scaling Laws were first systematically summarized by OpenAI and other institutions around 2020.
Scaling Laws underpinned the industry consensus that "bigger models are necessarily stronger" and influenced the decision-making logic behind compute investment and model scale.
DeepMind demonstrated in the Chinchilla paper that a considerable proportion of large models are data-insufficient, meaning more high-quality data should be used under a given compute budget.
Diffusion models perform better in global consistency but have slower inference speeds, while autoregressive models generate content frame by frame or segment by segment.
All Facts (6)
The Scaling Law proposed by OpenAI reveals that model performance improves in a predictable power-law fashion as the number of parameters, training data volume, and compute increase.
90%UnverifiedDeepMind demonstrated in the Chinchilla paper that a considerable proportion of large models are data-insufficient, meaning more high-quality data should be used under a given compute budget.
70%UnverifiedScaling Laws refer to the power-law relationship between model performance (such as loss function values) and model parameter count, data volume, and compute.
50%UnverifiedScaling Laws were first systematically summarized by OpenAI and other institutions around 2020.
50%UnverifiedScaling Laws underpinned the industry consensus that "bigger models are necessarily stronger" and influenced the decision-making logic behind compute investment and model scale.
50%UnverifiedDiffusion models perform better in global consistency but have slower inference speeds, while autoregressive models generate content frame by frame or segment by segment.
50%