Scaling Laws

The Scaling Law proposed by OpenAI reveals that model performance improves in a predictable power-law fashion as the number of parameters, training data volume, and compute increase.

The Scaling Law was first systematically quantified by Kaplan et al. in 2020 in language model research, pointing out a power-law relationship between model performance and parameter count, data volume, and compute.

In 2020, OpenAI researchers Kaplan et al. published a paper that first systematically revealed the predictable power-law relationships between language model performance and model parameter count, training data volume, and computational budget (Scaling Law)

Scaling Law refers to the predictable and regular improvement in model capability as the number of parameters and the volume of training data increase.

Scaling Law research has found that model performance exhibits a smooth power-law relationship with the number of model parameters N, dataset size D, and training compute C

The scaling laws were systematically demonstrated in 2020 by OpenAI researcher Jared Kaplan and others, with the core finding that the cross-entropy loss of language models follows a power-law relationship with model parameter count, training data volume, and compute

In 2020, Kaplan et al. at OpenAI revealed that model performance follows predictable power-law relationships with parameter count, training data volume, and compute budget.

缩放定律（Scaling Laws）由Kaplan等人在2020年提出，后经Chinchilla论文修正

Factors such as data quality, training techniques and hyperparameters, architectural details, and the choice of evaluation benchmarks all affect the model's final performance, and each factor introduces its own uncertainty

The paper "Scaling Laws for Neural Language Models" by Kaplan et al. came from OpenAI and was the first to systematically describe the relationship between model performance and parameter count, data size, and compute using power laws.

The paper found that for a fixed compute budget, training a smaller model for longer tends to yield better results than training a larger model, which contradicts the Chinchilla law.

The neural network scaling laws proposed by Kaplan et al. in 2020 indicate that model performance has a power-law relationship with data volume, parameter count, and compute, where the model's cross-entropy loss L and parameter count N satisfy L ∝ N^(-α).

Scaling Law research found that model performance (measured by cross-entropy loss) has a predictable power-law relationship with model parameter count, dataset size, and computational cost.

The Scaling Law paper by Kaplan et al. in 2020 revealed that, under a fixed compute budget, there is an optimal ratio between model parameter count and the number of training tokens (the Chinchilla optimal).

The industry has recently observed that benchmark improvements from scaling have noticeably diminished, and high-quality training text data is approaching the upper limit of what is obtainable from the web.

Scaling Laws describe the decrease in pre-training loss, not a linear improvement in all downstream capabilities; some capabilities, such as complex reasoning, exhibit emergent properties.

Under fixed computing resources, too many parameters with insufficient data prevents the model from learning adequately, while massive data with insufficient parameters means the model lacks the capacity to capture complex patterns

Even top-tier labs find it difficult to accurately predict the performance of a model at a given scale in advance, and typically need to approximate the answer through extensive scaling law experiments

Since GPT-3, OpenAI has established a technical roadmap driven by Scaling Laws, improving model capabilities by increasing model parameter count, training data volume, and computational resources.

Kaplan et al. found that the relationship between model loss and parameter count N approximately satisfies the power-law relationship L(N) ∝ N^(-0.076).