Scaling Law

The Scaling Law proposed by OpenAI reveals that model performance improves in a predictable power-law fashion as the number of parameters, training data volume, and compute increase.

The Scaling Law was first systematically quantified by Kaplan et al. in 2020 in language model research, pointing out a power-law relationship between model performance and parameter count, data volume, and compute.

The scaling laws established by Kaplan et al. in 2020 indicate that model performance follows a power-law relationship with parameter count, data volume, and compute.

Scaling Laws were first systematically proposed and quantified by OpenAI researchers in the 2020 paper "Scaling Laws for Neural Language Models."

In 2020, OpenAI researchers Kaplan et al. published a paper that first systematically revealed the predictable power-law relationships between language model performance and model parameter count, training data volume, and computational budget (Scaling Law)

Scaling Law was discovered by OpenAI researchers in 2020 and states that model performance improves predictably when model parameter count, training data volume, and compute scale at specific ratios.

Scaling Law refers to the predictable and regular improvement in model capability as the number of parameters and the volume of training data increase.

OpenAI proposed the Scaling Law in 2020, revealing a power-law relationship between model performance and parameter count, data volume, and compute.

The scaling laws were systematically demonstrated in 2020 by OpenAI researcher Jared Kaplan and others, with the core finding that the cross-entropy loss of language models follows a power-law relationship with model parameter count, training data volume, and compute

From GPT-1 to GPT-3, the model architecture itself did not undergo fundamental changes; the primary improvements came from the expansion of parameter scale and the increase in training data, confirming the Scaling Law.

Multiple research institutions have pointed out that high-quality publicly available text generated by humans may be exhausted between 2026 and 2028.

Scaling Law描述了Transformer模型性能与参数量、数据量、计算量之间存在可预测的幂律关系

Google翻译团队2006年的论文展示了类似于后来被称为Scaling Law的现象——随着数据量和计算量增加，模型质量呈现规律性提升

After large models entered the post-pretraining era, the capability gains from simply stacking parameters have become increasingly diminishing, and vendors have turned to the post-training phase in search of breakthroughs.

The neural network scaling laws proposed by Kaplan et al. in 2020 indicate that model performance has a power-law relationship with data volume, parameter count, and compute, where the model's cross-entropy loss L and parameter count N satisfy L ∝ N^(-α).

Latent space reasoning may shift AI from pursuing larger scale toward better architectures, which is closely related to the industry's reflection on the diminishing marginal returns of Scaling Laws.

Scaling Law research found that model performance (measured by cross-entropy loss) has a predictable power-law relationship with model parameter count, dataset size, and computational cost.

Investing more computational resources at the inference stage can yield performance gains (test-time compute scaling theory), and the gains exhibit a明显 region of diminishing returns.

Leading AI companies are shifting their competitive focus from parameter scale at training time to efficiency and cost-effectiveness at inference time, with price reductions serving as a strategy to capture market share and build an ecosystem moat.

Scaling Law research has found that model performance continues to improve when compute and data are scaled proportionally, but whether the rate of improvement is diminishing remains a subject of debate