Chinchilla

The Chinchilla law proposed by DeepMind further refined the optimal ratio relationship between data and parameters.

In 2022, DeepMind's Chinchilla paper pointed out that under the same compute budget, model size and training data volume should be scaled up proportionally.

DeepMind's Chinchilla research (2022) pointed out that optimal training should keep data volume and parameter count growing at roughly a 1:1 ratio

The Chinchilla law states that under a fixed compute budget, appropriately reducing model size and increasing training data volume often yields better performance.

Google DeepMind's Chinchilla paper optimized the optimal ratio between model parameter count and training data volume.

The emergent ability phenomenon was systematically documented by DeepMind researchers in the 2022 paper 'Emergent Abilities of Large Language Models'.

缩放定律（Scaling Laws）由Kaplan等人在2020年提出，后经Chinchilla论文修正

DeepMind demonstrated in the Chinchilla paper that a considerable proportion of large models are data-insufficient, meaning more high-quality data should be used under a given compute budget.

Chinchilla surpassed Gopher, which has 280 billion parameters, on multiple benchmarks with only 70 billion parameters.

The core finding of the Chinchilla scaling law is that under a fixed compute budget, simultaneously increasing model parameters and training data volume is required to achieve optimal performance.

The Chinchilla optimal scaling law was proposed by DeepMind in 2022, with the core formula C≈6ND, and the condition for optimal training being D≈20N.

The Chinchilla model, with 70B parameters and 1.4T tokens, significantly outperforms the Gopher model, which has 280B parameters and 300B tokens, under the same compute budget.

The paper attributes the gap of Manacá-1B-Instruct-2 in general capabilities to pretraining scale (a comparison of compute budgets anchored on Chinchilla) rather than to the alignment method itself.

Chinchilla's experimental scale was limited, and its conclusions are based on specific datasets and training setups; whether they remain applicable to very large-scale models or domain-specific applications is a matter of debate.

The conclusions of the Chinchilla scaling law challenged the prevailing practice of the GPT-3 era, in which the industry invested most compute into scaling model size while training data was relatively insufficient.

DeepMind's Chinchilla paper and subsequent research on inference-time compute scaling show that dynamically increasing computation during the inference phase can significantly improve the output quality of smaller models.

The Chinchilla scaling law originates from DeepMind's 2022 paper "Training Compute-Optimal Large Language Models"

The rule of thumb given by the Chinchilla scaling law is that the number of training tokens should be approximately 20 times the number of parameters

DeepMind's Chinchilla scaling law (from the 2022 paper "Training Compute-Optimal Large Language Models") shows that model performance depends on the optimal ratio of parameter count to training token count, with the optimal training volume being approximately 20 times the number of parameters

The LLaMA series adopts an over-training strategy, training a 7B-parameter model on 1T tokens, far exceeding the Chinchilla-optimal ratio of 140B tokens