Unverified50% confidenceFactExact time
The classic Chinchilla scaling law points out that for each FLOPs budget, one should scan different model sizes and choose the optimal one, with the rule of thumb being that the number of training tokens is about 20 times the number of parameters.
1
Sources
50%
Confidence
Long-term
Relevance
10/31/2026
First Seen
Sources
Stanford CS336: The Hardcore Philosophy of Building Large Models from Scratch
bilibili大模型前沿课程10/18/2026
Related Entities
Related Claims
Unverified训练FLOPs可通过Chinchilla缩放定律估算，计算量约等于6乘以参数量乘以训练token数79% similarVerifiedIn 2022, DeepMind published the Chinchilla paper, which corrected the optimal training ratio and indicated that, under a given compute budget, the number of model parameters and the number of training tokens should maintain an approximate ratio of 1:2076% similarUnverifiedThe experiment trains a small MLP on a modular addition task modulo 97, with the training set making up only 35%, using full-batch training for a total of 20k steps.68% similarUnverifiedSome regulatory proposals require that training runs exceeding 10²⁶ FLOPs be subject to mandatory reporting and auditing.67% similarUnverifiedTraining used a context length of 256 and a micro-batch size of 16 with gradient accumulation, and peak memory during the main training process was 1.54 GiB67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1121625API
curl https://kongchang.com/api/v1/knowledge/claims/1121625MCP
get_claim(id=1121625)