Unverified50% confidenceFactExact time
A systematic empirical study used nearly 200B tokens for pretraining on decoder-only LLMs ranging from 360M to 3.1B parameters, examining fourteen orthogonal training variants.
1
Sources
50%
Confidence
Long-term
Relevance
11/3/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe research team trained a batch of small models of about 1B parameters, some trained with tokens and some trained with bytes.78% similarUnverifiedExperiments confirm that in the controlled single-head setting, α_L exactly separates generalizing from non-generalizing (PE, task) cells in the taxonomy, based on measurements from 2625 trained models, stable at vocabulary size 64 and dimension 128.76% similarUnverifiedWhen the amount of labeled data is only tens to hundreds of samples, directly training a classifier easily leads to overfitting; one might consider fine-tuning the embedding model with a small number of samples or using k-nearest-neighbor classification.74% similarUnverified预训练阶段模型通过自监督学习方式（通常是下一个 token 预测任务）在海量文本语料（通常达数万亿 token 规模）上进行训练74% similarUnverified研究者已证明可以通过修改不到0.1%的训练数据实现LLM后门植入72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1129384API
curl https://kongchang.com/api/v1/knowledge/claims/1129384MCP
get_claim(id=1129384)