Pre-training
The foundational stage in which large language models acquire their capabilities through self-supervised learning—typically next-token prediction—on large-scale text corpora. This stage enables the model to internalize linguistic patterns, world knowledge, and reasoning modes, and determines the model's overall capability ceiling.
Core Facts
Timeline (last 90 days)
The technical learning path for the FDE route is: start with AI Agent development or private model deployment, gradually go deeper into fine-tuning and model compression (distillation/quantization, etc.), and then to pre-training; in terms of difficulty, fine-tuning < model compression < pre-training.
The data pricing methods currently being explored in the industry include: by data volume (per token/per record), tiered licensing by use case (separate pricing for pre-training/fine-tuning/evaluation), revenue sharing based on downstream model gains, and auction mechanisms.
Technical difficulty exists on a gradient: fine-tuning < SFT < pre-training, and the recommended learning order is to start with fine-tuning first.
One hard requirement for large model algorithm engineers is mastery of large model pre-training.
Pre-training from scratch requires massive amounts of data and extremely high computing power, and is typically only affordable for large laboratories.
The magnitude of technical innovation between pre-trained models and fine-tuned models is entirely different.
The development investment for pre-trained models can reach millions to tens of millions of dollars.
A language model in the pre-training stage only learns to continue text and does not understand the concept of answering questions or following instructions
Reinforcement learning training is the key stage in which a model transitions from general language capability to precise task execution, further optimizing reasoning, planning, and instruction-following through reinforcement learning after large-scale pre-training.
预训练阶段模型通过自监督学习方式（通常是下一个 token 预测任务）在海量文本语料（通常达数万亿 token 规模）上进行训练
2 more timeline events
All Facts (19)
大语言模型通过在数千亿乃至数万亿个词元（Token）的文本数据上进行自监督预训练
80%Verified当代前沿模型的训练方法包括RLHF（基于人类反馈的强化学习）及其后继者RLAIF和过程奖励模型（PRM）
80%Verified大语言模型的预训练阶段本质上属于自监督学习，是一种特殊的非监督学习，模型通过预测下一个词来从海量文本中自动学习语言规律
80%VerifiedReinforcement learning training is the key stage in which a model transitions from general language capability to precise task execution, further optimizing reasoning, planning, and instruction-following through reinforcement learning after large-scale pre-training.
75%VerifiedA pre-trained model refers to a model that has been trained in advance on large-scale datasets, which users can directly fine-tune or call to complete specific tasks.
75%VerifiedIn-context learning was first observed in large language models, and its capability fundamentally stems from the diversity of pretraining data.
75%VerifiedLarge model training goes through three core stages: pre-training, supervised fine-tuning (SFT), and preference alignment.
75%Verified仅用成功轨迹训练的模型往往存在分布外泛化的严重缺陷，遇到工具返回错误时便丧失纠偏能力
70%Unverified训练数据在输入模型前会经过严格的去重、过滤和质量筛选，低质量内容、重复文本、有害内容会被剔除
90%UnverifiedA language model in the pre-training stage only learns to continue text and does not understand the concept of answering questions or following instructions
60%UnverifiedAI的训练过程分为三个阶段：预训练、后训练和对齐
55%UnverifiedThe technical learning path for the FDE route is: start with AI Agent development or private model deployment, gradually go deeper into fine-tuning and model compression (distillation/quantization, etc.), and then to pre-training; in terms of difficulty, fine-tuning < model compression < pre-training.
50%UnverifiedThe data pricing methods currently being explored in the industry include: by data volume (per token/per record), tiered licensing by use case (separate pricing for pre-training/fine-tuning/evaluation), revenue sharing based on downstream model gains, and auction mechanisms.
50%UnverifiedTechnical difficulty exists on a gradient: fine-tuning < SFT < pre-training, and the recommended learning order is to start with fine-tuning first.
50%UnverifiedOne hard requirement for large model algorithm engineers is mastery of large model pre-training.
50%UnverifiedPre-training from scratch requires massive amounts of data and extremely high computing power, and is typically only affordable for large laboratories.
50%UnverifiedThe development investment for pre-trained models can reach millions to tens of millions of dollars.
50%UnverifiedThe magnitude of technical innovation between pre-trained models and fine-tuned models is entirely different.
50%Unverified预训练阶段模型通过自监督学习方式（通常是下一个 token 预测任务）在海量文本语料（通常达数万亿 token 规模）上进行训练
50%