Reinforcement Learning
Reinforcement Learning is a major branch of machine learning in which an agent learns an optimal decision-making policy through continuous interaction with an environment. At each state, the agent takes an action and adjusts its behavior based on reward or penalty signals from the environment, with the objective of maximizing long-term cumulative reward. It does not rely on labeled data, is applicable to sequential decision problems, and is widely used in game playing, robotics, autonomous driving, and resource scheduling.
Core Facts
Machine learning is typically divided into three major paradigms: supervised learning, unsupervised learning, and reinforcement learning.
DQN is an algorithm proposed by DeepMind in 2013 that for the first time combined deep neural networks with Q-learning reinforcement learning, enabling AI to learn to play 49 Atari games.
The pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
Timeline (last 90 days)
Comparing on the same 32B model, DeepSeek found that pure RL reached 47% on AIME 2024 while distillation from R1 reached 72.6%, indicating that for small models distillation from a stronger model is more effective than large-scale RL.
Reinforcement learning training relies on large numbers of resettable, parallelizable environment instances, because a policy needs millions of trial-and-error attempts to converge.
A new study proposes deriving the RL objective as maximum likelihood fitting to a model-generated target distribution, thereby unifying pretraining, SFT and RL into the same family of objective functions.
ShallowDefend training takes place at the "payload turn", the turn in which the harmful intent actually surfaces, and uses reinforcement learning to consolidate the decision behavior.
Memory mechanisms in reinforcement learning face two problems: attention mechanisms need to score the entire memory bank, which is costly; and when retrieved memories bring returns, credit assignment must span long sequences.
When fine-tuning precipitation diffusion models with reinforcement learning, improper reward design can lead to mode collapse (reduced generation diversity) or reward hacking (optimizing only the score at the expense of physical consistency), and training requires extensive sampling, resulting in high computational cost.
FitzRoy is a precipitation forecasting system that fine-tunes a conditional discrete diffusion model via reinforcement learning, and the study claims it is the first to simultaneously achieve leading forecast skill and physical realism across the continental United States.
Experiments on three real-world crisis datasets show that GRID outperforms RL and MCMC baselines in scenario diversity, realism, and tail-risk coverage.
In reinforcement learning, agents maximize cumulative reward through trial and error; in large-language-model-based agents, the objective function is implicit in prompts and training preferences.
In actual products, user input is not directly used as training data for instruction fine-tuning; behavior calibration is mainly accomplished through reinforcement learning.
40 more timeline events
All Facts (20)
Machine learning is typically divided into three major paradigms: supervised learning, unsupervised learning, and reinforcement learning.
90%VerifiedDQN is an algorithm proposed by DeepMind in 2013 that for the first time combined deep neural networks with Q-learning reinforcement learning, enabling AI to learn to play 49 Atari games.
90%VerifiedThe pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
90%Verified奖励函数设计不当会导致奖励欺骗（Reward Hacking）现象，即智能体找到在奖励函数上得分高但实际无意义甚至有害的行为路径
85%VerifiedThe concept of world models originates from cognitive science and reinforcement learning, strongly promoted by Yann LeCun and others, referring to an agent's internal representation of the external environment.
80%VerifiedReinforcement learning learns optimal policies through continuous interaction between an agent and its environment; its mathematical foundation is the Markov Decision Process (MDP), which consists of a state space, action space, transition function, reward function, and discount factor.
80%Verified大语言模型的预训练阶段本质上属于自监督学习，是一种特殊的非监督学习，模型通过预测下一个词来从海量文本中自动学习语言规律
80%VerifiedMachine learning is divided into three major categories: supervised learning, unsupervised learning, and semi-supervised learning.
80%VerifiedLearning-based methods have stronger performance when data is sufficient, but at the cost of robustness under domain shift.
75%VerifiedReward shaping refers to adding extra, denser guiding reward signals on top of the original sparse reward to help agents learn faster and more correctly.
75%VerifiedEach RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.
75%VerifiedIn sparse-reward scenarios, agents often need techniques such as curriculum learning, reward shaping, or intrinsic motivation-based exploration to converge effectively.
75%VerifiedThe full-stack GPU paradigm keeps environment state, dynamics simulation, policy network forward passes, and gradient computation entirely within video memory, eliminating the data transfer bottleneck between CPU and GPU.
75%VerifiedIn reinforcement learning training, the model's goal is to maximize the score given by the reward function.
75%VerifiedRewards in real-world tasks are often extremely sparse and delayed, causing long-horizon credit assignment problems that make it difficult for agents to determine which action led to the final outcome.
75%VerifiedWhen language models are repeatedly trained on data generated by themselves or similar models, systematic degradation occurs, with the diversity of model outputs decreasing with each generation and the tail information of the distribution gradually being lost.
75%VerifiedAI misalignment behavior is fundamentally a direct product of current mainstream training paradigms, particularly reinforcement learning.
75%VerifiedReinforcement learning training is the key stage in which a model transitions from general language capability to precise task execution, further optimizing reasoning, planning, and instruction-following through reinforcement learning after large-scale pre-training.
75%VerifiedDeepMind's AlphaGo learned Go strategies by playing millions of games against itself.
75%VerifiedA Markov Decision Process (MDP) is mathematically defined by a five-tuple (S, A, P, R, γ).
75%