Reinforcement Learning
Reinforcement Learning is a major branch of machine learning in which an agent learns an optimal decision-making policy through continuous interaction with an environment. At each state, the agent takes an action and adjusts its behavior based on reward or penalty signals from the environment, with the objective of maximizing long-term cumulative reward. It does not rely on labeled data, is applicable to sequential decision problems, and is widely used in game playing, robotics, autonomous driving, and resource scheduling.
Core Facts
The pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
奖励函数设计不当会导致奖励欺骗（Reward Hacking）现象，即智能体找到在奖励函数上得分高但实际无意义甚至有害的行为路径
Reinforcement learning learns optimal policies through continuous interaction between an agent and its environment; its mathematical foundation is the Markov Decision Process (MDP), which consists of a state space, action space, transition function, reward function, and discount factor.
Timeline (last 90 days)
Supervised learning requires humans to provide labeled data categories, unsupervised learning automatically categorizes and clusters data objects, and reinforcement learning guides model learning through a reward mechanism.
When the parallel scale rises to a certain level, the sampling strategy itself becomes more critical than the parallel scale.
For teams building large-scale RL pipelines, before increasing parallel environments, it is worth first examining the degree of match between the reset distribution and the policy's competence.
Dex-One2Many converts a single human video into serialized scene graphs, and uses these graphs to guide reinforcement learning.
The challenges of applying reinforcement learning in social systems such as transportation include the gap between simulation and reality, design biases in reward functions, and the computational cost of large-scale deployment.
MIT associate professor Cathy Wu's research direction is using reinforcement learning to provide computational tools for socially complex systems such as transportation, and her work spans the Department of Civil and Environmental Engineering and the Institute for Data, Systems, and Society.
Cathy Wu's team's reinforcement learning method does not require perfect a priori modeling of the transportation system, but instead gradually approaches better solutions through trial and error and reward signals.
Cathy Wu's methodology can be transferred to energy scheduling, supply chains, public health, and other domains equally full of uncertainty and multi-agent games.
In the dispatch strategies of ride-sharing platforms, reinforcement learning can simultaneously consider multiple objectives such as passenger waiting time, driver income, and road load, rather than optimizing only a single dimension.
On the question of how humans align children, imitation learning may be more fundamental than reinforcement learning.
40 more timeline events
All Facts (20)
The pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
90%Verified奖励函数设计不当会导致奖励欺骗（Reward Hacking）现象，即智能体找到在奖励函数上得分高但实际无意义甚至有害的行为路径
85%VerifiedReinforcement learning learns optimal policies through continuous interaction between an agent and its environment; its mathematical foundation is the Markov Decision Process (MDP), which consists of a state space, action space, transition function, reward function, and discount factor.
80%Verified大语言模型的预训练阶段本质上属于自监督学习，是一种特殊的非监督学习，模型通过预测下一个词来从海量文本中自动学习语言规律
80%VerifiedMachine learning is divided into three major categories: supervised learning, unsupervised learning, and semi-supervised learning.
80%VerifiedThe concept of world models originates from cognitive science and reinforcement learning, strongly promoted by Yann LeCun and others, referring to an agent's internal representation of the external environment.
80%VerifiedDQN是DeepMind在2013年提出的算法，首次将深度神经网络与Q-learning强化学习相结合，使AI能学会玩49款Atari游戏
80%VerifiedEach RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.
75%VerifiedIn sparse-reward scenarios, agents often need techniques such as curriculum learning, reward shaping, or intrinsic motivation-based exploration to converge effectively.
75%VerifiedIn reinforcement learning training, the model's goal is to maximize the score given by the reward function.
75%VerifiedWhen language models are repeatedly trained on data generated by themselves or similar models, systematic degradation occurs, with the diversity of model outputs decreasing with each generation and the tail information of the distribution gradually being lost.
75%VerifiedAI misalignment behavior is fundamentally a direct product of current mainstream training paradigms, particularly reinforcement learning.
75%VerifiedReinforcement learning training is the key stage in which a model transitions from general language capability to precise task execution, further optimizing reasoning, planning, and instruction-following through reinforcement learning after large-scale pre-training.
75%VerifiedUnlike supervised learning, which relies on labeled data, reinforcement learning does not require humans to provide correct answers; instead, it allows agents to autonomously discover optimal behavioral paths through trial and error.
75%VerifiedThe core challenge of reinforcement learning lies in the exploration-exploitation tradeoff, where agents need to find a balance between exploiting known effective strategies and exploring unknown possibilities.
75%VerifiedIn-context learning was first observed in large language models, and its capability fundamentally stems from the diversity of pretraining data.
75%VerifiedThe reasoning paradigm is essentially the application of reinforcement learning to the post-training stage of language models, allowing the model to autonomously explore reasoning paths and receive reward feedback based on answer correctness, first publicly demonstrated in the o1 series of models.
75%VerifiedGoogle's DeepMind applied reinforcement learning to data center cooling systems in 2016, reducing cooling energy consumption by 40%.
75%VerifiedGoogle DeepMind has proposed methods related to Nested Learning.
70%VerifiedReinforcement learning is a machine learning paradigm that learns optimal policies through interaction with an environment, with the agent-environment loop at its core.
70%