SAC
A maximum-entropy reinforcement learning algorithm proposed by a UC Berkeley team in 2018. As an off-policy algorithm, it can reuse historical experience and demonstrates strong performance in balancing exploration and exploitation.
Core Facts
Stable-Baselines3 is rebuilt on PyTorch and provides production-grade implementations of algorithms such as PPO, DQN, and SAC.
Model-free reinforcement learning is sample-inefficient, often requiring millions or even hundreds of millions of interactions to converge.
In a reproduction experiment training for 40 epochs on CIFAR-10, plain-56 achieved a training accuracy of 84.0% and plain-20 achieved a training accuracy of 95.1%, even though plain-56's parameter count (853,018) is approximately three times that of plain-20 (269,722).
Timeline (last 90 days)
SAC is an off-policy algorithm that can repeatedly utilize historical data from the experience replay buffer and incorporates an entropy regularization term in the reward, resulting in higher sample efficiency but more complex hyperparameter tuning
SAC (Senior Area Chair) in the NeurIPS review system is responsible for overseeing the work of multiple regular Area Chairs and has considerable influence over final acceptance decisions.
The NeurIPS organizing committee has historically provided SACs with priority registration channels or exclusive ticketing arrangements as compensation for their volunteer contributions.
适当缩短回放池长度或使用优先经验回放、定期检查Q值量级，是维持SAC长时间训练稳定性的工程手段
排查SAC训练崩溃应监控策略熵和α变化曲线，若熵持续下降至接近零说明探索消亡，可固定α为较大常数或重设目标熵为-action_dim
经验回放池中旧数据与新策略的分布偏移会使Bellman目标估计方差上升，可能触发critic梯度爆炸或Q值发散
SAC通过取两个Q网络中较小值的Clipped Double-Q技巧来缓解Q函数过估计偏差
SAC的自动调节机制中，目标熵通常设为-dim(A)，即动作空间维度的负值
SAC（Soft Actor-Critic）的核心机制是最大熵强化学习，在优化奖励的同时鼓励策略保持随机性，由温度系数α控制探索与利用的平衡
若自动温度调节的目标熵设置不当，α可能被优化到过小的值导致探索动机消失，策略坍缩到确定性输出
17 more timeline events
All Facts (20)
Stable-Baselines3 is rebuilt on PyTorch and provides production-grade implementations of algorithms such as PPO, DQN, and SAC.
75%VerifiedModel-free reinforcement learning is sample-inefficient, often requiring millions or even hundreds of millions of interactions to converge.
65%VerifiedIn a reproduction experiment training for 40 epochs on CIFAR-10, plain-56 achieved a training accuracy of 84.0% and plain-20 achieved a training accuracy of 95.1%, even though plain-56's parameter count (853,018) is approximately three times that of plain-20 (269,722).
65%VerifiedPPO（近端策略优化）以其训练稳定性成为连续控制任务的事实标准，SAC（软演员-评论家）通过最大熵框架在样本效率上具有优势
65%UnverifiedML-Agents支持近端策略优化（PPO）、软演员-评论家（SAC）等主流算法，并可与Python生态中的PyTorch深度集成
90%UnverifiedStable-Baselines3 is one of the most popular reinforcement learning algorithm libraries currently available, encapsulating mainstream algorithms such as PPO, SAC, and TD3.
60%UnverifiedSAC is an off-policy algorithm that can repeatedly utilize historical data from the experience replay buffer and incorporates an entropy regularization term in the reward, resulting in higher sample efficiency but more complex hyperparameter tuning
50%UnverifiedSAC (Senior Area Chair) in the NeurIPS review system is responsible for overseeing the work of multiple regular Area Chairs and has considerable influence over final acceptance decisions.
50%UnverifiedThe NeurIPS organizing committee has historically provided SACs with priority registration channels or exclusive ticketing arrangements as compensation for their volunteer contributions.
50%Unverified在强化学习中，更长的训练时间不等于更好的结果，SAC与DIAYN组合存在策略坍塌风险，性能峰值往往出现在训练中段
50%UnverifiedDIAYN内在奖励尺度若与SAC熵项量级不匹配会导致一方主导优化，应适当缩放判别器奖励使两者量级相当
50%UnverifiedSAC（Soft Actor-Critic）的核心机制是最大熵强化学习，在优化奖励的同时鼓励策略保持随机性，由温度系数α控制探索与利用的平衡
50%Unverified若自动温度调节的目标熵设置不当，α可能被优化到过小的值导致探索动机消失，策略坍缩到确定性输出
50%UnverifiedSAC的自动调节机制中，目标熵通常设为-dim(A)，即动作空间维度的负值
50%UnverifiedSAC通过取两个Q网络中较小值的Clipped Double-Q技巧来缓解Q函数过估计偏差
50%Unverified经验回放池中旧数据与新策略的分布偏移会使Bellman目标估计方差上升，可能触发critic梯度爆炸或Q值发散
50%Unverified排查SAC训练崩溃应监控策略熵和α变化曲线，若熵持续下降至接近零说明探索消亡，可固定α为较大常数或重设目标熵为-action_dim
50%Unverified适当缩短回放池长度或使用优先经验回放、定期检查Q值量级，是维持SAC长时间训练稳定性的工程手段
50%UnverifiedStable-Baselines3 is a PyTorch-based reinforcement learning algorithm library that encapsulates mainstream algorithms such as PPO, SAC, and TD3 and provides a unified API interface.
50%UnverifiedSAC is suitable for online fine-tuning on real robots due to its higher sample efficiency.
50%