ML-Agents
An open-source machine learning toolkit from Unity that lets developers train agents within Unity environments. It supports reinforcement learning algorithms such as PPO and is commonly used for robot simulation training.
Core Facts
Timeline (last 90 days)
When using shared_critic: False, consider changing it to True with a large number of parallel environments to reduce the number of parameters.
In ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
When the number of parallel environments expands from 10 to 1000 while buffer_size remains unchanged, the trajectory length allocated to each environment is compressed to 1/100 of the original, causing severe trajectory truncation and undermining GAE's estimation of returns.
When the number of parallel environments increases from N to M, batch_size should increase at least linearly by a ratio of M/N; buffer_size should remain 5 to 10 times batch_size.
When the number of environments increases from 10 to 1000, if an MLP with hidden_units of 512 and num_layers of 3 has insufficient capacity, the policy network will degenerate into outputting approximately average actions for all states, manifesting as being unable to learn or as stagnant rewards.
For scenarios with 100 to 1000 parallel robots, the recommended ML-Agents configuration is: batch_size 8192-32768, buffer_size 81920-327680, learning_rate 0.0001-0.0005, time_horizon 200-500, summary_freq 100000-300000, and max_steps kept at 30000000 or appropriately increased.
The square root scaling formula for the learning rate is: learning rate multiplied by sqrt(M/N); when scaling from 10 environments to 1000, the learning rate is adjusted to 0.0003 * sqrt(100) = 0.003, but this may be too aggressive in ML-Agents, so it is recommended to start by trying 1.5-2 times.
Enabling threaded: True may help with CPU parallel sampling under multiple environments in ML-Agents.
ML-Agents is Unity's reinforcement learning plugin.
All Facts (12)
ML-Agents is an officially open-sourced machine learning framework from Unity that enables agents to undergo reinforcement learning training by interacting with the game environment to receive reward signals.
65%UnverifiedML-Agents支持近端策略优化（PPO）、软演员-评论家（SAC）等主流算法，并可与Python生态中的PyTorch深度集成
90%Unverified强化学习训练出的NPC相比传统行为树能够在复杂动态环境中涌现出设计者未预设的行为策略
80%UnverifiedWhen the number of parallel environments increases from N to M, batch_size should increase at least linearly by a ratio of M/N; buffer_size should remain 5 to 10 times batch_size.
50%UnverifiedWhen the number of environments increases from 10 to 1000, if an MLP with hidden_units of 512 and num_layers of 3 has insufficient capacity, the policy network will degenerate into outputting approximately average actions for all states, manifesting as being unable to learn or as stagnant rewards.
50%UnverifiedFor scenarios with 100 to 1000 parallel robots, the recommended ML-Agents configuration is: batch_size 8192-32768, buffer_size 81920-327680, learning_rate 0.0001-0.0005, time_horizon 200-500, summary_freq 100000-300000, and max_steps kept at 30000000 or appropriately increased.
50%UnverifiedThe square root scaling formula for the learning rate is: learning rate multiplied by sqrt(M/N); when scaling from 10 environments to 1000, the learning rate is adjusted to 0.0003 * sqrt(100) = 0.003, but this may be too aggressive in ML-Agents, so it is recommended to start by trying 1.5-2 times.
50%UnverifiedEnabling threaded: True may help with CPU parallel sampling under multiple environments in ML-Agents.
50%UnverifiedWhen using shared_critic: False, consider changing it to True with a large number of parallel environments to reduce the number of parameters.
50%UnverifiedIn ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
50%UnverifiedWhen the number of parallel environments expands from 10 to 1000 while buffer_size remains unchanged, the trajectory length allocated to each environment is compressed to 1/100 of the original, causing severe trajectory truncation and undermining GAE's estimation of returns.
50%UnverifiedML-Agents is Unity's reinforcement learning plugin.
50%