buffer_size
A hyperparameter in reinforcement learning that specifies the size of the experience replay buffer. In on-policy algorithms such as PPO, it determines the number of experience steps collected before each policy update.
Timeline (last 90 days)
In ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
When the number of parallel environments expands from 10 to 1000 while buffer_size remains unchanged, the trajectory length allocated to each environment is compressed to 1/100 of the original, causing severe trajectory truncation and undermining GAE's estimation of returns.
When the number of parallel environments increases from N to M, batch_size should increase at least linearly by a ratio of M/N; buffer_size should remain 5 to 10 times batch_size.
All Facts (3)
In ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
50%UnverifiedWhen the number of parallel environments expands from 10 to 1000 while buffer_size remains unchanged, the trajectory length allocated to each environment is compressed to 1/100 of the original, causing severe trajectory truncation and undermining GAE's estimation of returns.
50%UnverifiedWhen the number of parallel environments increases from N to M, batch_size should increase at least linearly by a ratio of M/N; buffer_size should remain 5 to 10 times batch_size.
50%