Gradient Accumulation
A deep learning training technique for effectively increasing batch size under GPU memory constraints. A large batch is divided into several micro-batches that are processed sequentially through forward passes; gradients from each micro-batch are accumulated before a single unified parameter update is applied.
Timeline (last 90 days)
During large model training, operations such as gradient accumulation, momentum updates, and weight decay are essentially additions, and when weights are large while gradient updates are extremely small, some learning signals may be silently discarded.
在有限硬件条件下可使用混合精度训练、梯度累积等技巧来最大化资源利用效率
Non-deterministic GPU operations are designed to prioritize performance over determinism; forcing a fixed execution order introduces numerous synchronization barriers that significantly reduce parallel efficiency.
All Facts (3)
Non-deterministic GPU operations are designed to prioritize performance over determinism; forcing a fixed execution order introduces numerous synchronization barriers that significantly reduce parallel efficiency.
60%UnverifiedDuring large model training, operations such as gradient accumulation, momentum updates, and weight decay are essentially additions, and when weights are large while gradient updates are extremely small, some learning signals may be silently discarded.
50%Unverified在有限硬件条件下可使用混合精度训练、梯度累积等技巧来最大化资源利用效率
50%