Unverified50% confidenceFactExact time
Gradient accumulation splits a large batch into several micro-batches that are run forward and backward in sequence, accumulating gradients before updating parameters all at once, thereby reducing peak GPU memory while keeping the effective batch size unchanged.
1
Sources
50%
Confidence
Long-term
Relevance
10/31/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen batch_size scales linearly with the number of environments, the variance of the gradient estimate decreases at a rate of about 1/batch_size.66% similarUnverifiedGradient clipping works by proportionally scaling down the entire gradient vector when its norm exceeds a preset threshold, so that its norm equals exactly that threshold.66% similarUnverifiedThe reduced ancestor set compresses the graph traversal scale for a single gradient computation to polylogarithmic scale by truncating and retaining only a finite-depth ancestor set, with the bias introduced typically being controllable in practice.66% similarUnverifiedRepeatedly applying the same set of weights can easily cause gradients to explode or vanish across multiple iterations, similar to the gradient problems of early RNNs.66% similarVerifiedGradient accumulation accumulates gradients over multiple small batches before performing a single parameter update, with an effect equivalent to training with a large batch.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1119231API
curl https://kongchang.com/api/v1/knowledge/claims/1119231MCP
get_claim(id=1119231)