[KongchangAI]
Concept反向传播中更新 / in-backward optimizer

optimizer-in-backward

A training strategy in which, as soon as the gradient of a layer or tensor is computed during backpropagation, the optimizer immediately updates the corresponding parameters and releases the gradient, rather than performing a unified update after the entire backward pass completes.

Timeline (last 90 days)

Nov 3

optimizer-in-backward is a training strategy that advances parameter updates to the backpropagation stage: once the gradients of a layer or tensor are computed, the optimizer immediately updates the parameters and releases the gradients, saving the memory of gradient copies.

Unverified50%
Nov 3

Global norm clipping must be executed only after all gradients are computed, because the global norm needs to be aggregated first, which inherently conflicts with the "release immediately after computation" of optimizer-in-backward.

Unverified50%

All Facts (2)

Source Articles