Guarded Moment Compression
A method for compressing Adam optimizer states that verifies, through shared gradient bounds, whether the decoded compressed moments reproduce the native FP32 update within a tolerance, and accepts compression only for coordinates that are safe.
Timeline (last 90 days)
"Guarded Moment Compression" shares a gradient bound to guarantee that the compressed and decoded moments can reproduce the native FP32 update, with error within a tolerance τ (in units of the learning rate); otherwise, that coordinate is stored exactly.
The guarded compression process includes, at encoding time, using 8-bit logarithmic encoding for each moment and checking whether the decoded update is within tolerance, and if not, storing the original FP32 value; at recovery time, decoding the moments and computing the update, ensuring that the first update after recovery matches the original update within tolerance.
The total encoding and recovery time of guarded compression is 2.3-2.6 times longer than guarded BF16, but shorter than guarded block-wise 8-bit.
Jointly searching the encodings of the two moments saves less than 0.001% in bit rate but increases encoding and decoding time by 6%, so per-moment rounding is already sufficient.
The accuracy guarantee of current guarded compression only covers the first recovery update under a fixed Torch operation model, and the stability of subsequent training behavior (such as multi-step updates) still relies on empirical verification.
All Facts (5)
"Guarded Moment Compression" shares a gradient bound to guarantee that the compressed and decoded moments can reproduce the native FP32 update, with error within a tolerance τ (in units of the learning rate); otherwise, that coordinate is stored exactly.
50%UnverifiedThe guarded compression process includes, at encoding time, using 8-bit logarithmic encoding for each moment and checking whether the decoded update is within tolerance, and if not, storing the original FP32 value; at recovery time, decoding the moments and computing the update, ensuring that the first update after recovery matches the original update within tolerance.
50%UnverifiedThe total encoding and recovery time of guarded compression is 2.3-2.6 times longer than guarded BF16, but shorter than guarded block-wise 8-bit.
50%UnverifiedJointly searching the encodings of the two moments saves less than 0.001% in bit rate but increases encoding and decoding time by 6%, so per-moment rounding is already sufficient.
50%UnverifiedThe accuracy guarantee of current guarded compression only covers the first recovery update under a fixed Torch operation model, and the stability of subsequent training behavior (such as multi-step updates) still relies on empirical verification.
50%