[KongchangAI]
Concept

ResQAdam

A memory-efficient adaptive optimizer for quantization-aware training (QAT) that maintains only BF16 momentum, applies Adam updates to the coordinates of the dominant singular subspace, and adds a residual normalized sign update, substantially reducing optimizer state memory overhead.

Timeline (last 90 days)

Nov 9

The paper's authors observed that even if only the sign of the residual part of the QAT momentum is retained, this information is still useful.

Unverified50%
Nov 9

ResQAdam maintains only a single BF16-precision momentum, applies the Adam update only to the r×r coordinates of the dominant singular subspace, and adds a normalized sign update computed from the residual.

Unverified50%
Nov 9

ResQAdam retains only the sign direction for the residual part, normalizes it, and adds it to the update, thereby avoiding storing the full momentum state.

Unverified50%
Nov 9

ResQAdam reduces optimizer state memory by 71% to 74% relative to AdamW.

Unverified50%
Nov 9

ResQAdam achieves accuracy close to AdamW in most settings.

Unverified50%
Nov 9

ResQAdam's training throughput drops by less than 2%.

Unverified50%
Nov 9

The paper evaluates using two from-scratch QAT methods and three QAT fine-tuning pipelines, and quantizes the weights to 2 bits.

Unverified50%

All Facts (7)

Source Articles