ResQAdam
A memory-efficient adaptive optimizer for quantization-aware training (QAT) that maintains only BF16 momentum, applies Adam updates to the coordinates of the dominant singular subspace, and adds a residual normalized sign update, substantially reducing optimizer state memory overhead.
Timeline (last 90 days)
The paper's authors observed that even if only the sign of the residual part of the QAT momentum is retained, this information is still useful.
ResQAdam maintains only a single BF16-precision momentum, applies the Adam update only to the r×r coordinates of the dominant singular subspace, and adds a normalized sign update computed from the residual.
ResQAdam retains only the sign direction for the residual part, normalizes it, and adds it to the update, thereby avoiding storing the full momentum state.
ResQAdam reduces optimizer state memory by 71% to 74% relative to AdamW.
ResQAdam achieves accuracy close to AdamW in most settings.
ResQAdam's training throughput drops by less than 2%.
The paper evaluates using two from-scratch QAT methods and three QAT fine-tuning pipelines, and quantizes the weights to 2 bits.
All Facts (7)
The paper's authors observed that even if only the sign of the residual part of the QAT momentum is retained, this information is still useful.
50%UnverifiedResQAdam maintains only a single BF16-precision momentum, applies the Adam update only to the r×r coordinates of the dominant singular subspace, and adds a normalized sign update computed from the residual.
50%UnverifiedResQAdam retains only the sign direction for the residual part, normalizes it, and adds it to the update, thereby avoiding storing the full momentum state.
50%UnverifiedResQAdam reduces optimizer state memory by 71% to 74% relative to AdamW.
50%UnverifiedResQAdam achieves accuracy close to AdamW in most settings.
50%UnverifiedResQAdam's training throughput drops by less than 2%.
50%UnverifiedThe paper evaluates using two from-scratch QAT methods and three QAT fine-tuning pipelines, and quantizes the weights to 2 bits.
50%