Concept4-bit量化 / 优化器状态量化
4-bit Optimizer State Quantization
A technique that compresses optimizer states, such as the first and second moments of AdamW, from 32-bit to 4-bit, with the aim of reducing GPU memory overhead when training large models.
A technique that compresses optimizer states, such as the first and second moments of AdamW, from 32-bit to 4-bit, with the aim of reducing GPU memory overhead when training large models.