ZeRO-3
A memory optimization technique proposed by DeepSpeed that partitions model parameters, gradients, and optimizer states across multiple GPUs, enabling training of very large models within limited memory.
Timeline (last 90 days)
In self-distillation, keeping two full model copies, teacher and student, resident on the GPU at the same time increases forward-pass latency and exacerbates ZeRO-3 communication overhead.
Adapter switching itself must not trigger parameter reallocation or cache invalidation, otherwise it will cause memory spikes under ZeRO-3.
All Facts (2)
In self-distillation, keeping two full model copies, teacher and student, resident on the GPU at the same time increases forward-pass latency and exacerbates ZeRO-3 communication overhead.
50%UnverifiedAdapter switching itself must not trigger parameter reallocation or cache invalidation, otherwise it will cause memory spikes under ZeRO-3.
50%