Gradient Checkpointing
A deep learning memory optimization technique that stores only a subset of intermediate activations during the forward pass and recomputes the remaining activations during the backward pass in order to save GPU memory, at the cost of additional computation.
Timeline (last 90 days)
4-bit quantization, gradient checkpointing, and small-scale proxy-model pre-experiments can bring down the GPU requirements of experiments during the queueing period, reducing the number of trials once the cards are actually obtained.
Gradient checkpointing and mixed-precision training (FP16/BF16) are common memory-reduction measures.
Gradient Checkpointing trades recomputation for memory space and can reduce activation memory by approximately 60–70%.
Gradient checkpointing technology trades approximately 33% additional computational overhead for approximately 60–70% savings in GPU memory.
梯度检查点是一种用时间换空间的策略，通常能减少60-70%的显存占用，代价是约30%的额外计算时间
All Facts (5)
4-bit quantization, gradient checkpointing, and small-scale proxy-model pre-experiments can bring down the GPU requirements of experiments during the queueing period, reducing the number of trials once the cards are actually obtained.
50%UnverifiedGradient checkpointing and mixed-precision training (FP16/BF16) are common memory-reduction measures.
50%UnverifiedGradient Checkpointing trades recomputation for memory space and can reduce activation memory by approximately 60–70%.
50%UnverifiedGradient checkpointing technology trades approximately 33% additional computational overhead for approximately 60–70% savings in GPU memory.
50%Unverified梯度检查点是一种用时间换空间的策略，通常能减少60-70%的显存占用，代价是约30%的额外计算时间
50%