Stochastic Depth
A regularization training technique that randomly skips residual branches with a certain probability during training, making the network equivalent to an ensemble of many shallow subnetworks, alleviating overfitting and accelerating training.
Timeline (last 90 days)
The new paper proposes a two-stage scheme: pretraining uses random depth to induce contractive residuals, and post-training uses BeNice! to broadcast the KV cache and achieve KV sharing.
Introducing random depth during pretraining induces contractive residual dynamics, pulling the output of each loop toward a common fixed point so that representations across loops become increasingly similar.
All Facts (2)
The new paper proposes a two-stage scheme: pretraining uses random depth to induce contractive residuals, and post-training uses BeNice! to broadcast the KV cache and achieve KV sharing.
50%UnverifiedIntroducing random depth during pretraining induces contractive residual dynamics, pulling the output of each loop toward a common fixed point so that representations across loops become increasingly similar.
50%