1 related articles
bf16 precision can cause only small weights to update during fine-tuning while large weights freeze. Several highly cited papers are affected; Quiet-STaR reproduction shows high-precision optimizers need smaller learning rates to match results.