The bf16 Precision Trap: Your Fine-Tuning May Only Be Training Some Weights

An Overlooked bf16 Behavior: Small Weights Update, Large Weights Freeze

If you store model weights in bf16 precision and fine-tune directly with torch.optim.AdamW, you may encounter a strange phenomenon: only the smaller-magnitude weights are being updated, while the larger weights stay nearly unchanged.

This isn't a bug—it's an inevitable consequence of bf16's representation precision. bf16 (bfloat16) has only an 8-bit exponent and a 7-bit mantissa, which means it can only represent changes in weights down to about 1/512 of their magnitude. When the learning rate is small, the update computed by the optimizer is too small relative to large weights to be represented in bf16, so those updates are simply rounded to zero.

The result: you get a sparse fine-tuner—one that only trains small-magnitude weights, freezes the rest, and whose cutoff point is determined by your learning rate.

bf16 (bfloat16) is a 16-bit floating-point format proposed by Google Brain to accelerate deep learning training. It consists of 1 sign bit, 8 exponent bits, and 7 mantissa bits—the exponent matches fp32 (hence the same dynamic range), but the mantissa is drastically reduced. fp32 has a 23-bit mantissa, capable of representing about 7 decimal significant digits; bf16 has only a 7-bit mantissa, representing only about 2–3 decimal significant digits. This means bf16's relative precision is about 1/128 (2^{-7}), while fp32's is about 1 in 8 million (2^{-23}). When adding two numbers, if the smaller addend is less than 1/256 of the larger one (accounting for rounding), the result may be unchanged—this is called "absorption" or "rounding to zero." In the weight-update scenario, if the update is too small relative to the weight's magnitude, bf16 storage will round it away, preventing the weight from updating. This phenomenon is common in mixed-precision training, but the optimizer typically uses fp32 master weights to avoid it. Using bf16 weights and optimizer states directly amplifies the problem.

How Common Is the Problem? Several Highly Cited Papers Fall Prey

This isn't theoretical speculation. The original author's investigation found that at least half a dozen peer-reviewed papers with thousands of citations use this fine-tuning approach, yet none of them mention this behavior.

The author analyzed one such paper, Quiet-STaR, in depth for three reasons: the paper itself is interesting, the authors open-sourced the weights, and the code runs successfully. By comparing the base model and the fine-tuned model, the author confirmed: indeed, only small-magnitude weights were updated during training.

Subsequently, the author re-ran the Quiet-STaR training recipe with an optimizer capable of applying every update, and the model's performance diverged. To achieve the same results, the high-precision optimizer needed a learning rate 3 to 10 times smaller—and even then, it couldn't achieve better results.

Quiet-STaR is a method proposed by Eric Zelikman et al. in 2024 for training language models to perform internal reasoning. It extends the STaR (Self-Taught Reasoner) framework by having the model generate multiple "thinking" steps before each token, then optimizing these thoughts via reinforcement learning or rejection sampling to improve performance on complex reasoning tasks. Quiet-STaR's key innovation is seamlessly integrating the thinking process into standard language model training without additional supervision. The paper garnered attention for its novelty and potential impact, but the original author found that its open-source code fine-tunes directly using bf16 weights and the AdamW optimizer, potentially triggering the weight-freezing phenomenon unintentionally. This suggests that even cutting-edge research may overlook training-dynamics biases introduced by numerical precision.

Cross-Source Validation: From a Softmax Fix to a Reinterpretation of Training Dynamics

This finding is not isolated. According to the paper "A Fix for Softmax and Cross-Entropy" by Barron, Gnanapragasam, and Tschannen (October 2026), the standard cross-entropy computation rounds the gradient of the correct answer to exactly zero when the model is sufficiently confident (with a margin of about 5.5 under bf16, which they call the "precision cliff"). This rounding effect interacts with the Adam optimizer and is precisely the cause of the "slingshot" loss spikes observed during grokking. After fixing the rounding issue, the spikes disappear.

The two cases point in opposite directions: the cross-entropy rounding bug adds instability to training, while the bf16 weight-update rounding hides instability. Together they reveal a fact: the training dynamics we take for granted are sometimes just artifacts of numerical precision.

Grokking is a phenomenon in machine learning where a model's validation accuracy suddenly jumps dramatically long after the training loss has already converged. It typically appears on small-scale algorithmic tasks (such as modular arithmetic), where test performance surges after thousands of training steps. The mechanism behind grokking is still debated, but it is generally believed to relate to optimization dynamics, regularization, and representation learning. The "slingshot" loss spikes mentioned in the original article refer to brief, sharp fluctuations in the loss curve during grokking, followed by a sudden change in the model's generalization ability. The work by Barron et al. shows that standard cross-entropy rounds the gradient of the correct class to zero when the model is highly confident (with a logit margin of about 5.5 under bf16), interacting with the Adam optimizer to produce unstable spikes. After fixing the rounding, the spikes disappear, suggesting that some of grokking's dynamics may stem from numerical precision rather than pure learning mechanisms. This underscores the need to distinguish genuine learning signals from numerical artifacts when analyzing training phenomena.

Implications for Practitioners

If you're fine-tuning with bf16, here are a few things worth noting:

• Inspect the weight-update distribution: Compare the weights of the base model and the fine-tuned model to confirm whether only small weights have changed.
• Consider a mixed-precision strategy: Use fp32 for optimizer states and weight updates to avoid rounding loss.
• Adjust the learning rate: If you must use bf16, you may need a smaller learning rate to match the effect of full-precision optimization.
• Treat reproduction results with caution: The paper you're reproducing may have deviated from its original intent due to precision issues.

The author wrote up the full analysis in a long-form article, with code, logs, and evaluation results open-sourced on GitHub (repository: mweiss17/quietstar-bf16-adamw). For researchers and engineers working on fine-tuning, this is a precision trap worth examining carefully.

One-line Summary

Under bf16 precision, a small learning rate causes large-weight updates to be rounded to zero—several highly cited papers fall prey.

Paragraph Summary

The article reveals an overlooked bf16 precision issue: when weights are stored in bf16 and fine-tuned directly, a small learning rate makes the optimizer's computed updates too small relative to large weights to be represented in bf16, so these updates are rounded to zero. As a result, only small weights are trained, forming a "sparse fine-tuner." The author found that at least half a dozen highly cited papers (such as Quiet-STaR) use this fine-tuning approach without mentioning the behavior, and reproduction shows that a high-precision optimizer needs a 3–10x smaller learning rate to match results. Cross-source validation shows that a cross-entropy rounding bug interacting with Adam causes the loss spikes in grokking, which disappear after the fix. Together, the two cases demonstrate that some phenomena in training dynamics may simply be artifacts of numerical precision. Practitioners should inspect the weight-update distribution, consider mixed-precision strategies, adjust the learning rate, and treat reproduction results with caution.