How Does 4-bit Quantization Change Model Refusal Behavior? Matched Perturbation Attribution Analysis

The impact of 4-bit quantization on chat model refusal behavior depends on specific rounding errors and can cause shifts in opposite directions.
A pre-registered study examined the impact of 4-bit quantization on the safety behavior of three 3-4B chat models and found that quantization is not simply added noise. Qwen2.5-3B more frequently generated harmful content when prefilled with compliant responses, while Llama-3.2-3B showed a 10.3 percentage point decrease in compliance rate on jailbreak attacks—opposite directions. Through matched perturbation experiments—shuffled errors, Gaussian noise, and scaled real errors—the researchers demonstrated that behavioral changes are related to the specific structure of the error, not driven by bit count; at half the real error, Llama's compliance rate had already dropped 5.8 percentage points while likelihood was almost unchanged. llama.cpp's Q4_K_M did not cause behavioral changes at the same bit count, highlighting implementation differences. The conclusion is that each quantized build must be compared separately with the original model, and safety audits should target the specific version.
Introduction: Quantization Is More Than Just Precision Loss
When we run chat models on phones or laptops, we typically use 4-bit quantized versions. Are these versions consistent with the original models in terms of safety behavior? A pre-registered study from rss gives a surprising answer: the impact of 4-bit quantization on model refusal behavior is not uniform noise, but depends on the specific rounding errors. The study conducted paired testing, blind human annotation, and an 8-bit null-hypothesis control on three 3-4B chat models, and found two 4-bit behavioral shifts in opposite directions.

4-bit quantization refers to compressing model weights from 16-bit floating point to 4-bit integer representation, usually with group scaling factors (e.g., one scale shared per 32 or 64 weights) to reduce precision loss. The core of the quantization process is rounding: each weight is mapped to the nearest discrete level, and the resulting error depends on that weight's relative position within the group. For most weights the error is small, but a few weights may encounter large rounding deviations, and these deviations can accumulate across layers and potentially change the model's output distribution. Quantization-aware training (QAT) or post-training quantization (PTQ) are common implementation paths, and PTQ methods widely used in chat model deployment (such as GPTQ, AWQ) at 4-bit typically keep language modeling perplexity close to the original model—but this does not guarantee that downstream safety behavior remains unchanged. This is precisely the gap this study addresses.
Core Findings: Two Opposite Shifts
The study focused on three 3-4B chat models. Through a rigorous pre-registered experimental design, it observed two significant 4-bit behavioral changes. Qwen2.5-3B, when prefilled with compliant responses, more frequently continued generating harmful content; while Llama-3.2-3B showed a 10.3 percentage point decrease in compliance rate on held-out jailbreak attacks. These two shifts are in opposite directions, indicating that quantization does not simply "reduce safety" or "improve safety," but varies by model and specific error.
This finding challenges the common assumption that "4-bit quantization just adds noise," suggesting that behavioral changes may stem from specific rounding error patterns.
Matched Perturbation Method: Separating Error from Bit Count
To distinguish whether behavioral changes come from specific rounding errors or from any perturbation of comparable magnitude, the researchers used a matched perturbation method. They transformed the real quantization errors: shuffled and Gaussian versions of the error had averages close to zero, while mirroring and scaling the real error from its original size to twice its size produced behavioral changes that were approximately linear. This proves that behavioral changes are related to the specific structure of the error, rather than being driven solely by bit count.
You may not have noticed that at half the real error, Llama's likelihood was almost unchanged, but its compliance rate had already dropped by 5.8 percentage points. This suggests that behavioral changes may occur before likelihood changes, and traditional likelihood metrics are insufficient to capture safety behavior shifts.
Impact of Community Tools
The study also tested llama.cpp's Q4_K_M quantization method, which at the same bits per weight did not shift the behavior of the two models. This indicates that different quantization implementations (such as Q4_K_M) may produce different error patterns, and thus have different impacts on safety behavior. Therefore, one cannot generalize that 4-bit quantization necessarily leads to a certain safety outcome.
Matched perturbation is an experimental design concept in causal inference: construct control perturbations that are comparable in magnitude or distribution to the target variable (here, the real quantization error) but structurally different, thereby testing whether the effect is specific to the particular structure of the target variable. In this study, the researchers replaced the real rounding error with shuffled-order errors and Gaussian noise, whose average magnitudes were close to zero but which destroyed the spatial structure of the error; at the same time, they mirrored and scaled the real error and observed that behavioral changes were approximately linear with error magnitude. This design effectively distinguishes between "any perturbation of comparable magnitude will cause behavioral changes" and "only errors with a specific structure cause behavioral changes," and is a key methodological tool for demonstrating the causal role of quantization error structure.
Q4_K_M is a mixed-precision 4-bit quantization scheme in llama.cpp, belonging to the K-quant series. It does not apply a uniform 4-bit representation to all tensors, but retains higher precision (such as 5-bit or 6-bit) for some key weights (e.g., attention output layers or some matrices of the feedforward network), to improve perplexity while keeping the average bit count close to 4. For this reason, the error distribution of Q4_K_M differs from simple global 4-bit rounding error: its errors are partially suppressed in key layers, and the structure is more complex. This explains why, at the same "bits per weight," Q4_K_M may not induce the same behavioral shifts as uniform 4-bit quantization, and also suggests that quantization implementation details (rather than just bit count) are an important source of safety behavior differences.
Practical Implications: Each Build Needs Its Own Comparison
Based on these findings, the researchers emphasize: refusal behavior under 4-bit quantization depends on the specific rounding errors, not on bits per weight or likelihood. Therefore, each quantized build needs to be behaviorally compared with its original model, and one cannot assume that the quantized version will maintain the same safety characteristics. For chat models deployed on end devices, this means that safety audits must be conducted for the specific quantized version, rather than relying on evaluation results of the original model.
In addition, the study used blind human annotation and paired testing, which improved the credibility of the results. The pre-registered design also reduced the risk of p-hacking and provides a methodological reference for future research.
Conclusion and Outlook
This study reveals the complex impact of 4-bit quantization on model safety behavior: it is neither harmless noise nor consistent degradation, but a behavioral shift determined by specific rounding errors. In the future, the design of quantization methods may need to consider the preservation of safety behavior, not just the trade-off between precision and speed. For developers and auditors, this means developing more fine-grained quantization evaluation tools to ensure the reliability of quantized models in safety-critical scenarios.
Key Points
- 4-bit quantization can cause model refusal behavior to shift in opposite directions, depending on the model and specific error.
- Matched perturbation experiments show that behavioral changes are related to error structure, not bit count or likelihood.
- At half the real error, Llama's compliance rate dropped by 5.8 percentage points, but likelihood was almost unchanged.
- llama.cpp's Q4_K_M quantization did not cause behavioral changes at the same bit count, highlighting implementation differences.
- Each quantized build needs to be compared separately with the original model, and safety audits should target the specific version.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.