Quantization or Permutations? Choosing the Inference Budget for Local Small Models

Under a fixed inference budget, 6-bit weight quantization plus up to four-order averaging is the most cost-effective strategy.
A new study simulates weight round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B and tests option-order averaging to determine whether a fixed inference budget should go to bit count or permutation count. Comparing 8, 6, 4, and 3 bits against fp16 on ARC, MMLU, and HellaSwag, it finds 6-bit quantization preserves or slightly improves accuracy while 8-bit degrades; 4-bit greatly increases answer-slot skew, though a Gaussian noise control cannot confirm rounding adds extra bias. Averaging four cyclic orders recovers +12.6 points at 4-bit. Cost matching shows more permutations beat more bits only at 6 bits or above. The recommended strategy is 6-bit weights plus up to four-order averaging, but results are based on small-model simulation and need real-world validation.
Introduction
Running language models locally always comes with limited memory and compute budgets. A common way to save memory is to reduce weight precision (quantization), while a way to reduce sensitivity to answer-option order is to average over multiple reorderings (permutations). Can you have both? Or does quantization worsen order bias? A new study simulates weight round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B and tests the effect of option-order averaging, attempting to answer whether, under a fixed inference budget, resources should be spent on bit count or on number of permutations.

Experimental Setup
The study uses simulated round-to-nearest weight quantization, comparing the accuracy of 8, 6, 4, and 3 bits against an fp16 reference. Evaluation datasets include ARC-Easy, ARC-Challenge, MMLU, and HellaSwag, with 300 samples each. Model outputs are measured via the logits for letters A through D to determine answer selection. The study also introduces Gaussian noise as a control, adding noise at the same weight signal-to-noise ratio (SNR) to observe whether quantization-specific rounding error introduces additional order bias.
Weight round-to-nearest quantization is one of the most basic post-training quantization methods: floating-point weights are scaled, rounded to the nearest discrete integer grid, and then scaled back to approximate floating-point values. The error source is additive noise introduced by the rounding operation, typically assumed to be independent of weight magnitude. Simulated quantization temporarily inserts rounding nodes into the floating-point computation graph without changing the underlying hardware, and is used to quickly assess the impact of quantization on accuracy. This article adopts this simulation method to compare different bit counts and introduces a Gaussian noise control: random noise is added at the same weight SNR; if quantization and Gaussian noise produce comparable order bias, it suggests that rounding itself does not introduce effects beyond equal-power noise. This control design helps distinguish bias caused by "quantization-specific" effects from "general noise."
Answer-slot skew measures the degree of model preference for option positions (A/B/C/D slots), typically defined as the deviation between the frequency with which each slot is chosen as the answer and a uniform distribution. Order bias distorts model performance on multiple-choice questions, especially when the correct answer happens to fall in a slot the model prefers. This article mitigates this bias by cyclically permuting the options and averaging predictions; the effectiveness of permutation averaging depends on whether the bias is stable and can be canceled out by averaging.
Impact of Quantization on Accuracy and Order Bias
The results show that 6-bit quantization does not lose accuracy; instead, it is +1.8 and +0.9 percentage points higher than fp16, respectively (the study does not explain this phenomenon). 8-bit quantization, however, leads to accuracy drops of -0.7 and -0.4 percentage points. At 4-bit quantization, answer-slot skew rises from 0.15 to 0.53 (0.6B model) and from 0.07 to 0.12 (1.7B model). However, adding Gaussian noise at the same weight SNR also produced skews of 0.15 and 0.37 (two samplings) and 0.19 (one sampling), so the control experiment cannot determine whether rounding adds slot skew beyond equal-power noise. Flip rates rise but remain below the accuracy-matched baseline for independent uniform errors.
Effect of Permutation Averaging
Under 4-bit quantization, averaging four cyclic orders recovers +12.6 percentage points on the 0.6B model (+5.3 under fp16). This indicates that option-order averaging can effectively mitigate the order sensitivity introduced by quantization. But what about the cost?
Decision-Making Under Cost Matching
The study unifies bits and number of permutations into a machine-independent unit: weight bytes read times passes. In cost-matched pairs, increasing the number of permutations is better than increasing bit count only when weights have 6 bits or more. The decision table simplifies to: use 6-bit weights and spend the remaining budget on up to four orders, one pass per order.
Weight bytes read times passes is a hardware-independent cost proxy: for each answer generated, the model needs to read all quantized weight bytes once; if multiple permutation passes are performed, the total cost is the bytes read multiplied by the number of passes. This metric ignores factors such as compute units, caches, and parallelism, but it allows a fair comparison of the relative overhead of "increasing bit count" versus "increasing number of permutations." Under a fixed budget, increasing bit count linearly increases the read volume per pass, while increasing the number of permutations linearly increases the number of passes; the two are interchangeable in cost. Based on this, the article builds a decision table and finds that only when weights are greater than or equal to 6 bits does the marginal benefit of additional permutations exceed that of additional bits, because at low bit counts quantization noise dominates the order bias, and the benefit of more permutations is drowned out by high noise.
Limitations and Implications
All experiments are simulated on two small models and use the simplest quantizer. A title-level search found no prior research combining quantization with option order. This means there is still ample room for exploration in this area, especially for complex quantization schemes in real hardware deployments. For local model deployers, the current results indicate that 6-bit quantization is a sweet spot: it maintains accuracy while leaving budget for order averaging.
Conclusion
Under a fixed inference budget, 6-bit weight quantization plus averaging over up to four orders appears to be the most cost-effective strategy. However, note that this conclusion is based on a simulated environment and small models, and should be further validated before practical application.
One-line Summary
Under a fixed inference budget, 6-bit weight quantization plus averaging over up to four orders is the most cost-effective strategy.
Paragraph Summary
A new study simulates weight round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B and tests the effect of option-order averaging to answer whether, under a fixed inference budget, one should invest in bit count or number of permutations. Experiments compare 8, 6, 4, and 3 bits against fp16 and are evaluated on ARC, MMLU, and HellaSwag. Results show that 6-bit quantization does not lose accuracy and may even slightly improve it, while 8-bit actually degrades; 4-bit significantly increases answer-slot skew, but the Gaussian noise control cannot determine whether rounding adds extra bias. Averaging four cyclic orders recovers +12.6 percentage points under 4-bit, effectively mitigating order sensitivity. Cost-matching analysis shows that increasing permutations is better than increasing bits only when weights are at least 6 bits. The recommended strategy is 6-bit weights plus averaging over up to four orders, but the conclusion is based on small-model simulation and requires practical validation.
Related articles

Gemini 4 Hydrogen Released: A 2.5x Faster Everyday AI Model
Google releases Gemini 4 Hydrogen, claiming 2.5x the speed of Gemini 3.8 Flash, with four-tier thinking control, a million-token context, and Orchestrator multi-model collaboration. API pricing is $1.50 per million input tokens.

Gemini 4 Hydrogen Hands-On: Google's Fastest and Cheapest AI Model
Hands-on with Gemini 4 Hydrogen: the fastest, cheapest model in the Gemini 4 family. We break down the Carbon/Argon/Hydrogen split, API pricing traps, free-tier limits, and the cost trade-offs of four thinking levels—with a real bug-debugging demo.

Gemini 4 Hydrogen vs Sonnet Comparison Test: Uses Only 1% of Weekly Quota, Performance on Par
Hands-on test: Gemini 4 Hydrogen vs Sonnet on the same CRUD task—Hydrogen used only 1% weekly quota with comparable performance and better UI details, while Anthropic's model used 7%. Developer says daily tasks don't require expensive models.