[KongchangAI]
ConceptStochastic Rounding / 随机取整

Stochastic Rounding

A numerical technique used in low-precision computation that determines rounding direction probabilistically, with probability proportional to the magnitude of the rounding error, thereby maintaining unbiasedness in expectation and mitigating systematic drift in low-precision gradient accumulation.

Timeline (last 90 days)

Oct 13

Stochastic rounding chooses between two adjacent reconstruction levels with probability proportional to the fractional part, is unbiased in expectation, and is widely used in low-bit quantization to mitigate accumulated bias.

Unverified50%

All Facts (1)

Unverified

Stochastic rounding chooses between two adjacent reconstruction levels with probability proportional to the fractional part, is unbiased in expectation, and is widely used in low-bit quantization to mitigate accumulated bias.

50%

Source Articles

A New Approach to 4-bit AdamW Optimizer Quantization: Rounding in Preconditioner SpaceOct 13Format-Aware Fusion: The Engineering Practice That Doubles FP4 Pretraining ThroughputOct 4