Ternary Weight
An extreme low-bit quantization method that constrains neural network weights to three values, {-1, 0, +1}, achieving compression in both storage and computation by reducing multiplications to addition and subtraction operations.
Timeline (last 90 days)
Ternary weights currently appear more in research and experimental models, and are still some distance from becoming the default configuration for mainstream open-source weights.
Ternary weights constrain each weight to three values (usually negative one, zero, and positive one); weights can be represented with a very small bit width, and multiplication operations can degrade into addition, subtraction, and selection operations.
Ternary weights improve arithmetic intensity by reducing weight bit width, thereby alleviating the memory bandwidth bottleneck, which is why they attract attention in edge and dedicated accelerator scenarios.
MoE, hybrid layers, and ternary weights can be used in combination, and each also stands on its own.
Ternary weights use absmean quantization: γ is computed as the mean of the absolute values of the weights, |w| <= γ/2 becomes 0, w > γ/2 becomes +1, and w < -γ/2 becomes -1; activations use symmetric absmax quantization.
The follow-up paper "The Era of 1-bit LLMs (BitNet b1.58)" found that pure 1-bit requires very large scale to match full-precision quality, so it introduced a zero state, with weights becoming -1, 0 or +1.
Microsoft Research's BitNet b1.58 paper proposes training language models from scratch with ternary weights {-1, 0, +1}, eliminating floating-point matrix multiplication at inference time and instead completing tensor operations with integer addition.
Ternary weights refer to quantizing model weights to only three values (typically -1, 0, and +1).
The multiplication operations of -1, 0, and +1 can be reduced to addition and sign flipping, which can lower power consumption and computational latency in dedicated hardware.
All Facts (9)
The multiplication operations of -1, 0, and +1 can be reduced to addition and sign flipping, which can lower power consumption and computational latency in dedicated hardware.
60%UnverifiedTernary weights constrain each weight to three values (usually negative one, zero, and positive one); weights can be represented with a very small bit width, and multiplication operations can degrade into addition, subtraction, and selection operations.
50%UnverifiedTernary weights currently appear more in research and experimental models, and are still some distance from becoming the default configuration for mainstream open-source weights.
50%UnverifiedMoE, hybrid layers, and ternary weights can be used in combination, and each also stands on its own.
50%UnverifiedTernary weights improve arithmetic intensity by reducing weight bit width, thereby alleviating the memory bandwidth bottleneck, which is why they attract attention in edge and dedicated accelerator scenarios.
50%UnverifiedTernary weights use absmean quantization: γ is computed as the mean of the absolute values of the weights, |w| <= γ/2 becomes 0, w > γ/2 becomes +1, and w < -γ/2 becomes -1; activations use symmetric absmax quantization.
50%UnverifiedThe follow-up paper "The Era of 1-bit LLMs (BitNet b1.58)" found that pure 1-bit requires very large scale to match full-precision quality, so it introduced a zero state, with weights becoming -1, 0 or +1.
50%UnverifiedMicrosoft Research's BitNet b1.58 paper proposes training language models from scratch with ternary weights {-1, 0, +1}, eliminating floating-point matrix multiplication at inference time and instead completing tensor operations with integer addition.
50%UnverifiedTernary weights refer to quantizing model weights to only three values (typically -1, 0, and +1).
50%