[KongchangAI]
Conceptternary weights / 三值量化

Ternary Weight

An extreme low-bit quantization method that constrains neural network weights to three values, {-1, 0, +1}, achieving compression in both storage and computation by reducing multiplications to addition and subtraction operations.

Timeline (last 90 days)

Nov 5

Ternary weights currently appear more in research and experimental models, and are still some distance from becoming the default configuration for mainstream open-source weights.

Unverified50%
Nov 5

Ternary weights constrain each weight to three values (usually negative one, zero, and positive one); weights can be represented with a very small bit width, and multiplication operations can degrade into addition, subtraction, and selection operations.

Unverified50%
Nov 5

Ternary weights improve arithmetic intensity by reducing weight bit width, thereby alleviating the memory bandwidth bottleneck, which is why they attract attention in edge and dedicated accelerator scenarios.

Unverified50%
Nov 5

MoE, hybrid layers, and ternary weights can be used in combination, and each also stands on its own.

Unverified50%
Oct 17

Ternary weights use absmean quantization: γ is computed as the mean of the absolute values of the weights, |w| <= γ/2 becomes 0, w > γ/2 becomes +1, and w < -γ/2 becomes -1; activations use symmetric absmax quantization.

Unverified50%
Oct 17

The follow-up paper "The Era of 1-bit LLMs (BitNet b1.58)" found that pure 1-bit requires very large scale to match full-precision quality, so it introduced a zero state, with weights becoming -1, 0 or +1.

Unverified50%
Oct 15

Microsoft Research's BitNet b1.58 paper proposes training language models from scratch with ternary weights {-1, 0, +1}, eliminating floating-point matrix multiplication at inference time and instead completing tensor operations with integer addition.

Unverified50%
Sep 17

Ternary weights refer to quantizing model weights to only three values (typically -1, 0, and +1).

Unverified50%
Sep 17

The multiplication operations of -1, 0, and +1 can be reduced to addition and sign flipping, which can lower power consumption and computational latency in dedicated hardware.

Unverified60%

All Facts (9)

Unverified

The multiplication operations of -1, 0, and +1 can be reduced to addition and sign flipping, which can lower power consumption and computational latency in dedicated hardware.

60%
Unverified

Ternary weights constrain each weight to three values (usually negative one, zero, and positive one); weights can be represented with a very small bit width, and multiplication operations can degrade into addition, subtraction, and selection operations.

50%
Unverified

Ternary weights currently appear more in research and experimental models, and are still some distance from becoming the default configuration for mainstream open-source weights.

50%
Unverified

MoE, hybrid layers, and ternary weights can be used in combination, and each also stands on its own.

50%
Unverified

Ternary weights improve arithmetic intensity by reducing weight bit width, thereby alleviating the memory bandwidth bottleneck, which is why they attract attention in edge and dedicated accelerator scenarios.

50%
Unverified

Ternary weights use absmean quantization: γ is computed as the mean of the absolute values of the weights, |w| <= γ/2 becomes 0, w > γ/2 becomes +1, and w < -γ/2 becomes -1; activations use symmetric absmax quantization.

50%
Unverified

The follow-up paper "The Era of 1-bit LLMs (BitNet b1.58)" found that pure 1-bit requires very large scale to match full-precision quality, so it introduced a zero state, with weights becoming -1, 0 or +1.

50%
Unverified

Microsoft Research's BitNet b1.58 paper proposes training language models from scratch with ternary weights {-1, 0, +1}, eliminating floating-point matrix multiplication at inference time and instead completing tensor operations with integer addition.

50%
Unverified

Ternary weights refer to quantizing model weights to only three values (typically -1, 0, and +1).

50%

Source Articles