BitNet
A 1-bit/1.58-bit large language model architecture proposed by Microsoft Research that uses ternary weights (−1, 0, +1) and replaces matrix multiplications with integer additions and subtractions, substantially reducing inference computation and memory requirements for deployment on mobile ARM architectures.
Core Facts
Timeline (last 90 days)
The BitNet paper points out that the quality gap between 1-bit models and full-precision models shrinks as model scale increases.
During training, BitNet keeps high-precision latent weights (FP16/BF16), uses sign binarization in the forward pass, and uses STE in backpropagation to update the latent weights; the latent weights need sufficient resolution to accumulate tiny gradient updates.
BitNet replaces most of the Linear layers in a Transformer with BitLinear, with weights of ±1 (1 bit per weight) and activations of INT8.
PCEI is an open-source project whose full name is Physically Constrained Emergent Intelligence, combining three technologies—BitNet 1.58b, Mamba and FastKAN—for on-device embodied intelligence.
1-bit LLMs usually need to adopt a quantization-aware approach during the training stage (such as Microsoft's BitNet architecture and BitLinear layers), rather than simple rounding after training.
BitNet is positioned similarly to projects such as llama.cpp and vLLM, except that it specifically serves the extreme quantization scenario of 1-bit.
BitNet is explicitly positioned as an inference framework rather than a training framework; it is not responsible for training 1-bit models, but instead provides an efficient execution engine for existing 1-bit weights.
BitNet is Microsoft's open-source official inference framework, designed specifically for 1-bit large language models (1-bit LLMs) and written in C++.
BitNet is a representative example of a 1-bit quantization (binarization) scheme.
Before converting to ternary values, Prism ML applies a rotation matrix (orthogonal transformation) to smooth and disperse weight outliers so that the shared scale is more appropriate, an idea drawn from research such as BitNet and QuIP#.
12 more timeline events
All Facts (20)
Binarized neural network works such as the BitNet series typically operate on the premise of using low-bit-aware training from the training stage, rather than applying post-training quantization (PTQ) to an already fully trained full-precision model.
65%UnverifiedThe BitNet paper points out that the quality gap between 1-bit models and full-precision models shrinks as model scale increases.
50%UnverifiedDuring training, BitNet keeps high-precision latent weights (FP16/BF16), uses sign binarization in the forward pass, and uses STE in backpropagation to update the latent weights; the latent weights need sufficient resolution to accumulate tiny gradient updates.
50%UnverifiedBitNet replaces most of the Linear layers in a Transformer with BitLinear, with weights of ±1 (1 bit per weight) and activations of INT8.
50%UnverifiedPCEI is an open-source project whose full name is Physically Constrained Emergent Intelligence, combining three technologies—BitNet 1.58b, Mamba and FastKAN—for on-device embodied intelligence.
50%Unverified1-bit LLMs usually need to adopt a quantization-aware approach during the training stage (such as Microsoft's BitNet architecture and BitLinear layers), rather than simple rounding after training.
50%UnverifiedBitNet is positioned similarly to projects such as llama.cpp and vLLM, except that it specifically serves the extreme quantization scenario of 1-bit.
50%UnverifiedBitNet is explicitly positioned as an inference framework rather than a training framework; it is not responsible for training 1-bit models, but instead provides an efficient execution engine for existing 1-bit weights.
50%UnverifiedBitNet is Microsoft's open-source official inference framework, designed specifically for 1-bit large language models (1-bit LLMs) and written in C++.
50%UnverifiedBitNet is a representative example of a 1-bit quantization (binarization) scheme.
50%UnverifiedBefore converting to ternary values, Prism ML applies a rotation matrix (orthogonal transformation) to smooth and disperse weight outliers so that the shared scale is more appropriate, an idea drawn from research such as BitNet and QuIP#.
50%UnverifiedTernary quantization restricts weights to three values — -1, 0, and +1 — enabling addition and subtraction to replace multiplication in order to reduce hardware computational load, and is related to Microsoft's BitNet series of research.
50%UnverifiedThe project is more of a proof of concept, demonstrating the hardware lower bound reachable by combining extreme quantization techniques with distributed computing
50%UnverifiedWhen weights are only -1, 0, or +1, floating-point multiplication in matrix operations can be reduced to addition and subtraction, or zero weights can be skipped entirely
50%UnverifiedCompared to 16-bit floating point, ternary quantization can theoretically save approximately ten times the storage space
50%UnverifiedSHADOW-50M uses ternary weights, with all parameters constrained to three values {-1, 0, +1}, theoretically requiring only 1.58 bits per parameter.
50%Unverified实践中所谓的1-bit GGUF通常采用混合方案：embedding层和lm_head等关键组件保留较高精度，中间transformer层使用极低比特，整体平均比特率可能在1.5-2.0 bit/参数之间
50%Unverified1-bit量化研究可追溯至微软研究院2024年发布的BitNet论文，提出用三值权重（-1, 0, 1）替代全精度浮点数，BitNet b1.58进一步证明1.58-bit在从头训练场景下的可行性
50%UnverifiedBitNet b1.58采用三值量化，每个权重取-1、0或+1三个值，约1.58 bit（log₂3）
50%UnverifiedBitNet是微软研究院于2023年提出的架构级创新，核心思想是在Transformer架构中将线性层权重替换为1-bit表示
50%