BitLinear
A module in BitNet that replaces the standard linear layer, with weights quantized to 1-bit or 1.58-bit ternary values.
Timeline (last 90 days)
BitNet replaces most of the Linear layers in a Transformer with BitLinear, with weights of ±1 (1 bit per weight) and activations of INT8.
BitLinear keeps two scaling factors, one for weights and one for activations, to restore the scale information lost in quantization.
1-bit LLMs usually need to adopt a quantization-aware approach during the training stage (such as Microsoft's BitNet architecture and BitLinear layers), rather than simple rounding after training.
All Facts (3)
BitNet replaces most of the Linear layers in a Transformer with BitLinear, with weights of ±1 (1 bit per weight) and activations of INT8.
50%UnverifiedBitLinear keeps two scaling factors, one for weights and one for activations, to restore the scale information lost in quantization.
50%Unverified1-bit LLMs usually need to adopt a quantization-aware approach during the training stage (such as Microsoft's BitNet architecture and BitLinear layers), rather than simple rounding after training.
50%