INT8 Quantization
A model compression technique that quantizes model weights from 16-bit floating point to 8-bit integers. It can reduce VRAM usage by roughly half and increase inference speed, but introduces a small loss in accuracy.
Timeline (last 90 days)
Elastic by default applies int8 quantization to float32 vectors to reduce memory usage, BBQ further compresses vectors to nearly one bit, and disk BBQ nearly eliminates the memory overhead of vector search.
int8 quantization can compress vector size to about a quarter of float32, but recall must be re-evaluated to confirm the accuracy loss is acceptable.
Running inner-loop gradients in int8 causes the fast weights to diverge; the solution is to keep the inner loop in fp16 on the device and quantize only the forward path.
Model loading is divided into three categories: Clip model (select the int8 version), UNet large model (minimax_h3_int8 version), and VAE decoder.
The Singularity ref2va int8 checkpoint is used to reduce VRAM usage through int8 quantization.
The experiment employed a technical configuration of int8 quantization, 32-step sampling, and 1344×768 resolution.
int8 quantization is a model compression technique that compresses neural network weights from 32-bit or 16-bit floating point numbers to 8-bit integers, reducing VRAM usage by approximately 50%–75%.
All Facts (7)
int8 quantization is a model compression technique that compresses neural network weights from 32-bit or 16-bit floating point numbers to 8-bit integers, reducing VRAM usage by approximately 50%–75%.
60%UnverifiedElastic by default applies int8 quantization to float32 vectors to reduce memory usage, BBQ further compresses vectors to nearly one bit, and disk BBQ nearly eliminates the memory overhead of vector search.
50%Unverifiedint8 quantization can compress vector size to about a quarter of float32, but recall must be re-evaluated to confirm the accuracy loss is acceptable.
50%UnverifiedRunning inner-loop gradients in int8 causes the fast weights to diverge; the solution is to keep the inner loop in fp16 on the device and quantize only the forward path.
50%UnverifiedModel loading is divided into three categories: Clip model (select the int8 version), UNet large model (minimax_h3_int8 version), and VAE decoder.
50%UnverifiedThe Singularity ref2va int8 checkpoint is used to reduce VRAM usage through int8 quantization.
50%UnverifiedThe experiment employed a technical configuration of int8 quantization, 32-step sampling, and 1344×768 resolution.
50%