IQ3_S
An importance-aware quantization scheme proposed by Astra Labs within the llama.cpp ecosystem. Weights are stored at approximately 3 bits per parameter, and quantization error is reduced by allocating different precision levels according to each weight's importance. Performance remains close to BF16 full-precision inference.
Timeline (last 90 days)
IQ3_S is a non-uniform 3-bit integer quantization scheme that has greater precision loss compared to standard Q4 quantization but requires less storage space and VRAM.
IQ3_S shows a significant overall performance improvement compared to the earlier IQ3_XXS version.
In the name IQ3_S, the number 3 indicates that each weight uses approximately 3 bits of storage on average, and the suffix S represents a finer grouping strategy at the same bit count.
IQ3_S improves model output quality by allocating different precision levels to model weights based on their importance, allowing critical layers to retain higher precision, with almost no increase in computational complexity.
The IQ3_S model is about 7 GB larger than IQ3_XXS and has a slightly slower generation speed.
In the author's actual testing, the IQ3_S model consumed over 50 GB of memory at runtime, and memory usage spiked to 62 GB when opening the generated HTML and debugging.
For devices with tight memory, blindly using a larger quantization model may result in frequent OOM errors.
This update to Strata brings dual-card optimization and adds the IQ3_S quantization model to the official default configuration.
The benchmark performance of IQ3_S is essentially close to the original BF16 precision.
Two metrics in the published benchmark data for IQ3_S are anomalous, which may cause the average score to be inflated.
6 more timeline events
All Facts (16)
IQ3_S is a non-uniform 3-bit integer quantization scheme that has greater precision loss compared to standard Q4 quantization but requires less storage space and VRAM.
50%UnverifiedIQ3_S shows a significant overall performance improvement compared to the earlier IQ3_XXS version.
50%UnverifiedIQ3_S is one of the solutions in the Importance-aware Quantization series.
50%UnverifiedIn the name IQ3_S, the number 3 indicates that each weight uses approximately 3 bits of storage on average, and the suffix S represents a finer grouping strategy at the same bit count.
50%UnverifiedIQ3_S improves model output quality by allocating different precision levels to model weights based on their importance, allowing critical layers to retain higher precision, with almost no increase in computational complexity.
50%UnverifiedThe IQ3_S model is about 7 GB larger than IQ3_XXS and has a slightly slower generation speed.
50%UnverifiedIn the author's actual testing, the IQ3_S model consumed over 50 GB of memory at runtime, and memory usage spiked to 62 GB when opening the generated HTML and debugging.
50%UnverifiedFor devices with tight memory, blindly using a larger quantization model may result in frequent OOM errors.
50%UnverifiedThe benchmark performance of IQ3_S is essentially close to the original BF16 precision.
50%UnverifiedTwo metrics in the published benchmark data for IQ3_S are anomalous, which may cause the average score to be inflated.
50%UnverifiedIQ3_S and IQ3_XXS both belong to 3-bit quantization, but IQ3_S retains more precision for key weights and generally performs more stably in multilingual tasks, especially non-English scenarios, at the cost of a slightly larger file size.
50%UnverifiedRunning the 27B model with IQ3_S quantization on an RTX 5080, the generation speed for short conversations is approximately 48 tokens per second.
50%Unverified文章称IQ3_S档位通常能将准确率保留提升到90%以上、体积约13-15GB，Q4_K_M能保留95%以上准确率、体积约16-18GB
50%UnverifiedIQ2_XXS以约2.06位的平均比特率对权重编码，相比IQ3_S（约3.44位）或Q4_K_M（约4.83位）压缩率更高但信息损失更大
50%UnverifiedIQ2_XXS以约2.06位的平均比特率编码权重，IQ3_S约3.44位，Q4_K_M约4.83位
50%UnverifiedThis update to Strata brings dual-card optimization and adds the IQ3_S quantization model to the official default configuration.
50%