llm-compressor
An open-source library for quantization and compression of large language models, supporting advanced quantization workflows such as dataset calibration.
Timeline (last 90 days)
vLLM's `fp8` KV cache format corresponds to E4M3 and offers three calibration methods: no calibration (scaling factor defaults to 1.0), random-token warmup, and dataset calibration via llm-compressor.
In production, it is recommended to use llm-compressor for dataset calibration to obtain the FP8 KV cache scaling factors rather than relying on the default scaling factor of 1.0.
llm-compressor is a model quantization tool library in the vLLM ecosystem that supports FP8, INT8 and other quantization of weights and activations and provides dataset-based calibration workflows.
All Facts (3)
vLLM's `fp8` KV cache format corresponds to E4M3 and offers three calibration methods: no calibration (scaling factor defaults to 1.0), random-token warmup, and dataset calibration via llm-compressor.
50%UnverifiedIn production, it is recommended to use llm-compressor for dataset calibration to obtain the FP8 KV cache scaling factors rather than relying on the default scaling factor of 1.0.
50%Unverifiedllm-compressor is a model quantization tool library in the vLLM ecosystem that supports FP8, INT8 and other quantization of weights and activations and provides dataset-based calibration workflows.
50%