vllm-exl3
A bridge layer that integrates ExLlamaV3's quantization kernels into vLLM's scheduling and PagedAttention system, enabling vLLM to load models quantized in the EXL3 format.
Timeline (last 90 days)
The working solution relies on a combination: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote the vllm-exl3 bridge layer, Diffbot designed the per-expert split scheme, startup flow and kernels, and InferenceKey did the integration and measurement.
EXL3 is a 3-bit quantization format introduced by ExLlamaV3 and is the successor to EXL2; ExLlamaV3 is an inference library maintained by turboderp; vllm-exl3 is a bridge layer written by Victor Cruz that connects ExLlamaV3's quantization kernels into vLLM's scheduling and PagedAttention system.
All Facts (2)
The working solution relies on a combination: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote the vllm-exl3 bridge layer, Diffbot designed the per-expert split scheme, startup flow and kernels, and InferenceKey did the integration and measurement.
50%UnverifiedEXL3 is a 3-bit quantization format introduced by ExLlamaV3 and is the successor to EXL2; ExLlamaV3 is an inference library maintained by turboderp; vllm-exl3 is a bridge layer written by Victor Cruz that connects ExLlamaV3's quantization kernels into vLLM's scheduling and PagedAttention system.
50%