ExLlamaV3
An open-source large language model inference library maintained by turboderp. It interprets EXL3 quantized weights and performs forward computation, and is the third generation of the ExLlama series.
Timeline (last 90 days)
The working solution relies on a combination: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote the vllm-exl3 bridge layer, Diffbot designed the per-expert split scheme, startup flow and kernels, and InferenceKey did the integration and measurement.
EXL3 is a 3-bit quantization format introduced by ExLlamaV3 and is the successor to EXL2; ExLlamaV3 is an inference library maintained by turboderp; vllm-exl3 is a bridge layer written by Victor Cruz that connects ExLlamaV3's quantization kernels into vLLM's scheduling and PagedAttention system.
EXL3 uses vector and lattice quantization derived from QTIP, retaining more information at the same bit count than scalar rounding.
All Facts (3)
The working solution relies on a combination: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote the vllm-exl3 bridge layer, Diffbot designed the per-expert split scheme, startup flow and kernels, and InferenceKey did the integration and measurement.
50%UnverifiedEXL3 is a 3-bit quantization format introduced by ExLlamaV3 and is the successor to EXL2; ExLlamaV3 is an inference library maintained by turboderp; vllm-exl3 is a bridge layer written by Victor Cruz that connects ExLlamaV3's quantization kernels into vLLM's scheduling and PagedAttention system.
50%UnverifiedEXL3 uses vector and lattice quantization derived from QTIP, retaining more information at the same bit count than scalar rounding.
50%