ExLlamaV3 v1.0.0 Official Release: Faster Inference, Lossless KV Quantization, and Streamlined Dependencies

ExLlamaV3 Reaches Its First Stable Release

After more than a year of continuous iteration, the local large language model inference engine ExLlamaV3 has officially launched v1.0.0 — its first production-ready stable release. This release was completed jointly by developer Turboderp and the Fable team. According to the Reddit community, Turboderp maintained nearly 10 hours of intensive work per day to deliver a sweeping set of performance improvements.

ExLlamaV3 is the third generation of the ExLlama project family, designed to efficiently run quantized large language models in GGUF/EXL2 formats on consumer-grade NVIDIA GPUs. Quantization is the technique of compressing model weights from high-precision floating-point formats like FP16/BF16 into INT8, INT4, or even lower-bit integer representations, dramatically reducing VRAM usage and memory bandwidth pressure with minimal loss in model quality. The ExLlama series stands alongside llama.cpp and vLLM as a core pillar of the local inference ecosystem, distinguished by its deep CUDA optimizations and particularly strong support for the EXL2 format.

For users who have long followed the local inference space, ExLlamaV3 has been one of the go-to solutions for running quantized large models on consumer GPUs. This v1.0.0 release not only signals the project's maturity, but also delivers meaningful advances in inference speed, VRAM efficiency, and model compatibility.

Leaner Dependencies: Goodbye flash-attention-2 and xformers

One of the most engineering-significant changes in this release is the complete removal of flash-attention-2 and xformers dependencies.

Flash Attention, proposed by Tri Dao and colleagues in 2022, works by using an IO-aware tiled computation strategy that moves attention calculations from GPU HBM (high-bandwidth memory) to SRAM, reducing the memory complexity of standard attention from O(N²) to O(N) and significantly improving throughput for long sequences. xFormers is Meta's open-source efficient Transformer component library, offering memory-efficient attention and other optimized operators. While both are powerful, they require compiling C++/CUDA extensions, which frequently causes build failures or version incompatibilities on Windows or non-standard CUDA environments — a major barrier to local deployment.

ExLlamaV3 replaces them with custom-built attention kernels, lowering the deployment barrier and greatly improving portability across different hardware and system environments. This is a pragmatic trade-off between engineering reliability and peak performance, and for users who regularly switch between Windows, Linux, and various CUDA versions, it's a genuine quality-of-life improvement.

The new version also removes the dependency on causal_conv1d, replacing it with a brand-new custom conv1d kernel, further reducing reliance on external libraries. This "dependency reduction" philosophy reflects the project's conscious effort to keep engineering complexity in check while still pursuing top-tier performance.

New Attention Kernel: KV Quantization No Longer Slows Down Inference

The technical centerpiece of this release is undoubtedly the completely rewritten attention kernel, which delivers several key capabilities:

Online Cache Quantization

The KV cache (Key-Value Cache) is the core data structure during the Transformer decoding phase, storing the attention key-value vectors for past tokens to avoid recomputation at each step. As context length grows, KV cache VRAM usage expands linearly — for LLaMA-3 70B, for example, the KV cache can easily consume tens of gigabytes at 128K context. Quantizing the KV cache (e.g., from FP16 down to INT4/INT8) is the mainstream approach to relieving VRAM pressure, but traditional implementations introduce additional computation overhead for dequantization, increasing inference latency.

The new kernel eliminates this overhead by fusing the quantization/dequantization operations directly into the attention CUDA kernel (Kernel Fusion). Because the smaller data footprint also reduces the memory bandwidth bottleneck, KV cache quantization no longer causes inference slowdowns — and in some bandwidth-constrained scenarios, it can actually improve throughput further.

This means users can run longer contexts within limited VRAM while maintaining or even improving inference performance — a particularly important benefit for running large models on consumer GPUs.

Dual-Input SWA and Attention Sinks

The new kernel also adds dual-input support for Sliding Window Attention (SWA) layers and introduces the attention sinks mechanism.

Sliding Window Attention (SWA), widely adopted by Mistral AI in its models, limits each token's attention range to a fixed-size local window, reducing attention complexity from O(N²) to O(N×W) (where W is the window size), greatly improving long-text inference efficiency. Attention Sinks come from MIT's 2023 "StreamingLLM" research — a small number of tokens at the start of a sequence (Sink Tokens) are disproportionately important to the attention scores of all subsequent positions, and retaining them prevents attention distribution collapse during extremely long streaming inference. Native support for both mechanisms allows ExLlamaV3 to handle modern architectures like Mistral and Gemma smoothly, laying a solid foundation for compatibility with even more new architectures.

Expanded Tensor Parallelism and Operator Performance Improvements

Broader Tensor Parallelism Support

Tensor Parallelism is one of the core strategies for multi-GPU distributed inference: a single large weight matrix is split along a specific dimension across multiple GPUs for parallel computation, with results merged via collective communication primitives like AllReduce. Unlike pipeline parallelism — which assigns different layers to different GPUs — tensor parallelism splits within a layer, offering lower communication latency but more complex implementation. For ultra-large models that can't fit on a single GPU (e.g., 70B+ parameters), tensor parallelism is the most common local solution.

v1.0.0 extends tensor parallelism support to most mainstream models, including Gemma4. This means users with multiple GPUs can fully leverage their multi-GPU desktop setups to run very large models without relying on server-side frameworks like vLLM.

Major GEMM/GEMV Performance Improvements

GEMM (General Matrix Multiplication) and GEMV (General Matrix-Vector Multiplication) are the two most computationally intensive operation types in large model inference: batch inference is dominated by GEMM (compute-bound), while single-request streaming decoding is dominated by GEMV (memory bandwidth-bound).

NVIDIA's Ampere architecture (RTX 30 series, including RTX 3090/3080) introduced third-generation Tensor Cores with hardware acceleration for TF32, BF16, and sparse matrix operations — currently the most prevalent consumer-grade high-performance GPU architecture on the secondhand market. The new version fine-tunes Thread Block sizes and shared memory usage strategies for the Ampere architecture, delivering major GEMM/GEMV optimizations along with a brand-new INT8 GEMV kernel that further reduces bandwidth pressure. Users on RTX 30 series cards should notice a clear improvement in per-token generation latency.

MoE Kernel Scheduler

The Mixture of Experts (MoE) architecture replaces the FFN layers in Transformers with multiple "expert" sub-networks, with a gating network dynamically selecting a small number (typically 2–8) to activate for each inference pass. This allows total parameter counts to be very large while keeping actual activated parameters relatively small — for example, Mixtral 8×22B has 141B total parameters but only activates approximately 39B per inference, balancing model capacity with inference efficiency. Recent star models including Mixtral, DeepSeek, and Gemma 4 all use this architecture.

However, MoE inference is challenging due to the irregularity of expert routing: different tokens may be routed to different experts, creating highly unbalanced GPU workloads. The newly added MoE kernel ticket scheduler addresses this load imbalance through a dynamic task allocation mechanism, optimizing the scheduling efficiency of expert routing and compute tasks — a forward-looking improvement with strong practical value.

Expanded Model Compatibility

Beyond performance improvements, v1.0.0 also adds support for several new model architectures:

GptOssForCausalLM: support for the GPT-OSS model family
NemotronHForCausalLM: support for NVIDIA's Nemotron-H model family

Combined with the already-mentioned Gemma4 tensor parallelism support, ExLlamaV3's model coverage continues to expand, enabling it to keep pace with the open-source community's rapid model iteration.

Comprehensive Engineering Quality and Usability Improvements

Beyond the headline features, v1.0.0 includes a wide range of detail-level improvements:

Graph paths for all attention/GDN modules to improve execution efficiency
Faster extension build times by increasing compilation units to shorten compile times
Numerous small performance optimizations, bug fixes, and quality-of-life (QoL) improvements

These detail improvements are often what separates an open-source project that merely "works" from one that's genuinely pleasant to use. Shorter compile times, fewer dependency conflicts, and more stable runtime behavior together form the foundation for v1.0.0's credibility as a production-grade release.

Summary: A Major Leap Forward for the Local Inference Ecosystem

The release of ExLlamaV3 v1.0.0 marks the project's transition from active development into a mature, stable phase. From removing external dependencies and rewriting the attention kernel to expanding tensor parallelism and adding new model support, this update touches virtually every critical component of the inference engine.

The most noteworthy breakthrough is that KV cache quantization no longer degrades inference speed — by using Kernel Fusion to merge quantization overhead with attention computation, while simultaneously improving throughput through reduced bandwidth pressure, this directly addresses the long-standing tension between VRAM and speed in local large model deployment. For developers and enthusiasts looking to run larger models with longer contexts on limited hardware, this is a major update worth paying attention to right now.

Interested users can visit the official ExLlama Discord community to discuss further with Turboderp and other developers, or consult the official release notes for detailed performance benchmarks and technical documentation.