Anifer Open-Source Inference Engine Benchmarked: Single RTX 5090 Hits 150 token/s

Built for One Thing: Raw Speed

In the world of local LLM deployment, inference speed has always been a pain point. A recently open-sourced inference engine called Anifer — nicknamed "Gale Wolf" by one content creator — is pushing that boundary to a new extreme. Written in C++ and CUDA by an independent developer, it operates close to bare metal and focuses entirely on squeezing maximum performance out of a small set of models and GPUs.

Building an inference engine in low-level languages like C++ and CUDA is a significant challenge — the code is complex and maintenance costs are high. But the payoff is deep hardware-level optimization, which for end users translates to faster decoding speeds and lower latency.

Anifer officially supports only a single RTX 5090 and is optimized for just a handful of models. Its registered model list contains only five entries, focused on Qwen3.8 27B, the previous-generation Qwen3.6 27B, and the pruned (MoE) variant of Qwen3.6 35B. This "less is more" strategy is precisely what makes it so fast.

Why It's Faster Than vLLM and SGLang

To understand Anifer's speed advantage, you first need to understand the constraints mainstream inference frameworks operate under. vLLM, SGLang, and llama.cpp are the three most widely used frameworks today, and they share a common trait: they aim to be comprehensive. They must support hundreds of model architectures (Qwen, Gemma, Llama, etc.), handle various quantization formats (NVFP4, AWQ, etc.), and run on everything from NVIDIA A-series and B-series data center cards to consumer GPUs.

The cost of this broad compatibility is architectural bloat. Maintainers can only deliver a "compromise" performance solution — something that runs any model on any card, but can never be deeply tuned for a specific GPU. Performance inevitably takes a hit.

A fitting analogy: Anifer is like a PS4 Pro — a dedicated gaming console — while vLLM and llama.cpp are more like the wildly diverse PC market. The PS4 Pro's raw compute is roughly equivalent to an RTX 5070, but because there's only one fixed hardware configuration, developers can optimize to the limit, making games run as smoothly as they might on a 5090. Anifer targets only five models and one GPU, concentrating all development effort on speed. That's the fundamental reason it's fast.

vLLM and SGLang are "serving-oriented inference frameworks" designed to efficiently schedule large numbers of concurrent requests in production, supporting techniques like Continuous Batching and PagedAttention to maximize throughput in multi-user scenarios. llama.cpp takes a different route — implemented in pure C/C++, it prioritizes cross-platform compatibility and can run quantized models on CPUs, Apple Silicon, and a wide range of consumer GPUs. The shared cost of all three is heavy abstraction layers. To drive hundreds of models and hardware configurations from a single codebase, they must introduce extensive conditional branching, generic kernels, and runtime dispatch logic — making it impossible to fine-tune register counts, shared memory layouts, or Tensor Core utilization for any specific hardware. CUDA kernel performance is extremely sensitive to "alignment" — even slight mismatches in matrix dimensions, thread block sizes, or memory access patterns can cause throughput to drop dramatically. Because Anifer serves fixed model shapes on a fixed GPU architecture, every kernel can be written in its "exactly optimal" form. That's something a general-purpose framework structurally cannot do.

Official Benchmarks: From 200 to 1,147 token/s

Anifer caps concurrency at 8, which is more than enough for typical users — at 8 concurrent requests on a single 5090, VRAM is nearly maxed out anyway. Here are some official benchmark figures (under ideal conditions):

Qwen3.6 27B Dense Model

With NVFP4 quantization, single concurrency (C=1) reaches 202 token/s, doubling to 400 token/s at C=2 with almost no degradation. At C=4, throughput reaches approximately 700 token/s (theoretical ceiling is 800, with slight losses), and at C=8, it hits a remarkable 1,147 token/s.

With GGUF quantization, dual concurrency peaks at 247 token/s — actually faster than single concurrency.

Qwen3.6 35B MoE Model

The MoE model posts even more impressive numbers, reaching 1.38k token/s at C=8. However, as the reviewer notes, running 8 concurrent sessions at this speed leaves almost no context window to work with, limiting practical utility.

Real-World Test: Qwen3.8 27B Sustains 144 token/s at C=1

The reviewer ran their own benchmarks on Qwen3.8 27B with a personal 5090, and the results hold up:

• C=1: Official spec is 144 token/s — fully achieved in testing, occasionally reaching 150
• C=2: 268 token/s, averaging ~130 token/s per session
• C=4: All four streams combined reached ~400 token/s with no degradation over time — still sustaining 94–112 token/s per stream after 6–7 minutes of continuous generation

With context extended to 120k tokens, the 5090's 32GB VRAM was nearly full, yet speeds remained steady. This confirms that at least for Qwen3.8 27B, the official numbers are trustworthy.

One key contributor to the speed is speculative decoding. Anifer supports MTP (Multi-Token Prediction) with an acceptance rate of 46–49%, which testing confirmed is close to actual performance. It also supports D Flash speculative decoding, though at higher VRAM cost. The reviewer ran MTP at 3 and matched the speeds shown in official demos.

Speculative Decoding is a technique that uses a lightweight "draft model" to quickly generate candidate tokens, which the main model then verifies in parallel in a single forward pass — if accepted, multiple decoding steps are completed at once, significantly boosting throughput. MTP (Multi-Token Prediction) is a variant that doesn't rely on a separate small model; instead, it attaches additional prediction heads inside the main model to directly predict multiple future tokens, with a more compact structure and lower VRAM overhead. The acceptance rate is the key metric for speculative decoding effectiveness: a 46–49% acceptance rate means roughly half the draft tokens are accepted by the main model, which is considered quite high in practice. This delivers substantial speed gains without affecting output quality — rejected tokens are discarded and the process rolls back, and the final output is mathematically equivalent to pure autoregressive decoding.

The .enifer Single-File Format: Pros and Cons

Anifer uses a proprietary format called .enifer, which bundles model weights, the chat template (system prompt), and any multimodal-related content into a single file.

This design is a double-edged sword. The downside: you must download .enifer files specifically — standard GGUF files cannot be loaded directly. The upside: it's hassle-free. Anyone who's worked with Qwen open-source models knows that the system prompt needs to be manually configured, but this format ships with everything pre-packaged — download and run, no prompt engineering required.

GGUF is the dominant model file format in the llama.cpp ecosystem — short for GPT-Generated Unified Format — designed to pack model weights, quantization parameters, vocabulary, and metadata into a single file. It has become the de facto standard for distributing quantized models in the community, and the vast majority of community-quantized models on HuggingFace are published in this format. A Chat Template is a dialog format specification embedded in a model's configuration, defining how the system prompt, user turns, and assistant responses are concatenated into the final input sequence. Chat Template formats vary significantly across model families (Llama, Qwen, Gemma), and misconfiguration directly degrades instruction-following quality. The .enifer format bundles weights and Chat Template together, sacrificing interoperability with the broader ecosystem, but for everyday users it genuinely lowers the barrier to entry and avoids the common pitfall of degraded performance due to incorrect template configuration.

Hardware Requirements and GPU Support

The official project only supports RTX 5090, and on the OS side only Linux or WSL/Docker (Docker also requires a Linux environment). But since the project is open source, the community has already spawned numerous forks:

• GPUs: RTX 4090 is supported, and there are even adaptations for RTX 2080Ti and 20-series cards — mostly within the NVIDIA ecosystem
• OS: Native Windows builds exist, no virtual machine needed

Setting up the CUDA environment does require some tinkering, but dropping the project page into an AI agent (such as CUDAX) and letting it handle GPU adaptation and configuration changes is quite manageable. The prerequisite, of course, is that your GPU hardware meets the bar — a card with 12GB VRAM will struggle just to run Qwen3 27B, so no amount of config tweaking will save you. For users already doing local deployment, cards with 16GB or even modded variants with 20+ GB VRAM should handle this framework without issue.

Is It Worth Trying?

If your GPU is on Anifer's supported list, it's absolutely worth a try — benchmarks are reliable, performance doesn't degrade over long runs, prefill speed is fast enough that token generation starts almost immediately.

The trade-off is clear: it sacrifices generality in exchange for peak speed. This is the polar opposite of the "jack of all trades" approach pursued by vLLM and llama.cpp — but for users with a 5090 who simply want to run a few mainstream Qwen models as fast as possible, Anifer offers something close to a perfect solution.