Agens Volundr 32B: A Hybrid Architecture Where Only 18 of 72 Layers Retain KV Cache

A small Hong Kong team called Blockway posted on r/LocalLLaMA about their first model built on a custom hybrid architecture — Agens Volundr 32B Preview. The most striking design choice: out of 72 total layers, only 18 retain a KV cache. For long-context local inference, this trade-off goes straight at a pain point that keeps coming up — what actually limits whether a model fits in VRAM is often not the weights, but the KV cache.

Why Slash Most of the KV Cache

Blockway's motivation is refreshingly practical: their customers run models on their own machines. As context length grows, the KV cache expands linearly with sequence length and quickly dominates VRAM usage — not the model weights themselves. Starting from that observation, the team designed an architecture where the vast majority of layers maintain no KV cache at all.

This is a classic engineering trade-off. In a standard Transformer, every layer caches the Key/Value representations of all past tokens. At something like 262K context, that overhead rapidly consumes your entire VRAM budget. Compressing the number of caching layers to a minimum means the same hardware can support much longer context windows — which is exactly what local deployment users care most about.

Architecture Breakdown: Four Types of Layers

The model is a dense architecture with approximately 32B parameters across 72 layers, where every layer runs on every token. The layer composition is quite interesting:

Linear Attention as the Backbone

• 54 layers of KDA (Kimi Delta Attention): Linear attention using a fixed-size recurrent state, with no KV cache. This accounts for three-quarters of all layers and is the core source of cache savings.
• 17 layers of BCSA (Block-wise Compressed Sparse Attention): Performs exact window attention over the most recent 4,096 tokens; older context is pooled at 4:1 into blocks, then a learnable indexer retrieves the top 512 blocks.
• 1 layer of full attention (layer 72): Retains full attention capability as an architectural safety net.

KDA (Kimi Delta Attention) is a variant of linear attention. Standard Transformer attention scales quadratically with sequence length (O(n²)), while linear attention factors the attention matrix into the product of kernel functions, reducing complexity to O(n) and replacing the ever-growing KV cache with a fixed-size recurrent hidden state. The trade-off is that exact global attention between arbitrary token pairs is no longer possible, which theoretically limits expressivity. BCSA (Block-wise Compressed Sparse Attention) represents a middle ground: it preserves exact sliding-window attention for recent tokens to maintain local precision, compresses distant history through pooling, and then uses a learnable module to retrieve the most relevant blocks — similar in spirit to Longformer and BigBird, but with a learnable indexing mechanism to improve retrieval quality for long-range information. Stacking these low-cache or no-cache attention types as the primary substrate, supplemented by a small number of full-attention layers, is one of the dominant research directions for balancing long-context capability with inference efficiency.

Memory and Residual Enhancements

Beyond the attention layers, the team introduced two auxiliary mechanisms: Engram is a hash-based n-gram memory stored in host RAM, attached to 2 of the 72 layers; mHC replaces the traditional single residual stream with 4 parallel residual streams.

The end result: only 18 of 72 layers retain a KV cache, with a context window of 262K. This "linear attention as the primary body, sparse precise attention as a supplement" hybrid approach aligns with several long-context architectures being explored across the industry.

The Engram mechanism takes inspiration from the neuroscience concept of an "engram" — the physical trace a memory leaves in a neural network. In practice, it uses hash functions to map n-gram sequences to fixed slots in CPU RAM, allowing the model to retrieve statistical information about longer token combinations without consuming any GPU VRAM. This kind of external memory design shares the spirit of Memorizing Transformers and Unlimiformer, trading cheap memory bandwidth for broader effective context coverage. mHC (multi-stream Highway Connections) corresponds to another recent architectural exploration: the single residual stream in a standard Transformer creates a bottleneck in inter-layer information routing. Multiple residual streams allow different types of features to propagate through independent channels, theoretically improving representational diversity — though at the cost of increased parameter count and implementation complexity.

Speed: Barely Any Degradation at Long Contexts

The team published single-user performance numbers from their custom sglang build. The most noteworthy result is how stable decode speed remains as context grows:

• BF16 (two 48GB GPUs): Decode speed goes from 25.1 tok/s at 1K context to 23.9 tok/s at 128K — almost flat.
• BF16 prefill: From 2,122 tok/s at 1K to 1,297 tok/s at 128K.
• INT4 (31.7 GiB, single 48GB GPU): Decode at 31.0 tok/s at 1K, still 29.1 tok/s at 32K.

The near-flat decode speed from 1K to 128K context is the direct payoff from eliminating most KV caching — the cache doesn't grow linearly with sequence length, so the marginal cost of decoding long text drops dramatically.

The team also provides a DFlash2 drafter (speculative decoding accelerator) in a separate repository. When enabled on the same server, it delivers up to 3.6× speedup for JSON/tool-call outputs, ~2.0× for code, and ~1.6× for reasoning/thinking mode. They're upfront that beyond roughly 8 concurrent users, the overhead makes it no longer worthwhile. On aggregate throughput: BF16 reaches 127 tok/s at 8 users and 130 tok/s at 16 users.

Speculative Decoding accelerates autoregressive generation through a "small model drafts, large model verifies" approach. DFlash2 drafter is one such scheme: a lightweight draft model rapidly generates several candidate tokens, which the main model then batch-verifies and accepts the correct ones — reducing the number of full forward passes without changing the output distribution. This works best when output patterns are predictable (JSON, code), since the drafter guesses correctly more often and acceptance rates are high. Open-ended reasoning has higher output entropy, so acceptance rates fall and the speedup shrinks. The team's reported 3.6×/2.0×/1.6× breakdown perfectly illustrates this pattern. At higher concurrency, GPU batching efficiency improves and the drafter's marginal contribution diminishes, which is why the approach loses its cost-effectiveness at scale.

Benchmarks: Strong on Code and Math, Weak on Agent Tasks

The team emphasizes that all benchmarks were run by them using the same harness and settings for all models compared. Against Qwen3.8-27B, the results show a clear split:

Ahead:

• LiveCodeBench v6 (+4.2)
• HumanEval (+4.3)
• AIME 2025 (+2.9)
• MATH-500 (+1.6)

Roughly even: MMLU-Pro, IFEval, GPQA Diamond.

Clearly behind: tau2-bench at 74.2 vs. 79–80; SWE-bench Verified (50-problem subset) at only 44 vs. 58–64.

In short: the model holds its own on coding and mathematical reasoning, but has a meaningful gap on agent tasks. The team doesn't shy away from this — they explicitly state that the full v1 release will focus on closing that gap through continued pre-training to approximately 10 billion additional tokens and adding long agent session training data.

Known Limitations and How to Get Started

The team's transparency is worth noting. They listed limitations directly in their post:

• Currently requires their own custom sglang build; upstream sglang and vLLM cannot load the model yet.
• GGUF / llama.cpp support is planned but not yet available.
• Long agent sessions are the weakest point in this Preview.
• The model is still training — this is a preview, not a final release.

Setup is provided via Docker images targeting 48GB Ada architecture GPUs (sm89) and H100/H200 (sm90). The model is released under Apache-2.0, with BF16 and INT4 weights and the DFlash2 drafter uploaded to Hugging Face; the project page and inference code are hosted on GitHub.

Why Small-Team Architecture Innovation Matters

Setting aside the specific scores, this project's deeper value lies in the path it demonstrates: even with limited compute, a small team can address real deployment pain points through deliberate architectural trade-offs. Agens Volundr doesn't try to beat all peers across the board — it focuses resources on the specific, well-defined goal of VRAM efficiency under long-context conditions.

The team repeatedly emphasizes throughout their post: "trained under limited compute," "this is not perfect," "please read the limitations before trying it" — and actively invites the community to submit failure cases and benchmarks they'd like to see covered. This willingness to put weaknesses front and center is a rare posture in today's benchmark-marketing release culture. For readers who care about local deployment, long-context inference, and the evolution of attention architectures, this is an experimental reference worth keeping an eye on.