Patch Makes DeepSeek V4.1 Local Agent Startup 20x Faster

When Altman Called Home DRAM "Quite Terrible," This Developer Decided to Respond with Action

Sam Altman's recent comment about DRAM allocation sparked plenty of discussion in the community. He said: "From a global allocation standpoint, having everyone put the world's limited DRAM supply in their own computers, and maybe only using it 5% of the time, is actually quite terrible." The remark itself was a defense of the centralized compute approach, but a developer on Reddit took it as a challenge.

His "quite terrible" home setup: an RX 9070 XT 16 GB GPU, a 5800X3D processor, 64 GB of DDR4 memory (he jokes that the RAM now costs more than the whole computer), and a PCIe 3.0 NVMe SSD. This machine runs DeepSeek^Ad V4.1 Flash entirely offline, supports 1M context, uses IST-DAS 1.75 bpw quantization, and the lab reports it achieves 98.7% of the full model's average task score. Expert weights are distributed across VRAM, RAM, and SSD. Generation speed is about 7 tok/s, which is acceptable for local inference.

The real problem isn't generation—it's prompt reading. Every time a new OpenCode session is created, the engine has to re-read about 37K tokens of system prompt and tool definitions, and the time to first token approaches 100 seconds. In his words, this is "wait times without a data center, but suffering at data-center levels."

Two Engine Patches: KV Splicing and Resident Routing

KV Splicing: Stop Re-reading What's Already Been Read

The root cause is that the agent framework stuffs session-specific text into the system prompt—working directory, date, the project's CLAUDE.md, skill paths with random IDs. This content changes every time, so prefix caching is completely defeated.

The developer modified the DeepSeek engine: the new prompt is aligned with all saved conversations, stale rows in the cached state are trimmed (shifting cache rows down, rotating the window ring, recording RoPE gaps), and only the changed rows plus the user message are read. Changed rows move in whole-row units, so the model never sees fragmented text.

The idea is close to "KV cache surgery," but the implementation is more pragmatic—rather than pursuing formal proofs, the goal is simply to get the cache hit rate up.

Prefix Caching is the key background here. In Transformer inference, the model generates and stores Key/Value matrices (the KV cache) for tokens it has already processed, so when a subsequent request shares the same prefix (e.g., a fixed system prompt), the engine can directly reuse those computed results and skip the repeated prefill stage. Whether prefix caching hits depends on whether the new request's token sequence is "token-by-token identical from the very beginning" with some record in the cache. Once the system prompt contains content that changes every time (working directory, timestamp, random ID), the prefix diverges at some point, and the cache for all subsequent tokens is invalidated—forcing recomputation from scratch. This is exactly the direct source of the 100-second time-to-first-token in the article. Models like DeepSeek that use MLA (Multi-head Latent Attention) have a more compact KV cache structure than standard MHA, but the alignment requirements for caching are equally strict.

Resident Routing: Make Memory Actually Do Its Job

Even with a short prompt, almost every expert in every layer gets touched, so during the prompt-reading phase most of the model has to be streamed from SSD—a 1.2K token prompt pulls in 58 GB. The patch's approach: during the prompt-reading phase, tokens only select from experts already resident in VRAM or RAM; the generation phase returns to normal routing.

The effect is immediate. Home DRAM is finally fully utilized instead of sitting idle. The author jokes that the only idle component in this system is his wallet, thanks to memory prices.

Mixture-of-Experts (MoE) architecture and expert routing are the prerequisites for understanding this section. DeepSeek V4.1 is an MoE model: each layer contains dozens to hundreds of "expert" feed-forward networks, but each token is routed to only a few of them (usually top-k). This means a single forward pass activates far fewer parameters than the total parameter count, but different tokens activate different expert combinations. When the model's total parameter count far exceeds VRAM capacity, expert weights must be stored in tiers, and unactivated experts are loaded on demand from RAM or SSD during inference. The problem is that the prompt-reading phase processes tens of thousands of tokens at once, and the combination of these tokens covers most experts in every layer—so the entire model has to be hauled out of slow storage. The idea behind resident routing is to restrict the candidate expert set during this phase, selecting only among already-resident experts, trading a bit of routing freedom for avoiding full streaming loads—essentially trading "approximate routing" for "IO locality."

Results and Costs: The Compromises Behind the 20x Speedup

According to the author's Reddit post, the two patches speed up new agent session startup by about 20x. This number corresponds to the reduction in prompt-reading time, not generation speed. For agent workflows that frequently create new sessions, the experience improvement is quite noticeable.

But the author also candidly lists the limitations:

• Currently only built and tested on HIP (gfx1201); CUDA, SYCL, and Metal are unverified.
• KV splicing is an approximation: tokens after the cut point are computed in a context that includes the removed text. The author says he hasn't noticed an impact on short lead-in lines, but there's been no formal evaluation.
• Resident routing slightly changes responses (experts are substituted during prompt reading). The author's hardcoded test suite is unchanged (12/13 in both modes), but each was only run once.

Both features are enabled by default in the branch and can be disabled with STRATA_DS4_REORDER=0 and STRATA_DS4_PROMPT_RESIDENT=0. The branch also incidentally fixes a GPU page fault (#4912) encountered with a 37K token prompt.

The HIP and gfx1201 mentioned in the article deserve a brief explanation. HIP is AMD's heterogeneous computing programming interface, highly syntax-compatible with CUDA, and can be seen as a bridge for running and porting CUDA code on AMD GPUs; gfx1201 is the GPU target codename for AMD's RDNA 4 architecture (RX 9000 series). The author uses an RX 9070 XT, so he can only build and test on HIP, while the CUDA (NVIDIA), SYCL (Intel), and Metal (Apple) backends all rely on their own different kernel implementations and memory management semantics—portability of the patches cannot be assumed without compilation verification. This is also why the author is publicly soliciting CUDA test data: for a change to go upstream, it typically needs to pass consistent numerical and performance regression tests across multiple backends; otherwise, the merger cannot confirm it won't introduce correctness issues on other hardware.

A Pragmatic Path for Local Inference

The value of this story isn't in slogans like "home GPU beats data center." The author himself admits that generation speed is mediocre, that it relies on SSD streaming, and that the patches are approximate. What's truly interesting is that it demonstrates an approach to using local hardware in tiers: VRAM holds hot experts, RAM holds warm experts, SSD is the fallback, and every tier is used where it matters most.

Altman said that 5% utilization of home DRAM is wasteful; this developer has offered another answer through action: when software is willing to treat memory as memory, rather than as a standby warehouse for when VRAM runs short, even so-called "terrible allocation" can produce real value. By his account, this setup achieves 100% memory utilization.

The code is on the dsv41-kv-splice branch of the PhialsBasement/Strata repository on GitHub, and the author especially welcomes CUDA users to submit test data to help get the patches merged upstream.