How DeepSeek Cut AI Inference Bills by 150x: A Four-Step KV Cache Compression Breakdown

A 437x Number

When running a coding agent, the most expensive line on your bill is rarely the code it writes — it's the context being resent with every single turn. You're paying to re-read what you already sent. Priced against mainstream U.S. frontier models, this alone accounts for roughly two-thirds of total costs. DeepSeek's latest release prices that same line at $0.00003 per million tokens — three hundredths of a cent.

This isn't a promotional discount. They rebuilt the attention mechanism until that cost nearly vanished. This article breaks down a technical analysis by an overseas blogger, tracing the full four-step path DeepSeek used to get there — and where it starts to break down.

A 437x Number

Every change revolves around a single metric: bytes of GPU memory consumed per context token.

In DeepSeek's first-generation model, one token occupied 389,000 bytes. In the model released this month, that same token occupies just 890 bytes. Same company, four generations of architecture, 437x smaller.

What makes this counterintuitive: the new model has nearly twice the parameters of the older model whose capabilities it reproduces. Larger, cheaper, faster, stronger — conventional wisdom says you can't have all four. Yet it appears to have them. This 55.2-billion-parameter model is publicly released under the MIT license.

On benchmarks, it matches Claude Opus on agentic coding tasks but trails by roughly 20 points on the hardest terminal-class benchmarks. The real story isn't the scores — it's the bill.

Where the Money Actually Goes

One developer shared their token logs from a single long coding session: a medium-sized codebase, 447 conversation turns. Throughout the session, the agent sent roughly 1 million fresh input tokens — but read back approximately 36.5 million tokens from cache. The same context, round after round, repeated 36 times.

At GPT-tier pricing, that session cost around $55, with two-thirds of that on the cache-read line. Running the same logs against DeepSeek's new model: $0.36. The gap speaks for itself.

When a model reads your prompt, what it retains isn't the words themselves — it's each layer's "understanding" of each word. That's the Key-Value Cache (KV Cache). Think of it as notes on a desk, so you don't have to re-read the entire book every time. Notes take up space, and writing them takes compute. So the cost of "reading a prompt" hides two charges: read cost and storage cost. DeepSeek attacked both simultaneously.

A note on KV Cache mechanics: When a Transformer processes each token, it computes a set of Key and Value vectors at every layer, encoding that token's semantic information at that depth. To generate the next token, the model must use the new token's Query vector to compute similarity scores against every historical token's Key vectors, then weighted-read the corresponding Values. If you recomputed all historical key-value pairs from scratch on every generation step, compute would scale quadratically with context length. KV Cache stores already-computed key-value pairs in GPU memory so they can be read directly next time, avoiding redundant computation. The cost is that GPU memory usage grows linearly with conversation length — across dozens of layers, each with a full key-value matrix, a context of hundreds of thousands of tokens can easily saturate an entire high-end GPU's VRAM. This is the core bottleneck DeepSeek's four-step compression strategy sets out to solve.

Step One: Split the Network in Half

The 40-layer network is cut in two at the middle. The bottom 20 layers act as an encoder; the top 20 act as a decoder, joined by a single seam.

When a prompt arrives, only the encoder's 20 layers actually run through the full content. The decoder's top 20 layers don't read your prompt at all — their global key-values aren't computed from their own hidden states. Instead, they're projected in a single pass from the encoder's final hidden state, using one dedicated weight matrix per layer. One pass, 20 copies, no secondary reads.

The result: the cost of reading a prompt drops from "run the entire network" to "run half the network." The paper describes prefill complexity as n×L/2 — "nearly halved."

Token generation is a different story — each generated token must genuinely pass through all 40 layers, so decode activates 16 billion parameters while prefill activates only 8 billion. Every layer also retains a 128-token sliding window with a "bounded replay" mechanism that replays only the last window and approximately reconstructs state, avoiding repeated computation across thousands of tokens.

A clarification on "encoder-decoder" here: This differs from the classical Transformer usage (BERT as pure encoder, GPT as pure decoder). In DeepSeek's architecture, "encoder" refers to the bottom layers responsible for understanding input during the prefill phase, and "decoder" refers to the top layers responsible for token-by-token generation during the decode phase — a hybrid architecture, not two separate models stitched together. The prefill phase processes the entire input prompt at once and can be parallelized; the decode phase generates one token at a time and must run serially. These two phases have fundamentally different compute characteristics: prefill is compute-bound (processing many tokens at once), decode is memory-bandwidth-bound (one token per step, but accessing the full historical cache). Differentiating the network's treatment of these two phases is exactly why "splitting in half" reduces prefill cost without degrading decode quality.

Step Two: Compress the Cache on Three Dimensions

There are only three places to shrink a cache: the size of individual entries, the number of entries along the sequence dimension, and the number of layers that retain a cache at all. Previous techniques typically targeted just one. This time all three move at once, and effects multiply.

Entry size first: 64 query heads share a single latent, and each layer has only one key-value head. Sequence dimension next: inside the encoder, every two adjacent tokens are compressed into one cache entry, halving the count — because adjacent tokens in a codebase dump aren't independent to begin with.

The heaviest reduction is in the layer dimension. Running through the released configuration reveals that only 4 of the 40 layers actually retain a cache; the other 36 store nothing. This isn't stated in the paper — it's written in the attention class comments in their code. The remaining 36 layers each receive a "borrowing mode": re-indexing layers use their own queries to re-score shared keys and compile a new candidate list; reuse layers skip even the scoring and directly carry forward the previous layer's list. Neither mode writes a single byte to cache.

Finally, precision: surviving cache entries are stored at 4-bit precision, with one 8-bit scale shared per 16 channels, quantized after rotary positional encoding — cutting another 2x versus the previous generation. Four factors stacked together produce that landmark figure: 890 bytes per token.

Step Three: Deciding Which Keys to Read

With only 4 layers retaining full keys and the other 36 borrowing from neighbors, a critical question arises: something must select which entries each query actually reads, or the model will perform far worse on long contexts than benchmarks suggest.

This task falls to a small 32-head side-channel attention module DeepSeek calls the "indexer." It scores candidates and retains the top 512. But it also creates a new bottleneck: scoring every visible entry costs linear compute in context length, and each indexing layer repeats this.

The solution is hierarchical convergence: the decoder's first full-mode layer (layer 20) scans the entire visible context once, then takes a second pass in blocks of 8 positions and keeps the max value per block, retaining the top 2,048 blocks — roughly 16,000 candidate positions — and passes this pool upward. Every deeper indexing layer then scores within those 16,000, not across millions. Cost becomes roughly constant with depth.

The trade-off is explicit: this pool reflects layer 20's opinion alone. If an entry needed by a deeper layer didn't make it into layer 20's candidate blocks, no layer below can ever retrieve it. The bottleneck is moved, not eliminated.

Step Four: Moving Memory Off the GPU

One more module gets relocated. Sitting at layers 1 and 14 are two modules that perform no matrix multiplication at all — they are lookup tables. DeepSeek calls them n-grams.

The model hashes each token's preceding 2, 3, and 4 tokens separately, using 8 hash heads per length, for 24 hashes total, each indexing a table of roughly 16 million rows. Table sizes are distinct primes to avoid collision alignment. All these tables together reach 196 billion parameters — over a third of total model parameters — and not one of them performs a multiplication. Each token touches only 24 rows, so despite the enormous parameter count, per-token work is a single lookup.

Because addressing is deterministic, these tables don't need to live on the GPU at all. They reside in host memory, with background transfers prefetching the rows needed next. Roughly 200 GB of tables sit in ordinary system RAM; the GPU holds only the main network trunk. This is not an isolated decision: a Qwen Flash model released by Alibaba at the end of August also moved 51 billion n-gram parameters into system memory. Two labs, weeks apart, made the same move.

What n-gram lookup tables do here: In language models they function like "hardcoded local statistical memory." Classical n-gram models tabulate how often sequences of n consecutive words co-occur and use that to predict the next word — the dominant language modeling approach before deep learning. DeepSeek embeds n-gram information as hash-indexed lookup tables inside the neural network, letting the model directly query local word-sequence patterns without having to "rediscover" those statistical regularities through attention. The advantages: lookup involves no matrix multiplication, compute cost is minimal, and addressing is fully deterministic so rows can be prefetched on the CPU side — a natural fit for system memory rather than GPU VRAM. The cost: the tables themselves are large (~200 GB) and only capture short-range local dependencies; they cannot replace attention's ability to model long-distance semantic relationships.

The Real Limits and Trade-offs

Each of these four mechanisms is an approximation: half the network reads the prompt, bounded replay approximately reconstructs the window, most layers borrow keys they never scored themselves, and the indexer discards the majority of context. DeepSeek's own limitations section explicitly names two failure modes — sparse attention and approximate state reconstruction in bounded replay — which, in their words, "may still cause capability degradation in untested edge cases." The deployment section is blunter: because replayed prefix states are approximate, identical prompts with different cache-hit positions will yield slightly different results. That sentence is theirs, not a critic's.

At the benchmark level, the wins are real but narrow. It leads by 0.2 points on a software engineering benchmark and by 1.5 points on one terminal benchmark — margins thin enough that switching test frameworks could flip the result. The gaps in the other direction are large: 20 points behind on one terminal benchmark, 19.5 points behind on a graduate-level exam. DeepSeek acknowledges that "matching on average does not mean matching the capability of closed-source frontier systems." It also outputs "very verbosely" at roughly 200 tokens per second, generating nearly twice as much content as a median model.

Conclusion: Wins on Cost, Loses on Expert Capability

The trade-off stated plainly: this model wins "capability bought with money" and loses "capability requiring expert knowledge." If your agent spends most of its time re-reading long contexts, this is the obvious choice, and by a wide margin — on the workload described at the opening, it achieves comparable results at 76x lower cost. If your problem is a hard scientific question, pay Anthropic.

The most compelling evidence is DeepSeek's own behavior: they routed their flagship traffic into the budget tier, priced it at budget rates, and are gradually retiring the flagship. A lab willing to replace its own flagship with a cheaper model genuinely believes in that cheaper model. And with weights on Hugging Face under the MIT license, teams that need to freeze a model version can deploy freely — an option unavailable to any closed-source API user.

The blogger closes with a falsifiable prediction: DeepSeek says a larger version of this architecture is coming, and he predicts it will not close the 20-point terminal benchmark gap. The truly open question is whether there's a fifth compression hiding inside attention — or whether the next breakthrough has to come from somewhere else entirely.

A note on the MIT license in this context: In the context of AI model releases, MIT means the weights can be freely used commercially, modified, and redistributed without paying the original authors and without requiring derivative products to be open-sourced — similar to Apache 2.0, but far more permissive than GPL. For teams that need to freeze a model version in production, this is especially significant: closed-source API endpoints can be unilaterally upgraded or discontinued by the provider at any time, while a team holding MIT-licensed weights can run a specific version on their own infrastructure indefinitely, unconstrained by any vendor roadmap. This is the essential distinction between "open weights" and "open capability" — the former grants deployment sovereignty; the latter provides only technical transparency.