#KV Cache
5 related articles
How KV Cache Reshapes AI Agent Cost Structures
Analyzing how KV Cache affects AI Agent operating costs: shrinking cache saves memory, reusing prefixes cuts redundant compute, nearby storage reduces transmission latency — each saves different resources.
DeepSeek Silently Swapped Models: The KV Cache Arithmetic Is the Real Announcement
DeepSeek silently swapped deepseek-flash for the V4.1-Flash-1017 checkpoint, calling it a minor update while benchmarks rose across the board. Starting from the public config.json, this article recomputes KV cache cost per token, explains why the 890-byte figure is a category error, and unpacks the pricing logic behind the 50x cache discount and the risks of silent model-alias swaps.
Implementing GPT-2 KV Cache from Scratch: Correctness, Latency, and Memory Benchmarks
Implementing GPT-2 small KV Cache from scratch with benchmarks: 6.1e-5 logit error, only 6x single-token speedup at context 1000 (not 1000x), 72KiB/token formula plus 8.5MB cuBLAS workspace offset, and a multi-token decoding failure case.
How an LLM Generates a Word: The Complete Pipeline from Token to KV Cache
From tokenization to transformer attention, from Prefill/Decode phases to KV Cache costs, and why LLMs have no memory—a clear explanation of the complete pipeline for generating a word and the root cause of expensive long context.
Tech FrontiersCloudflare Contributes Critical KV Cache and Mooncake Fixes to SGLang
Cloudflare contributes decode KV cache offload and Mooncake recovery fixes to SGLang, resolving garbled output under high concurrency for Kimi K2.6 and enabling automatic fault recovery in distributed inference.