DeepSeek Flash 1017: An LLM That Runs a Million Tokens on a Single GPU

A million-token context used to be a data-center luxury—by conventional memory overhead, the KV-cache alone could eat up dozens of accelerator cards. DeepSeek^Ad v4.1-Flash 1017 attempts to rewrite that math: a sparse MoE model with 284B total parameters that activates only 13B per token, yet can process million-scale context on a single A100-40GB. This isn't overturning the Transformer scaling laws—it's re-engineering the "memory wall" that has long blocked long-context applications.

Core mechanism: how HSQ-A compresses the KV-cache by 85%

The key component of Flash 1017 is Hybrid Sparse + Quantized Attention (HSQ-A). Its approach has two steps: first, a lightweight scoring network selects roughly 15% of "high-impact" keys and keeps only those; then the remaining values are stored in FP8 format while the rest are discarded.

The result is that the KV-cache drops from the 0.8 GB/k typical of a dense 70B model to 0.12 GB/k—a compression ratio of about 85%. This number determines whether single-GPU deployment is feasible: a 750K-token contract document has an original cache of about 90GB, which after the compression pipeline falls to the order of about 10GB—small enough to fit in the A100's remaining memory.

The model overall is a sparse MoE: 284B total parameters, 13B activated per token (i.e., the Flash-13B sub-model). Inference relies on DeepSeek's "Harness" runtime, with optional TensorRT acceleration kernels.

KV-cache (key-value cache) is the memory region in autoregressive Transformer inference that stores the Key and Value matrices of historical tokens, used to avoid recomputing the entire context for every new token generated. Its memory footprint grows linearly with context length: each token requires storing 2×d_model floating-point numbers per layer (one each for Key and Value). Taking a 70B dense model as an example, at FP16 precision the KV-cache per token per layer is about 2×8192×2 bytes ≈ 32KB, and across 80 layers totals about 2.5MB/token, i.e., 2.5 GB/k (k = thousand tokens). The "0.8 GB/k" in the article corresponds to the typical value for a dense 70B model after optimizations such as layer sharing and GQA. In long-context scenarios, the KV-cache often exceeds the model weights themselves and becomes the memory bottleneck—which is the fundamental reason million-token context has long depended on multi-GPU parallelism. HSQ-A's compression targets exactly this bottleneck.

Sparse MoE (mixture of experts) is an architecture that splits the feed-forward network into multiple "expert" sub-networks, activating only a few of them per token. The total parameter count can be very large (e.g., 284B), but the parameters actually involved in computation per token are far smaller than the total (e.g., 13B), thereby controlling inference compute while preserving model capacity. Unlike dense models, MoE's memory footprint still needs to accommodate all expert weights, so it demands more memory capacity but is friendlier to compute throughput. Flash 1017's "284B total parameters, 13B activated" falls into this category: at inference the computation graph actually executed is that of the 13B sub-model, but weight loading must cover 284B parameters (which can be mitigated through expert parallelism or paged loading). Understanding this is essential to see why its throughput can approach that of a 13B-class model while its accuracy can match a 70B-class dense model.

Hard data: the triangular balance of throughput, accuracy, and power

According to the published benchmark table, Flash 1017 delivers 45 tokens/s throughput on a single A100—higher than GPT-4-Turbo's 38, Llama-3-70B's 32, and its sibling V4-Pro's 28. Its zero-shot MMLU score is 71.2%, essentially on par with GPT-4-Turbo's 71.8% and slightly below V4-Pro's 73.5%.

Long context is where it truly shines: it scores 68.9% on LongBench QA top-5, beating V4-Pro's 66.1% and GPT-4-Turbo's 62.3%. Power consumption is 0.42 kWh per million tokens, about one-third lower than the comparison group. Speculative decoding delivers a 2.3× speedup, also leading.

In other words, Flash 1017 trades "only 13B computed per token" for throughput and trades HSQ-A for context length, at the cost of conceding a little accuracy to the flagship model—a trade-off that pays off in most medium-to-long text tasks.

Speculative decoding is an inference technique for accelerating autoregressive generation: a smaller, faster "draft model" first generates several candidate tokens in succession, and then the target large model verifies these candidates in parallel in one pass—accepting those consistent with its own distribution and rejecting and correcting the inconsistent ones. Because the large model's forward pass can process multiple candidates in parallel, wall-clock latency drops significantly, and the output distribution remains strictly identical to the target model's (lossless acceleration). The "2.3× speedup" mentioned in the article stems from this. Using Flash-13B as the draft model and the full 284B MoE as the verifier is a typical modular form of this technique, and is also the technical basis for the "small model leads the charge, large model reviews" idea discussed later.

Real-world single-GPU deployment scenarios

A mid-sized law firm wants to run an internal contract assistant, doesn't want to send confidential PDFs to the cloud, and has a hardware budget of just one A100-40GB. Following the path given in the source material: loading Flash-13B takes about 12GB of memory, leaving 28GB for the KV-cache; importing a 300-page commercial lease (about 750K tokens) yields a compressed cache of about 10GB—it fits; a lawyer asks "what termination clauses can the landlord invoke under force majeure," and the model generates a 256-token answer in about 2 seconds while retaining the full context for follow-up questions.

The entire document review takes under a minute, billed at $0.001 per thousand generated tokens, with all data staying local. The value of this case isn't in the performance numbers but in the fact that it turns "million-token context" from a cloud API capability into a capability on a single local card.

On the cost side, the free tier is 100K tokens per month, suitable for academic prototypes; pay-as-you-go is 30% cheaper than V4-Pro. For a typical small-business chatbot workload of 10 million tokens per month, the bill comes to about $10.

Risks and open questions

Aggressive pruning is not without cost. Compression artifacts may lose subtle long-range dependencies, which warrants particular caution on code and formal-logic tasks. DeepSeek provides a verify=true API switch that uses the full MoE to re-verify key passages, but this incurs extra overhead.

Hardware binding is also worth noting: the Harness runtime is optimized for A100-class GPUs, and performance drops noticeably on older cards. The official fallback is an FP8-quantized CPU path, with about 2 seconds of latency per 256 tokens at batch size ≤ 4—suitable only for non-real-time workloads. In addition, pay-as-you-go pricing may fluctuate with demand, and ecosystem tools (tokenizer, streaming API) are still maturing.

The model weights themselves are released free under the Apache-2.0 license, which partly offsets lock-in risk—enterprises can build their own inference stack, provided they accept the engineering effort.

Pruning artifacts refer to the semantic distortion introduced during sparsification or quantization compression by discarding some Key/Value entries or reducing numerical precision. In HSQ-A, keeping only about 15% of "high-impact" keys means the other 85% of context information is directly discarded, and the model may be unable to trace back long-range dependencies that the scoring network didn't identify as important. Such artifacts are particularly dangerous on tasks requiring precise reference to earlier details (e.g., variable binding in code completion, premise backtracking in formal logic), because critical information may happen to fall in the pruned low-score region. The verify=true switch uses the full MoE to re-verify generated results, acting as a fallback with an uncompressed "gold standard," but it offsets part of the throughput advantage—an adjustable lever between accuracy and speed.

The next step for long-context LLMs

Flash 1017 points in several directions. Enterprises can stuff an entire knowledge base (over a million tokens) into a single inference pass, replacing multi-step retrieval pipelines with one forward pass; RLHF agents can retain their entire planning history instead of resetting after a few thousand tokens, which matters for supply chain optimization and long-cycle scientific research; and industries requiring data residency—finance, healthcare, legal—now have, for the first time, a localization option that is both compliant and cost-effective.

The speculative decoding line is equally worth watching: Flash-13B can serve as the draft model for a large MoE system, producing candidate sequences first, which the full 284B model then verifies or corrects token by token. The arXiv paper SpecDec-Hybrid reports a 2.3× wall-clock speedup on a 4-GPU node, cutting latency for 2K-token long-form generation from 12 seconds to about 5 seconds. This modular pipeline of "small model leads the charge, large model reviews" may become a common form for MoE systems in the next stage.

Flash 1017 doesn't rewrite Transformer theory—it works around the memory bottleneck. Single-GPU million-token context has moved from demo to production; this step has already landed. Going further to tens of millions of tokens depends on how much more the compression pipeline can be optimized.

One-line Summary

DeepSeek Flash 1017 uses a 284B sparse MoE and HSQ-A to compress the KV-cache by 85%, achieving million-token context on a single GPU.

Paragraph Summary

DeepSeek v4.1-Flash 1017 is a sparse MoE model with 284B total parameters that activates only 13B per token. Its core innovation, HSQ-A, compresses the KV-cache from 0.8 GB/k to 0.12 GB/k (about 85%) by sparsely filtering high-impact keys and FP8-quantizing values, enabling a single A100-40GB to handle 750K-token-scale context. Single-GPU throughput of 45 tokens/s exceeds GPT-4-Turbo, MMLU of 71.2% is roughly on par, LongBench QA of 68.9% surpasses it, and power consumption is about one-third lower. In a real-world scenario, a 750K-token contract occupies only about 10GB of memory after compression, generating a 256-token answer in 2 seconds with all data staying local. Risks include pruning artifacts, A100-class hardware binding, price fluctuations, and ecosystem maturity. Weights are open under Apache-2.0, and the model can serve as a draft model with speculative decoding for a 2.3× speedup.