How Spark X Achieves Million-Token Context on a Small Local Model

An open-source model with roughly 4 billion parameters — small enough to run on a typical laptop — officially claims native support for a context window of around one million tokens. That sounds counterintuitive: how can a model with such limited parameters possibly remember that much text? A teardown of the Spark X 2.5 (4B) architecture by a Bilibili creator provides the answer, while also revealing the real cost hiding behind that headline number.

Two-Thirds of the Layers Have "Goldfish Memory"

The core design of Spark X 2.5 is that not every layer needs to track the full document. The model has 36 layers, organized into repeating groups of four. Within each group, three layers are sliding window layers that only attend to the nearest 512 tokens; the fourth layer is a global layer that actually reads the entire text.

This means only 9 of the 36 layers are taking notes on the full document. The remaining 27 layers — roughly three-quarters — forget everything from a few paragraphs ago almost immediately. The creator uses a vivid analogy: it's like running an office where three-quarters of the employees have goldfish memory, handling only the local grammar and phrasing of the sentence right in front of them, while the global layers can trace all the way back to a variable defined at the top of a codebase.

The reason this still works is that the global layers periodically pass their full-document retrieval results upward every few layers. Local layers handle nearby syntax and phrases; global layers "read the whole document." The division of labor lets the model retain the ability to recall long-range information even when the vast majority of layers are doing only local computation.

This interleaved "local layer + global layer" design is an academic variant of Sparse Attention architectures. Traditional Transformer self-attention requires every token to compute relevance against all other tokens in the sequence, meaning computation and memory scale quadratically with sequence length (O(n²)). At one million tokens, that cost is practically unacceptable in engineering terms. The core idea behind sparse attention is that not all token-to-token interactions matter equally for the final output, so only the "valuable" attention connections need to be preserved. Sliding Window Attention is the most direct implementation — each token only interacts with a fixed number of neighboring tokens, reducing complexity to O(n×w) where w is the window size. Models like Longformer and BigBird adopt similar hybrid strategies: most layers do local attention, a few do global attention, preserving awareness of long-range dependencies at a much lower cost. Spark X 2.5 is essentially this idea engineered into a small-parameter model.

KV Cache: The Real Memory Bill

Understanding Spark X's memory footprint means splitting it into two separate costs. The first is the fixed overhead of the model weights themselves: roughly 4 GB for the common 8-bit version, around 8 GB at full precision. Even if you just ask it to write a three-word shopping list, you pay this cost up front — loading the weights into memory is a prerequisite for anything else.

The second cost is the one that actually grows with context length. Engineers call it the KV Cache (key-value cache). In a standard model, every layer writes a note for every token, so the more tokens you have, the more memory fills up — like a parking lot on Black Friday.

Spark X breaks this pattern. The three sliding window layers stop growing once they've stored 512 notes — old notes are simply discarded; only the 9 global layers keep permanent notes for every token. According to the published design, each global layer stores roughly 4 KB per token, so every token added to the prompt contributes about 37 KB across all 9 global layers combined. As a result, KV Cache memory consumption is roughly one-quarter of what a full-layer approach would require.

Based on public design figures cited by the creator: if all 36 layers tracked the full document, the cost of handling one million tokens would balloon to roughly 155 GB. Spark X's sparse design compresses that number to about 39 GB.

KV Cache stands for Key-Value Cache, the core optimization mechanism during Transformer inference. In self-attention, each token generates three vectors: Query, Key, and Value. Since text generation is autoregressive — producing one token at a time — recomputing the Keys and Values for all previous tokens on every step would scale linearly with sequence length. KV Cache addresses this by storing previously computed Key and Value vectors in memory for direct reuse, trading space for time. The cost is that memory usage is strictly proportional to sequence length: the longer the sequence, the more key-value pairs are cached, and the greater the memory pressure. Cache size is the product of four factors: number of layers, number of attention heads, dimension per head, and sequence length. This is precisely where Spark X's sparse design pays off — by drastically reducing the number of layers that need to maintain a full KV Cache, memory pressure becomes proportional only to the number of global layers (9) rather than all 36.

The Real Memory Bill for a Million Tokens

A clever architecture alone isn't enough — whether you can actually run it depends on your hardware. The creator provided a few concrete calculations:

• 32,000 tokens: KV Cache is about 1.3 GB; combined with the 4 GB model weights, total usage is roughly 5–6 GB. A typical 16 GB laptop handles this comfortably.
• 128,000 tokens: KV Cache grows to about 4.9 GB; combined with the model weights, total usage is around 9 GB — still well within the comfort zone of a 16 GB machine.
• 1,000,000 tokens: KV Cache alone requires about 39 GB; add the model weights and you're looking at roughly 43 GB — already exceeding a 32 GB machine's total capacity before the OS even starts.

Worth noting: the official SGLang deployment script sets the context window to the full one million tokens at launch. In other words, that enticing "million-token" label on the download page won't give you any advance warning about the enormous memory bill waiting for you at runtime.

Fitting It In ≠ Being Able to Use It

A context window is just the upper limit of text a model can "swallow" — it doesn't mean the model can actually make use of all of it. The standard method for testing this in machine learning is the "needle in a haystack" test: hide a specific fact inside a large volume of irrelevant text, then see whether the model can accurately retrieve it.

As cited by the creator, when Orca Router compared Spark against Gemma 4 (12B) in evaluations, it noted that the advertised million-token capability is an architectural claim and a deployment setting, not a verified everyday capability — at the time, no one had published independently verified needle-in-a-haystack results covering the full context range.

Community tests from the local LLaMA forums are more grounded: testers who ran benchmarks at various lengths reported stable performance up to roughly 128,000 tokens, after which the model started "losing the needle." Testers speculated the model was likely trained primarily at this shorter length. The creator also notes this is an unconfirmed report rather than a complete lab test, and the exact cause has not been established.

The practical takeaway: under 128,000 tokens is the range with real empirical support. Beyond that, you're in unverified design-ceiling territory — you'll need to test it against your own documents before drawing any conclusions.

The Needle-in-a-Haystack (NIAH) test is currently the most widely used benchmark method for evaluating the actual retrieval ability of long-context models, introduced by Greg Kamradt in 2023 and broadly adopted by the community. The method inserts a short target fact (the "needle") at various positions within a large body of irrelevant text (the "haystack"), then asks the model to retrieve that fact. By systematically varying both the haystack length (how full the context window is) and the needle's insertion position (beginning, middle, or end of the text), you can generate a two-dimensional heatmap showing retrieval accuracy across different lengths and positions. A model claiming to support a million-token context window, and a model that can reliably find a key detail within a million tokens, are two entirely different things. The former is a theoretical architecture limit; the latter requires full-coverage NIAH testing to verify. Many models exhibit a "position bias" problem — recall rates are much higher for content near the beginning and end than in the middle — which shows up on heatmaps as a dark "dead zone" in the center.

How Large a Window Should You Set for Your Machine

Based on hardware constraints, the creator offers clear practical recommendations:

• Everyday chat and single-file processing: Start at 32,000 tokens. KV Cache is only about 1.3 GB; combined with the 4 GB 8-bit model, an ordinary laptop has more than enough memory.
• Analyzing a large codebase in one pass: Set 128,000 tokens. A 16 GB machine can handle it, and this falls squarely within the range where empirically tested recall remains stable.
• Million-token context: Reserve this for workstations with 32 GB or more — and before feeding in an entire codebase, hide a fact somewhere and test whether the model can find it.

Whether you're using Ollama, LM Studio, or llama.cpp as your runtime, make sure Spark 2.5 support is properly set up before feeding in an entire codebase. This model uses a cleverly designed sparse attention architecture to prove that a small model can support the "skeleton" of an ultra-long context — but what actually determines practical usefulness is the memory in your machine and the validation tests you run yourself.