Stanford NCM Paper Fact-Check: Where Exactly Does the 31x Speedup Come From?

Title

Stanford NCM Paper Fact-Checked Line by Line: Where Exactly Does the 31x Speedup Come From?

Meta Description

Stanford's NCM paper claims 31x faster text generation, but a line-by-line check shows it's compared against diffusion models, not GPT. This article breaks down NCM's multi-scale residual architecture, the real premise behind the 31x figure, its differentiated impact on local inference vs. data centers, and the three limitations keeping it off your machine today.

Article Content

At 2 a.m., staring at the blinking cursor on your screen, waiting for the model to finish a 2,000-word analysis, you probably start doing some unflattering math: wouldn't I have finished writing these 2,000 words myself by now? It's 2026, and the GPU in your hands can render 3D scenes in real time, yet it has to stop and catch its breath after every single word.

That's when a team at Stanford posted something on X, and 500,000 people went to read about a term called NCM—Next Concept Modeling. The word going around was that it could write text 31 times faster, and was even more accurate on certain tasks. So the clickbait machine kicked into gear: large models are finished, GPT is dead.

According to a line-by-line verification of the original paper and the open-source code repository by a Bilibili creator, as well as a YouTube channel's interpretation of the same paper, both parties reached the same conclusion about what the "31x" figure actually means: it is not being compared against GPT at all. This article will walk you from the hype back down to the ground—what NCM actually is, whose rice bowl it disrupts, whose it doesn't, and what it means for you.

The Bottleneck of the Old Architecture: Read All Weights for Every Word Written

Every chatbot you use today writes one token at a time, where a token is roughly a word, sometimes half a word. Architecturally, there's an iron rule: the 500th token must wait until the 499th is completely finished before it can begin. If you want a 1,000-token answer, it takes 1,000 steps, and every step costs the same amount of effort.

And to write that single token, the GPU has to read the model's entire weights from VRAM from start to finish. To put it another way: writing an article means that for every word you write, you first have to read that 800-page writing textbook from the beginning, then read it again for the next word. That is exactly what AI does computationally when writing from left to right. The smarter the model and the more parameters it has, the more weights there are and the harder it is to move—the very thing that makes AI useful is the very thing that makes it slow.

This isn't a vague wish of "it would be nice if it were faster"—it's a measurable bottleneck in the architecture that burns real time and real money every day. The Stanford folks say they've found a way to cut the number of weight reads by a large margin when writing answers, which is why 500,000 people clicked in.

But I've been burned by this kind of excitement too many times. So now I look at the paper before the cover: I've gotten excited about a model before, only to find out the benchmark questions were its training set; I've gotten excited about speed numbers before, only to find out they were measured at batch size 512. But I work from home and I'm not running 512 conversations at once.

Autoregressive Generation is the standard way nearly all large language models work today: the model predicts the next token one at a time based on the sequence of tokens already generated, appends the new token to the input, and continues predicting. This pattern is mathematically equivalent to a chain decomposition of the joint probability of the sequence, which makes the training objective simple and optimization stable. But it has a hard physical consequence: generating the Nth token requires waiting for all N-1 previous tokens to complete, with no parallelism. More critically, every token generated requires moving all model parameters from VRAM to the compute units once, and the transfer time often far exceeds the computation itself. Therefore, the generation speed of autoregressive models is primarily determined by memory bandwidth, not compute—which is why high-end consumer GPUs running large models locally often achieve tokens per second far below what their theoretical compute would suggest.

How NCM Works: Rough Sketch First, Details Later

You probably think 31x faster means finding a faster way to pick words. Quite the opposite—it picks words only at the very end. It doesn't write from left to right; it writes from top to bottom, layer by layer.

The easiest picture to understand is a portrait. The old method is to paint the final draft stroke by stroke, from left to right, with no going back and no rough drafts allowed. NCM is a different kind of painter: first blocking in the large shapes, then laying in light and shadow, then refining details, leaving the eyelashes for last.

Broken down, there are two pieces. The first is a separate compression network that compresses a passage into several layers of codes shaped like a pyramid: the top of the pyramid has only a few codes—a blurry sketch of the entire paragraph's meaning and structure; each layer below is longer and stores only what the layers above didn't capture; at the bottom layer, there's one code per token. All layers stacked together go into the decoder, and the original text is reconstructed. The paper calls this multi-scale residual VQ.

The second piece is a standard Transformer, except that instead of predicting the next token, it predicts an entire layer of codes in one go, completing one layer per forward pass before moving to the next. So an eight-layer model takes eight passes, regardless of how long the output is. The paper demonstrates step by step: after the first pass, the output is a half-finished product with the right general direction but every single character wrong; by step 11, it's already a clean, accurate sentence.

One thing to clarify here: "next concept" has nothing to do with any advanced semantics or ideas—it refers to a number drawn from a codebook, essentially a numeric category. And it has absolutely no relation to Meta's earlier "Large Concept Model"—their concept is a vector for an entire sentence. Same word, completely different things.

Vector Quantization (VQ) is a class of techniques that maps continuous vectors to discrete indices in a finite codebook, commonly used in image compression and generation. The basic approach is to maintain a learnable codebook; after the encoder outputs a continuous vector, nearest-neighbor lookup replaces it with the closest codeword in the codebook, representing the entire piece of information with an integer index. The decoder reverses this by looking up the table to reconstruct. Multi-scale residual VQ introduces a hierarchical structure on top of this: the first-level codebook captures only the coarsest overall information, the second level encodes the residual that the first level failed to reconstruct, and so on. This way each layer only needs to handle the "missing part" of the layer above, resulting in fewer total codes and more compact information, while naturally forming a coarse-to-fine generation order. NCM leverages this hierarchical structure to have the Transformer predict an entire layer of discrete codes at once, rather than predicting token by token.

It's Not the Same as Diffusion Language Models

Both write multiple positions at once over several passes and both claim to be faster than word-by-word generation, but there's a critical structural difference.

A typical diffusion language model starts with an entire block of empty slots and touches every single slot on every pass. The first pass touches all slots, and the fiftieth pass still touches all slots, so every pass costs the same, and if you want to save time by running fewer passes, quality drops fast. NCM is the opposite: once a layer is done, it never goes back. The first few passes only handle the few codes at the top of the pyramid—just a handful of positions, almost no work, so it's very fast; as layers get finer toward the end, only the final pass touches the full text.

In one sentence: diffusion revises the entire draft many times, while NCM writes the outline once and the details once. This is the root of why NCM's early passes are cheap. And it's precisely why almost every comparison group in the paper is a diffusion model—the authors were pitting it against the strongest branch of parallel writing all along.

Diffusion Models were originally the mainstream paradigm for image generation: during training, noise is gradually added to data until it's pure noise; during inference, starting from pure noise, a learned denoising network iteratively recovers a clear sample. When applied to text, the token sequence is typically treated as discrete states, and all positions are simultaneously denoised or corrected in each round of iteration. Since every round must process all positions, the total computation is roughly "number of iterations × sequence length"; more rounds means better quality but slower speed. The key difference between NCM and diffusion models lies in computation allocation: diffusion models have fixed and uniform computation per round, while NCM's early rounds only process a small number of high-level codes with extremely low computation, and only the final round covers all positions. Therefore, NCM's "early rounds" are nearly free, while diffusion models' "early rounds" are just as expensive as their "late rounds."

The 31x Math: It's Compared Against Diffusion Models, Not GPT

This 31x comes from an experiment: summarizing news articles at a fixed length per summary, NCM takes 8 passes while the strongest diffusion model it's compared against takes 128 passes, with roughly the same quality score—NCM finishes 31 times earlier. The 31x is measured, not made up.

But two lines in the experimental setup carry far more weight than the cover image. First, the generative model itself isn't large in parameters, but the compression network is a separate set of parameters—without it, not a single word can be generated. Second, the number of passes is fixed per model, not per answer: the summarization model always uses 8 levels, the translation model always 11, and the web text one 32. Whether you want a three-word answer or a 300-word one, the number of levels is the same, and the cost is the same.

The most important sentence: the entire paper never once times a mainstream autoregressive model—every timing comparison in it is NCM vs. diffusion models. So that 31x is not 31x faster than GPT; it's 31x faster than a certain diffusion model on a news summarization task of a specific length. The paper has one spot where the authors are quite bold: in the German-English translation task, they included an unremarkable, word-by-word autoregressive model as a control, and it actually won by 2.6 points. NCM closed the gap using minimum Bayes risk decoding, but that came at roughly twice the compute of its opponent. For news summarization, the wind blows the other way—NCM wins decisively.

So scoring by the authors' own data: NCM wins one, word-by-word wins one, and NCM wins big on structured reasoning. This is a genuinely interesting paper with real contributions, but it's not the revolution the headlines suggest—it's a new tool that's strong in some places and weak in others.

What It Means for Local Players: The Bottleneck Shifts from Bandwidth to Compute

The real story isn't at OpenAI, isn't at Google—it's on the machine on your desk.

When a word-by-word model runs on a GPU, the GPU isn't doing what it was designed to do—most of the time it's waiting for weights to travel from VRAM to the compute cores. What's stuck isn't the compute, it's the bandwidth. Switch to NCM and the math changes completely: for an answer of the same length, the number of weight reads drops dramatically, the transfer volume drops proportionally, and on the same machine with the same VRAM budget, you save a big chunk of time.

But the saved transfers don't just vanish—they turn into compute. NCM processes more positions than output words, and every position consumes compute, so it's trading a bandwidth problem for a compute problem—a trade anyone who's watched a model read a long prompt is familiar with: that's the prefill moment, where all positions are computed together, relying on compute, not waiting on VRAM.

There are very practical consequences. Today, when choosing a machine to run AI locally, you look at memory bandwidth, because generation is bottlenecked right there. But if approaches like NCM become widespread, the bottleneck shifts to pure compute. Most unified memory setups have impressive bandwidth but mediocre graphics compute—in the long run, that's not good news. But that's not something you need to worry about today.

Prefill and Decode are the two stages of large model inference. The prefill stage processes the entire prompt input by the user; all tokens can be computed in parallel, making it compute-intensive—time correlates with input length but parallelism is high. The decode stage generates output tokens one by one, processing only one position per step but reading all weights, making it bandwidth-intensive. NCM's approach is essentially transforming the generation process into a parallel compute pattern similar to prefill: multiple positions processed per forward pass, weight reads amortized, compute density increased. This also explains why NCM shows clear gains in single-stream local inference—where bandwidth is the main bottleneck—while in high-batch-size data center scenarios, the decode stage's weight read cost has already been amortized by batching, so NCM's parallelization gains are no longer prominent and may even waste compute due to fixed output length.

In the Data Center, the Premise of Revolution Doesn't Hold

Go to a data center and you'll find the premise doesn't hold. OpenAI isn't serving just you: the sentence you send to ChatGPT gets batched together with hundreds or thousands of other requests, weights are read once, and that entire batch of conversations all uses them. The drudgery of reading all weights for every token is amortized across the whole batch. At that scale, the GPU is already busy—batch size is there, and the chip is doing real compute. In that environment, trading bandwidth for compute shows no obvious benefit and might even make each answer more expensive for the provider. Google's notes on Diffusion Gemma state that the impressive speed numbers were measured at low batch size; the team that released the good results also wrote clearly that every measurement had batch size 1—one conversation at a time. One comment put it most bluntly: text diffusion only wins in single-stream inference—one person, one GPU, one model.

That's you at 2 a.m. at home, cold coffee, blinking cursor. So if parallel writing is going to pay off anywhere first, it will pay off on the desktop first. Ironically, the headlines frame it as a threat to the giants, but structurally the giants are precisely the ones least afraid of that old bottleneck—the ones most easily stuck are you.

Why NCM Still Can't Run on Your Machine Today

The first limitation is structural: every NCM model writes into a fixed-size box—the summarization model always generates a fixed number of positions, and the web text one always fills its level. Whether you want 5 tokens or 5,000, it fills that many positions regardless. Type short and compute is wasted on empty slots; type long and it either truncates or you need a bigger model. For building chatbots, this is a hard flaw—within a single conversation, question lengths can vary by several times.

Then there's scale. The largest NCM in the paper is pitifully small by today's standards—people seriously running models at home are used to seven or eight billion parameters. Whether this path scales up, whether the quality advantage holds, and whether training becomes prohibitively expensive are all open questions.

Finally, practicality: the code on GitHub is evaluation-only—pretrained models can be downloaded and run, but the training pipeline isn't released, and the whole thing depends on a specific vendor's GPUs and system. The model most relevant to general use, the web text one, has a folder on Hugging Face containing a single file named todo, five bytes—that's the entire deployment status of the NCM model most deserving of the headlines.

There's also a quality issue worth calling out separately: when the compression step makes mistakes, the errors are highly concentrated—proper nouns and numbers get swapped out. Names of people, places, parameters, amounts. If you're using AI to handle configurations, prices, people, dates, or statistics, that's exactly where you can least tolerate it swapping a name or writing a number wrong.

An Alternative You Can Run Tonight

There is one, but it's not NCM. Google DeepMind's Diffusion Gemma was released on June 10 this year—25.2 billion total parameters, 3.8 billion active at a time in a mixture-of-experts approach, Apache 2.0 license, 15 to 20 tokens per forward pass, nearly 700,000 downloads on Hugging Face. The speedup in single-stream inference is real—on your machine, in your one-conversation scenario, it's noticeably faster than an autoregressive model of the same size. But Google put the quality and speed numbers side by side without hiding anything: on a math competition benchmark, Diffusion Gemma scores 69.1 while Gemma 4 of the same size scores 88.3; on a code benchmark, 69.1 vs. 77.1. The question becomes what exchange rate this speed is bought at—for summarizing, rewriting, drafting, and casual chat, it's a worthwhile trade; but if you ask AI to write code, one wrong variable name or one number off and it won't compile, and an hour is wasted—that math doesn't work out. Also, some users report its KV cache is absurdly large—a 32GB GPU barely fits much context—so check your memory before getting excited.

Conclusion

NCM is real, the paper and code are real, the numbers are indeed impressive, and the structure is new—writing by level of abstraction rather than linear order, while cutting bandwidth requirements by a large margin. It deserves serious attention. But NCM is a language model—a Transformer trained on text that excels at text—and by any reasonable definition, NCM is a large model. You can't end large models with a better large model.

What it actually threatens is much narrower, and worth thinking about: the habit of large models writing from left to right, one word at a time, reading all weights for every word. That habit is a bottleneck, and it's more your bottleneck than OpenAI's—you don't have the luxury of batching thousands of conversations to amortize costs. This paper doesn't change what you should buy this month; every model you run tonight still writes one token at a time, and what determines how fast it feels is still that memory bandwidth line. But next time you look at a spec sheet, start looking at the compute line too, not just the bandwidth line. If this momentum continues, the bottleneck will move. Keep buying bandwidth today; start watching compute from tomorrow.