Why Are Local LLMs Suddenly So Good? Three Barriers Were Removed One by One

From Nonsense to Fluent Answers: What Happened to Local Models

Two or three years ago, running an open-source model on a laptop and asking it "Why is the sky blue?" would produce something like this: first it would call it a "common controversy," then offer two contradictory answers—one about fog and tiny particles in clouds reflecting light, another about the Earth's own temperature—before adding "currently there is no..." and only on the second pass circling back to "there is no universally accepted explanation." The model could start, could converse, could understand Chinese, yet couldn't answer a basic common-sense question.

Today, on the same machine, offline, with the same question, the answer is normal. What exactly happened in between? Breaking down "good" reveals three barriers: fitting in memory, being able to reason, and running fast. These three barriers were stuck at different times, and were released in different ways.

Barrier One: Fitting In — Quantization Squeezes Models into Ordinary Memory

Model weights are originally 16-bit floating point numbers, two bytes each. An 8-billion-parameter model has 16GB of weights alone, and most laptops can't spare that much memory. The first step is inevitably compression, via a method called quantization: storing each number with fewer bits, from 16 down to 8, then down to 4.

Compression always has a cost, but fortunately the cost can be measured. This measurement is called perplexity—how "surprised" the model is by the next piece of text; lower is better, and the more it rises, the more confused the model becomes. According to Llama CPP's records on the same 8-billion-parameter model: 8-bit rises by 0.003, 6-bit by 0.02—essentially lossless; 4-bit rises by 0.175, shrinking the file from 8GB to 4.5GB; 3-bit rises by 0.657, nearly 4× that of 4-bit; 2-bit jumps straight to 3.5×, and the model is essentially ruined.

Following the bits downward, 4-bit is the last gentle slope; 3-bit starts to get bumpy, and 2-bit falls off. Another group verified the same thing on math problems: 16-bit accuracy 77.6, 4-bit 77.4—almost unchanged; 3-bit drops to 68.3. So the value of 4-bit isn't "smaller"—it's stopping right at the edge of the cliff. At 4-bit, the model becomes small enough to fit on ordinary machines: a 7B model is just over 4GB, a 14B is 9GB, a 32B needs 20GB—that last one crosses the 16GB most laptops can spare. What runs reliably is the 7B and 14B class. The model is still the same model, just stored differently.

Perplexity is a common metric for measuring a language model's predictive ability, defined as the inverse of the geometric mean of the model's predicted probabilities for each word in a test text. Intuitively: the more "surprised" the model is by the next word, the higher the perplexity; the more accurate the prediction, the lower the perplexity. It requires no manual annotation—just a piece of text—so it's often used to quickly compare the loss of the same model at different quantization precisions. But perplexity has limited sensitivity to quantization damage—when perplexity rises slightly, the model may show no difference in simple conversation, while already degrading noticeably on tasks requiring precise reasoning like math and code. So in practice it's often cross-validated with downstream task accuracy (e.g., math problem accuracy) to avoid underestimating quantization cost by looking only at perplexity.

Barrier Two: Being Able to Reason — Distillation Moves the Reasoning Process into Small Models

Once it fits, you immediately hit the second wall: a 7B model, even at full precision, struggles to reason step by step.

In January 2025, DeepSeek^Ad pushed this limit back a step. They first trained a 671B model with reinforcement learning, then had it write down its problem-solving process—800,000 entries in total—using the large model's output to teach small models. This method is called distillation. They fine-tuned six sizes of small models with this teaching material, from 1.5B up to 70B. The 1.5B distilled version scored 28.9 on math competitions, while GPT-4o at the time scored 9.3; but more telling is the comparison at the 32B tier: the same base model doing reinforcement learning directly scored 47; switching to distillation, still that same base, scored 72.6. The size is identical—only where the training signal comes from changed. The paper says it directly: a small model fumbling on its own is no match for moving the large model's reasoning over.

That's the solution to the second barrier: it moves the method, not the size.

Knowledge Distillation was first proposed by Hinton et al. in 2015. The core idea is to have a small model (student) imitate a large model's (teacher's) output distribution, not just hard labels. Traditional distillation transfers "soft labels"—the teacher's probability distribution over each class, which encodes similarity information between classes. In the era of large language models, distillation has evolved into two forms: first, output distillation, where the student directly fits the teacher's token probabilities; second, reasoning chain distillation, the approach DeepSeek adopted in this article—the teacher generates complete solution steps (chain of thought), and the student fine-tunes with these steps as supervision signals. The latter transfers not just the answer, but the process of "how to think step by step to the answer," so its improvement on small-model reasoning is especially significant. The limitation of distillation is that the student can only learn the abilities the teacher has already demonstrated and cannot surpass the teacher's knowledge boundary.

Barrier Three: Running Fast — Mixture of Experts Makes More Parameters Faster

Distillation moves capability, not size. A 32B model at 4-bit still needs 20GB of memory; wanting stronger usually means bigger, which means it fits even less. In 2025, Mixture of Experts split this contradiction apart: cutting the model's interior into many experts, with each word passing through only a few of them. Qwen3-30B-A3B has 30.5 billion total parameters but activates only 3.3 billion per word; GPT-OSS-20B has 21 billion total parameters but activates 3.6 billion per word.

The key is in the decoding step: speed depends on how many bytes are read. All experts must be kept in memory, but each word only reads the selected few. So memory is counted by total parameters, speed by activated parameters. On the same machine, same 4-bit, the 30B mixture of experts does 95 tokens per second, while the dense Qwen3-32B does only 34—more parameters, yet more than twice as fast. The third barrier isn't computing faster; it's reading less each time.

Mixture of Experts (MoE) has as its core components a "gating network" and several "expert networks." For each input token, the gating network computes weights for each expert, selecting only the top-K highest-scoring experts to participate in computation (commonly K=2), leaving the rest unactivated. This way total parameters can be very large, but the computation and memory reads per forward pass depend only on activated parameters. MoE is not a new concept—it was proposed as early as 1991—but it wasn't until 2017, when Shazeer et al. introduced sparse gating into large-scale neural networks, that it showed advantages in language models. Its cost is high memory usage—all expert weights must reside in memory—and training is prone to uneven expert load (a few experts overused), requiring additional load-balancing losses to constrain.

Three Things Stacked on the Same Timeline

Put these three things on the same timeline, and the suddenness vanishes: March 2023, Llama CPP let Meta models run on ordinary computers; July of the same year, Ollama wrapped it into a single command; after that, 4-bit became the default tier offered by download tools; January 2025, distilled small models open-sourced; April and August of the same year, two mixture-of-experts models released. Each step is small, but they stacked together. The "suddenly good one day" you feel is the moment the last step was awarded.

After Release, New Boundaries Are There Too

Where these three barriers were released, new boundaries were also left behind. Mixture of experts saves speed, not memory—all experts must be present: GPT-OSS-20B needs 14.9GB at 8000 context, rising to 17.9GB at 130,000 context; the longer the context, the slower decoding—on the same machine, from 500 tokens to 32,000, speed drops from 46 to 30. Distillation also has a ceiling; it moves reasoning methods, not new knowledge—the paper itself says breaking the capability boundary still requires a stronger base. Quantization is even more of a hard floor: below 4-bit isn't gradual degradation, it's a direct cutoff.

So local models didn't suddenly get smarter; the three barriers of fitting in, reasoning, and running fast were removed one by one. The conversation you take for granted rests on three different things over three years. The next time it gets better again, it'll likely follow the same pattern: some specific constraint gets released.

The relationship between context length and memory deserves separate mention. Transformer's attention mechanism needs to cache the Key and Value vectors of each generated token during decoding (the KV Cache), whose size grows linearly with context length. For dense models, the KV Cache is usually small compared to model weights; but for mixture-of-experts models, weights already fill the memory headroom, so KV Cache growth becomes the last straw. That's why GPT-OSS-20B takes 14.9GB at 8000 context and rises to 17.9GB at 130,000 context—the extra ~3GB is almost entirely KV Cache. Moreover, the longer the context, the heavier the attention computation, and decoding speed also drops. This explains the phenomenon in the article that "the longer the context, the slower decoding," and shows that long-context capability in local deployment is limited by memory rather than compute.

One-line Summary

Local models got good because three barriers—quantization, distillation, and mixture of experts—were released one after another.

Paragraph Summary

The article breaks down local models' transformation "from nonsense to fluent answers" into three barriers: fitting in memory, being able to reason, and running fast. Fitting in relies on quantization, with 4-bit the balance point between precision and size—below 4-bit performance falls off a cliff; reasoning relies on distillation, letting small models learn the reasoning process of large models—a 32B model jumps from 47 to 72.6 after distillation; running fast relies on mixture of experts—memory counted by total parameters, speed by activated parameters, a 30B MoE more than twice as fast as a 32B dense model. These three landed in sequence between 2023 and 2025, stacking into the "suddenly good" users perceive. Meanwhile new boundaries remain: MoE doesn't save memory, distillation brings no new knowledge, and quantization has a hard floor.