No GPU, Real Tests: Performance Differences Between Old and New llama.cpp Running Gemma Models

Running large models locally doesn't necessarily require an expensive GPU. One developer used a pure CPU environment to compare the actual performance of two different versions of llama.cpp on the same quantized model, and the conclusion was somewhat surprising.

Experiment Design: Controlled Variables for Comparison

The core idea behind this comparison was simple: change only the llama.cpp build version while keeping all other conditions identical. The two builds used were build 9976 (commit e3546c794) and build 11571 (commit 097cf33d9), the latter based on llama.cpp's 0.6.0-dev version, compiled from source with GNU 12.2.0 on Linux x86_64.

For the model, both builds loaded the same file: gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf, with Q4_K_XL quantization. The author deliberately locked the model and quantization parameters so that differences would only come from the builds themselves, not from noise introduced by the model or quantization level.

The test environment was entirely CPU-based, with no GPU offloading. Inference threads were set to 3, batch threads to 4, context length to 4128, and parallel slots to 1. Sampling parameters (temperature 0.7, top-p 0.8, top-k 20) and prompts were kept consistent, the service was restarted before each run, and output was fixed at 256 tokens.

This level of control is fairly rigorous for a personal experiment—especially the details of restarting services and clearing caches, which many informal benchmarks overlook.

Quantization and the GGUF Format: The Q4_K_XL in the model filename refers to a specific quantization scheme. GGUF is the model file format used by the llama.cpp ecosystem, packaging weights, vocabulary, and metadata into a single file, and quantizing weights by blocks. Q4 means weights are compressed to about 4 bits, K indicates k-quant block quantization where different tensors can use different precisions based on sensitivity, and XL is typically a community (such as Unsloth) mixed-precision variant further tuned on that basis. Compared to FP16, 4-bit quantization reduces model size and memory usage to about one-quarter—this is precisely the key prerequisite that allows ordinary CPU machines to run local large models. The trade-off is a slight loss in precision, so different quantization levels of the same model show perceptible differences in speed and quality. The author locked the same .gguf file specifically to eliminate quantization differences as a variable.

Performance Data: Generation Faster, but Prefill Slower

Each build was run three times, and averages were taken. The results showed a "half good, half bad" pattern:

Metric	Old Build 9976	New Build 11571	Change
Generation speed (tok/s)	6.05	6.60	+9.1%
Prefill speed (tok/s)	11.91	11.14	-6.5%
Time to generate 256 tokens	42.57 s	38.64 s	Saves ~4 seconds

The new build was on average about 9.1% faster in text generation, reducing 256-token generation from 42.57 seconds to 38.64 seconds. But prefill (prompt processing) was instead 6.5% slower, dropping from 11.91 tok/s to 11.14 tok/s.

This highlights an often-overlooked fact: LLM inference performance is not a single number. Prefill and generation are two computationally distinct phases—the former processes the input sequence in parallel, while the latter produces autoregressive output token by token. Build optimizations may only affect one of these phases, so it's not accurate to broadly claim "the new version is faster."

Why Prefill and Generation Are Two Different Workloads: LLM inference has two phases. Prefill (prompt processing) sends the entire input into the model at once, allowing all tokens to be computed in parallel—it's a high-throughput, compute-intensive matrix operation, typically measured in "input tokens processed per second." Generation (decode / generation) is autoregressive: each token produced requires running through the model again, and the next token cannot be computed until the previous one is done, making it highly dependent on memory bandwidth and caching, difficult to parallelize, and measured in "output tokens per second." The two have different hardware bottlenecks—prefill is compute-bound, generation is bandwidth-bound—so optimizations targeting one phase (such as operator fusion or KV cache strategies) can absolutely make the other phase slower. This is why broad claims like "the new version is faster" often don't hold up in LLM inference.

Run-to-Run Variance: Why Average Over Three Runs

Data from a single run actually fluctuates considerably. The old build's generation speed ranged from 5.41 to 6.45 tok/s, while the new build ranged from 6.39 to 6.82 tok/s. Prefill fluctuated even more—the new build's three runs were 13.45, 10.62, and 9.36 tok/s respectively, a range of over 4 tok/s.

This kind of variance is common in CPU-only environments, where background processes, thermal management, and memory bandwidth contention all affect results. The author used three-run averages to reduce the randomness of single samples but acknowledged that three samples are still insufficient for statistical inference.

Reading this data practically: a 9.1% generation improvement isn't a qualitative leap, but on a pure CPU machine, saving a few seconds each time adds up to something noticeable in daily use.

Bonus Finding: Load Model on Demand

The new build also introduced a noteworthy feature—load model on demand. For users with limited hardware resources, this direction offers more potential than pure inference acceleration.

However, the author did not measure this feature's actual effects in this experiment, including service startup time, on-demand model loading time, RAM usage before and after loading, and memory release behavior when idle. These are dimensions independent of inference speed and would require separately designed experiments to verify.

This attitude of "noticing new features but not drawing conclusions" is commendable—the value of a benchmark lies in honestly delineating what has been tested versus what hasn't.

The Significance of Load Model on Demand: Traditionally, llama.cpp's service reads the entire model weights into memory (RAM) at startup and keeps them resident. This is unfriendly to machines with limited memory—a medium-sized 4-bit quantized model can occupy several GB, and with KV cache and system overhead, it's easy to run tight. The idea behind "load model on demand" is lazy loading: instead of loading all weights immediately at service startup, it reads them from disk only when the first request arrives and a specific parameter is actually needed, and may release already-loaded portions back to memory when idle. This can significantly reduce the service's idle memory footprint and speed up service startup, but it may also shift loading overhead to the first request, increasing time-to-first-token. Whether it's a memory-saving tool or a new variable introducing jitter must be determined by actually measuring startup time, loading latency, and RAM curves—not just by the feature name.

Practical Recommendations

The author ultimately chose to continue using the new build, for three reasons: generation speed showed an observable improvement, the prefill regression is barely noticeable in actual interaction, and the new features are worth following up on. But he also made clear that this decision wasn't about chasing benchmark numbers—it was based on real work scenarios: writing documentation, drafting formal letters, handling daily tasks.

For users who want to run local large models on ordinary computers, this experiment offers several reference points:

• Keep the model and quantization consistent, change only one variable, to determine whether a build upgrade is worthwhile
• Look at prefill and generation metrics separately; don't be misled by a single number
• Run-to-run variance is significant in CPU-only environments; averaging multiple samples is more reliable than picking the fastest run
• When paying attention to new features, first confirm whether they've actually been measured

The barrier to local LLMs is lowering. Ordinary laptops or old desktops without high-end GPUs, combined with appropriate quantization and build versions, can already run usable models. This accessibility itself may be more meaningful than a few extra percentage points on a benchmark.

One-line Summary

Pure CPU comparison of two llama.cpp versions: new build generates 9.1% faster, but prefill is 6.5% slower.

Paragraph Summary

A developer compared llama.cpp build 9976 and build 11571 in a pure CPU environment using the same quantized model (Gemma series Q4_K_XL), changing only the build version while locking all other conditions, running each build three times and averaging. Results showed the new build's generation speed improved from 6.05 to 6.60 tok/s (+9.1%), with 256-token generation time dropping from 42.57 to 38.64 seconds; but prefill speed instead dropped from 11.91 to 11.14 tok/s (-6.5%). This shows inference performance is not a single number—prefill and generation are two distinct phases. The experiment also observed significant run-to-run variance; averaging three runs is more reliable than picking the fastest, but the sample size is still insufficient for statistical conclusions. The new build's load-model-on-demand feature was not measured in this experiment. The author ultimately chose the new build based on real usage scenarios.