Local LLM Speed Benchmarked: Interpreting llmspeed Hardware Data

How fast can a local LLM actually run? This is the first question anyone who wants to turn their hardware into a private AI workstation asks. llmspeed does something very practical: it estimates tokens per second for 30 machines and 15 models, turning "how fast can my GPU run" from folklore into a lookup reference. This article breaks down what its numbers actually mean, the memory bandwidth pattern behind them, and why the prefill stage matters so much for coding agents.

What llmspeed Measures: A 30 Machines × 15 Models Quick-Reference Table

llmspeed's core output is a "hardware × model" speed matrix. Models run horizontally, machines vertically, and each intersection gives tokens generated per second. The value of this organization is that it doesn't answer "how does a given model benchmark" but rather "how much does the same model differ across hardware" and "how much does the same hardware differ across models."

For ordinary users, this solves a high-frequency decision problem: upgrading memory, buying a new GPU, or switching to a smaller quantized version—which investment pays off most. For developers, this table is the starting point for estimating user experience—a local model running at 5 tokens/s and one at 40 tokens/s are completely different products.

It's important to be clear-eyed: these numbers are inherently estimates. Quantization method, context length, batch size, and inference framework (llama.cpp, vLLM, MLX, etc.) all cause measured values to fluctuate. The order-of-magnitude reference from llmspeed is more meaningful than any exact reproduction.

Memory Bandwidth: The Invisible Ceiling on Decoding Speed

The most important pattern behind llmspeed's data is: autoregressive decoding speed is mainly determined by memory bandwidth, not compute.

The reason is mechanical. Generating each token requires reading the model weights (or the currently activated portion) from VRAM/memory once, while the computation is tiny relative to the memory traffic. So decoding speed is roughly "memory bandwidth ÷ bytes that must be read per token." This explains several common phenomena:

• The same GPU running 7B and 70B models often shows a speed gap close to the parameter ratio, because more weights must be moved.
• Apple M-series unified memory's bandwidth advantage lets it often achieve good decoding speeds at the same price tier.
• Quantizing to 4bit doesn't just save VRAM—it directly reduces bytes moved per token, boosting speed.
• When VRAM is insufficient and the model spills into system memory, speed falls off a cliff—essentially because bandwidth switches to a much slower channel.

So when reading llmspeed's numbers, bandwidth is a hardware spec more worth watching than TFLOPS. This also means that if your scenario is single-user, short-output interactive chat, decoding speed is the core of the felt experience; for batch throughput scenarios, the pattern is different.

Division of Labor Between Compute (FLOPS) and Memory Bandwidth: A GPU's peak compute is usually expressed in TFLOPS, measuring how many floating-point operations per second it can complete; memory bandwidth is expressed in GB/s, measuring how much data can be moved from VRAM per second. LLM inference has two fundamentally different stages: prefill is matrix-multiply-matrix, with large compute and relatively small memory traffic—it is "compute-bound" and limited by compute; decoding is matrix-multiply-vector, requiring very little computation per generated token but reading all weights once—it is "memory-bound" and limited by bandwidth. So the same card's bottleneck switches across tasks, which explains why a gaming GPU's high TFLOPS doesn't necessarily make it faster at LLM decoding than Apple silicon with unified memory—the latter often has more ample bandwidth at the same price.

Why Prefill Is Especially Critical for Coding Agents

llmspeed specifically emphasizes prefill's impact, something speed tables often ignore. Prefill is the stage where the model processes the input prompt by computing all tokens in parallel at once; it is compute-intensive, with a completely different bottleneck from token-by-token decoding.

Coding agents happen to be one of the most prefill-stressed scenarios. The reason is direct: code tasks have extremely long contexts—the entire repository, multiple files, error logs, and conversation history stacked together, easily tens of thousands of tokens. Every time the agent initiates a new round of inference, this long context must be prefilled again (unless prefix cache hits).

This has two practical implications. First, time to first token can be far larger than you'd expect from the "tokens/s" number. What the user perceives is "waiting after pressing Enter," not slow generation. Second, inference frameworks that support KV cache reuse and prefix caching see significantly amplified gains in coding scenarios—the same hardware, properly configured, can eliminate a lot of redundant prefill.

Therefore, evaluating whether a machine suits coding agents based only on decoding tokens/s will lead to wrong conclusions; you need to look at prefill throughput and caching strategy together.

KV Cache and Prefix Caching: During autoregressive generation, the model caches the Key and Value vectors of processed tokens (the KV cache) and reuses them when generating new tokens, avoiding recomputing the history. But KV cache is only valid within "the same generation process"; when an agent starts a new request with an identical context prefix, "prefix caching" is needed to reuse across requests. Mainstream frameworks such as vLLM's Automatic Prefix Caching and llama.cpp's prompt cache implement this mechanism. When the prefix cache hits, redundant prefill of tens of thousands of tokens can be nearly skipped, reducing time to first token from seconds to milliseconds. However, the KV cache itself occupies VRAM, and the longer the context, the larger the occupation—one of the important reasons VRAM runs tight in long-context scenarios.

How to Use This Speed Table for Decisions

Applying these numbers to real choices can be layered by scenario:

Interactive chat and writing: Decoding speed is the main metric. Above 20 tokens/s is generally smooth; below 10 tokens/s feels noticeably sluggish. Prioritize hardware with high bandwidth that can hold the quantized version of your target model.

Coding agents and long-context tasks: Prefill throughput and time to first token matter more, and check whether the framework supports prefix caching. The longer the context, the greater this weight.

Batch processing and offline tasks: Throughput (total tokens/time) takes priority over per-request latency; here compute and parallelism become important again, complementing the bandwidth logic of decoding.

One easily overlooked point: llmspeed's estimates are good for ruling out obviously unsuitable combinations (like running 70B on integrated graphics), not for precisely predicting your own measurements. After you have candidate hardware, run a benchmark with your own model, quantization format, and framework—that's the final basis.

Quantization's Dual Impact on Speed and Quality: 4bit quantization is popular in local deployment because it relieves both VRAM usage and bandwidth pressure at once—weights drop from 16bit to 4bit, reducing bytes moved per token to about a quarter, theoretically approaching a 4x decoding speedup. But quantization is not lossless; low-bit quantization (especially 2-3bit or crude 4bit schemes) causes accuracy degradation, manifesting as long-context reasoning decline and higher code generation error rates. Different quantization formats (GGUF's Q4_K_M, AWQ, GPTQ, MLX's 4bit, etc.) also differ in speed and quality. So when reading llmspeed's numbers, confirm which quantization level they correspond to; directly comparing 4bit speeds against 16bit hardware combinations yields misleading conclusions.

Summary

llmspeed's real contribution isn't precise numbers but breaking "local LLM speed" into explainable dimensions: decoding depends on memory bandwidth, long context depends on prefill, and scenario determines the weights. Once you understand these two patterns, the speed table becomes a transferable judgment framework—given a machine not in the list, you can still roughly infer whether it can run and how it will feel.

For heavy long-context workloads like coding agents, especially don't be fooled by a single tokens/s figure; time to first token and cache hit rate are the keys to whether it's actually usable.