Before Running LLMs Locally, Ask Two Questions: Does It Fit? Is It Actually Fast?

Why Your Local LLM Deployments Keep Failing

Many people run into the same trap the first time they try to run a large model locally: either the hardware simply can't fit the model plus the memory needed for context, or they trust a tokens/s number measured under completely different conditions. This isn't a product pitch—it's a repeatedly validated checklist, plus a small tool written because doing the arithmetic over and over was too much hassle.

The root of the problem is that people conflate "does the memory fit" with "how fast is it," and these two things require completely different kinds of evidence.

1. Does the Memory Fit? This Is Arithmetic You Can Do Yourself

Weights, KV Cache, and Headroom

Memory fitting is an arithmetic problem you can work out by hand. Start with the storage size of the quantized weights, add the KV cache for the context you actually need, then leave headroom for the runtime, drivers, and desktop environment. A 70B model at Q4 quantization is roughly 40 GB of weights—before any other overhead. So "it runs on a 24 GB GPU" usually means the configuration offloads to system memory and then runs very slowly.

This calculation requires no benchmarks, just honesty about your own hardware. Do the math first, then decide whether to buy.

KV cache is the most commonly overlooked part of understanding memory fit. During autoregressive generation, a Transformer caches the key (Key) and value (Value) vectors of all previous tokens at every layer to avoid recomputation; this memory/VRAM footprint is proportional to "layers × 2 × hidden dimension × context length × batch size × dtype bytes." It grows linearly with context: the same model might need only a few GB at 2k context, but more than ten times that at 32k. This is why "the model fits" doesn't mean "long context also fits"—weights are a fixed cost, while KV cache is a variable that expands with how you use it. When estimating, you usually also need to reserve 10%–20% of headroom for the computation graph, temporary activations, and fragmentation.

Quantization means storing weights in a lower-bit numeric type. Common formats include GGUF (the llama.cpp ecosystem), GPTQ, AWQ, and MLX's 4bit/8bit schemes. Q4 roughly means about 4 bits per weight, so 70B parameters is about 35–40 GB, including some unquantized layers and metadata overhead. Lower quantization saves more memory, but precision loss, which runtimes are supported, and whether GPU acceleration is available all vary by format—which is why "the same model, different quantization" speeds and results can't be compared directly.

2. Speed Is Measured, But It Must Come With Conditions

Seven Things Behind a Single Number

Speed is a measured value that only means something when accompanied by conditions: the exact hardware, the exact checkpoint, the quantization method, the runtime and its version, the context and prompt length, the concurrency, and how the number was produced (prefill / TTFT / decode). A single-stream decode number does not equal throughput under eight concurrent requests. A vendor's marketing claim does not equal a measurement. If the source doesn't list all of this, the honest answer is "unknown," not a green checkmark.

In particular: prefill time and tokens/s are not interchangeable. If all you have is TTFT, what you're seeing is a derived prefill estimate and should be labeled as such.

Prefill, TTFT, and decode are the three most commonly confused metrics for describing inference performance. Prefill is the stage where the model processes your entire input prompt at once—a highly parallel matrix operation that's usually fast, and whose duration determines TTFT (Time To First Token). Decode is the stage of token-by-token autoregressive generation, computing only one token per step, limited by memory bandwidth, and it's the main factor determining sustained generation speed (tokens/s). So "tokens/s" generally refers to decode throughput, while "TTFT" reflects prefill plus queuing latency. If only TTFT is given, you can only infer prefill speed and cannot deduce how fast decode is—the two describe completely different workload characteristics, and conflating them will seriously distort capacity planning.

3. Choose the Runtime First, Then Think About What to Buy

The runtime determines what's possible. CUDA, ROCm, Metal, Vulkan, CPU, and NPU are separate lanes with separate lower bounds, and a feature existing in one lane doesn't prove it exists in another.

On Apple Silicon, MLX is the native path; on NVIDIA servers, vLLM or SGLang is the throughput path; on CPU+GPU hybrid machines, offloading runtimes for mixture-of-experts models are yet another trade-off. Decide on the runtime first, then look at hardware—get the order backwards and you'll likely waste money.

Turning Apple Silicon Into an OpenAI-Compatible Endpoint

If you use MLX, there's a detail worth knowing: mlx-lm ships with mlx_lm.server, an OpenAI-compatible HTTP server. It exposes /v1/chat/completions, /v1/completions, and /v1/models, listens on 127.0.0.1:8080 by default, and accepts --model plus --host / --port arguments. That's convenient for pointing an existing OpenAI client at your Mac.

An important warning comes from upstream's own documentation: mlx_lm.server only implements basic security checks and is not recommended for production environments. Unless you put authentication and network controls in front of it, keep it bound to localhost. A single-user local setup is fine; any externally exposed scenario is not.

4. Write Down the Conditions

Whatever you measure, record the conditions next to the number. Six weeks later, "42 tok/s" without "M5 Max, Q4, 8k context, single-stream, mlx-lm x.y" is useless to you and misleading to others.

If you want a starting point, there's a continuously maintained reference that presents memory fit, practical context, measured speeds, vendor claims, and unknowns separately, with every number linked to its source: LocalForge. It's a decision guide, not a benchmarking lab—it doesn't publish its own numbers.

FAQ

Why would a 70B model run very slowly on a 24GB GPU?

Because a 70B model at Q4 quantization has roughly 40 GB of weights, far exceeding 24 GB of VRAM, so it must be offloaded to system memory, which drastically reduces speed.

Can tokens/s numbers be compared directly?

No. You must confirm whether the hardware, quantization method, runtime version, context length, and concurrency match—otherwise the numbers aren't comparable.

Can mlx_lm.server be used in production?

Upstream explicitly does not recommend it. It only implements basic security checks; keep it bound to localhost, or add authentication and network controls in front.

Background

MLX is Apple's array computing framework designed specifically for Apple Silicon's unified memory architecture. It can directly use memory shared between CPU and GPU without explicit copying, so on a Mac it's often more efficient than general-purpose solutions. vLLM and SGLang, by contrast, are server-oriented inference engines whose core is optimizations like PagedAttention and continuous batching, managing KV cache in pages to improve throughput under multiple concurrent requests—suited to NVIDIA server scenarios. The llama.cpp/GGUF ecosystem excels at hybrid CPU+GPU offloading, especially for placing some layers in system memory when VRAM is insufficient. Runtime differences show up not only in speed but also in supported quantization formats, concurrency capabilities, and hardware features—which is the underlying reason for "choose the runtime first, then the hardware."