A Practical Guide to Choosing Local LLMs: Stop Obsessing Over Parameters and Leaderboards

Running large language models on your own machine has obvious appeal: your data stays local, there's no per-token billing, and you can work offline. But most people start their model selection process on the wrong foot — they begin by looking at parameter counts or some leaderboard ranking.

The problem is that a model great at code completion might perform poorly at summarizing long documents, while a model that "fits" on paper can easily blow up your memory once conversations get long. The reliable approach isn't chasing the highest score — it's using a structured process to match your requirements, hardware, and model together.

Step 1: Start with the Task, Not the Model

Before comparing models, define clearly what you actually need them to do. Four questions can mostly scope this out:

Privacy: Can the data leave the machine at all?
Latency: Do you need interactive, real-time responses, or is this a batch processing job?
Quality: What kind of errors would make the output completely worthless?
Modality: Text only, or do you also need to handle images, audio, and files?

The metrics that matter vary completely by scenario. Here's a clear reference table:

Scenario	Priority	What to Measure
Chat & Writing	Perceived latency	Time to first token, consistency
Long Document Processing	Usable context	Detail retrieval and citation accuracy
Coding	Correctness	Tests passed, unnecessary changes
Batch Processing	Throughput	Tokens per second, energy cost

Local LLM Selection Process

Get the task and its success metrics clear first — every subsequent step depends on having a real basis for judgment. Otherwise you're just letting someone else's benchmarks make decisions for you.

Step 2: Budget Your Memory, and Leave Headroom

Available memory (RAM or VRAM, depending on how you run the model) is almost always the real bottleneck. The most common mistake is treating the model file size as the total memory cost.

In reality, context, attention cache, and the runtime itself all require additional space, and the operating system takes a cut too. In other words, an 8GB model file absolutely does not mean 8GB of memory is sufficient.

A sensible planning order: first define your input and quality requirements, set a realistic memory budget, then select the size and format that fits within that budget, and finally test with your own use cases. The key principle — don't plan right up to the limit; always leave headroom for the system. Otherwise, a slightly longer conversation will trigger memory overflow or thrashing, and the experience falls apart completely.

Step 3: Quantization Is a Trade-off, Not a Free Lunch

Reducing numerical precision (quantization) makes models smaller and often speeds up inference. But many people treat it as a pure win — in reality, quantization's impact on quality is uneven.

Two quantized models of the same file size can perform very differently on your specific task. Some quantization levels are nearly lossless on mathematical reasoning but noticeably degrade long-text consistency; others show the opposite pattern.

A simple, actionable rule: compare at least two quantization levels using the same questions and the same settings. More importantly, save the outputs so you can compare them line by line for real differences, rather than deciding based on a vague feeling that it "seems about right." Impressions can deceive you; archived outputs won't.

Quantization works by replacing high-precision floating-point numbers with low-bit integers to represent model weights. Common precision levels range from FP16 (16-bit float) down to Q8, Q5, Q4, and even Q2 — lower bit depth means smaller files and faster inference, but also more information loss. The most popular format in local deployment today is GGUF (introduced by llama.cpp), which packages quantization information and model metadata together for easy cross-platform use. The same base model is typically offered in multiple quantized variants; for example, the letter suffixes in Q4_K_M and Q5_K_S denote different quantization strategies — the K series uses mixed precision, preserving higher precision for critical layers, which generally outperforms older uniform quantization schemes at the same file size. For most users, Q4_K_M is a reasonable starting point that balances quality and size — but "reasonable balance" is itself an average, and actual performance on your specific task still requires empirical testing.

Step 4: The Maximum Context Window Is Not the Usable Context

The context limit vendors advertise describes how much input the model can accept — it does not tell you whether the model can accurately use all of that input, nor whether all of it will actually fit in your memory.

An easily overlooked fact: the attention cache grows as conversations get longer, and can become the single largest consumer of memory, even after model weights are already loaded. That advertised 128K context may simply never be achievable on your hardware.

The recommendation is to start with the smallest context that solves the task. For long documents, compare two approaches: feeding the entire file in versus retrieving only relevant passages. Then measure systematically:

Memory budget: Record weights, cache, runtime, and system headroom separately.
Retrieval quality: Check the returned passages, overlap, and citation accuracy.
Degradation point: Place key facts at the beginning, middle, and end of the input to detect where attention starts to "drop out."
Latency: Measure prefill and generation time separately at realistic lengths.

Remember: a fluent-sounding answer does not prove the model correctly used all of the input.

The attention cache typically exists in the form of a KV Cache (key-value cache). In a Transformer architecture, when processing each token, the model saves the corresponding Key and Value vectors so that subsequent tokens can reuse them when computing attention, rather than recomputing the entire history. This means the longer the conversation, the more memory the KV Cache consumes — and the growth is linear or even super-linear. Take a 7B model with Q4 quantization as an example: the weights themselves occupy roughly 4GB, but at full 32K context, the KV Cache may consume an additional 2–4GB, directly eating into the space available for weights and the system. The prefill phase refers to the model processing the entire input prompt in one pass; the computation scales with input length, which is why the time-to-first-token (TTFT) can stretch significantly in long-document scenarios — directly conflicting with the latency expectations of interactive conversation.

Step 5: Check the License, Provenance, and Safety

Before integrating a weight file into a project, read the license for that specific model and its variant. The word "open source" carries a lot of ambiguity — having access to the weights doesn't necessarily give you the same rights to commercial use, redistribution, modification, or training derivative models. Keep a record: the repository URL, version number, and the license text you reviewed.

Go further: treat the model, prompt templates, and downloaded files as external dependencies. Verify their provenance, be cautious of file formats that can execute code, limit what the process can access, and don't connect tools with broad permissions during early testing. Running locally gives you stronger control over your data, but it does not eliminate the risks of generated content, nor the risks of supply chain attacks.

A minimal pre-installation record should include: file identifier and hash, license, author or repository, format, compatible runtime, download date, process permissions, and instructions for how to uninstall it.

Step 6: Run a Reproducible Test

Intuition is not evaluation. For a test to be truly trustworthy, it must be repeatable:

Freeze the test set: Use the same ~20 real tasks for every variant.
Record the environment: Note hardware, runtime version, context size, and parameters.
Score blind: Hide model names during scoring to avoid preconceived bias.

If you're using Ollama, you can inspect a model's details before running it:

ollama show model-name
ollama run model-name

For lower-level control and GGUF format support, llama.cpp is the standard reference project.

GGUF is the de facto standard weight format in the local deployment ecosystem, defined and maintained by the llama.cpp project. It stores the model architecture description, quantization parameters, vocabulary, and weight data in a single file, loadable without any additional configuration files. Ollama provides a higher-level abstraction on top of GGUF: it manages the llama.cpp runtime in the background, exposes model capabilities through a unified REST API, and handles model downloading, version management, and process lifecycle. The ollama show command lets you inspect a model's quantization level, context length, parameter count, and other metadata without actually running it — a practical way to quickly verify that a model's configuration matches your expectations before formal testing. If you need finer-grained control — such as adjusting KV Cache precision, pinning specific GPU layers, or testing different sampling strategies — using llama.cpp's command-line tools directly provides parameters that Ollama doesn't expose by default.

Putting It All Together

The best local model is the one that can solve your task within your memory budget, under a license you can accept, and that performs consistently in a test you can run again.

Parameter count and leaderboard rankings are inputs to this decision — not the decision itself. Rather than fixating on a model's lead of a few points on some public benchmark, just run your 20 real tasks and see what happens — the answer is often very different from what the leaderboard suggests.