Local LLM Deployment in Practice: A Deep Comparison of Ollama, llama.cpp, and vLLM

Running large language models locally is no longer just a geek's toy. Today, running 7B to 70B parameter models on consumer-grade hardware delivers performance approaching cloud APIs—with no API keys, no cloud dependency, and data that stays entirely on your machine. This guide focuses on three mainstream inference engines—Ollama, llama.cpp, and vLLM—and breaks them down systematically from ease of use to throughput.

Positioning Differences Among the Three Engines

Although all three can run the same model, their design philosophies are completely different. Ollama pursues out-of-the-box usability—one command to pull a model and start a service—making it ideal for developers' daily chat and rapid validation. llama.cpp pursues low-level control—from quantization type to the number of GPU layers, everything can be tuned manually—making it the top choice for embedded and edge deployments. vLLM aims for production-grade throughput, relying on PagedAttention and continuous batching to maximize concurrency, making it suitable for scenarios serving many users simultaneously.

A detail often overlooked: all three expose OpenAI-compatible APIs. This means you can switch backends between them without changing frontend code, with near-zero migration cost.

Dimension	Ollama	llama.cpp	vLLM
Ease of use	Very high	Medium	Lower
Model format	GGUF	GGUF	HF/Safetensors
Batching	Not supported	Not supported	PagedAttention
Quantization	Automatic	Manual	AWQ/GPTQ
Best scenario	Local chat	Edge/embedded	Production serving

PagedAttention is vLLM's core technology, inspired by the virtual memory paging of operating systems. In traditional inference, each request's KV Cache must occupy a single contiguous block of VRAM, sized for the maximum possible sequence length, causing significant fragmentation and waste. PagedAttention divides the KV Cache into fixed-size blocks, allocated on demand and stored non-contiguously, then linked logically via a block table. This can raise VRAM utilization from the traditional 20%–40% to over 90%, dramatically increasing the number of concurrent requests a single card can hold. It's also the foundation of Continuous Batching—new requests can be inserted into a running batch at any moment rather than waiting for the entire batch to finish, which is precisely the order-of-magnitude source of vLLM's throughput advantage.

Hardware Thresholds and VRAM Estimation

The key to whether a model can run is VRAM. The industry rule of thumb is: VRAM ≈ parameter count × 0.5 (Q4 quantization) to ×1.0 (Q8 quantization), in GB.

Model	Quantization	VRAM/RAM	Pure CPU
7B	Q4_K_M	6 GB	Runs but slow
7B	Q8_0	8 GB	Usable
13B	Q4_K_M	10 GB	Runs but slow
33B	Q4_K_M	20 GB	Not feasible
70B	Q4_K_M	40 GB	Not feasible

Anything above 33B basically rules out pure CPU. A used RTX 3090 with 24GB VRAM costs about $800, covers Q4 models from 7B to 33B, and runs about $15/month in electricity. This is the sweet spot for many individual developers and small teams.

Ollama: The Most Hassle-Free Starting Point

Ollama's value lies in hiding all the complexity. Installation is a single line:

# Linux
curl -fsSL https://ollama.com/install.sh | sh
# macOS
brew install ollama
# Windows
winget install Ollama.Ollama

Pulling and running a model is likewise a single command, and the model is automatically downloaded in GGUF format:

ollama run mistral:7b-instruct-q4_K_M
ollama list
ollama show mistral:7b-instruct-q4_K_M

The service listens on port 11434 by default, and the OpenAI-compatible interface can be called directly with curl or Python requests. For scenarios requiring custom system prompts and parameters, you can define them with a Modelfile. For example, fix the temperature, top_p, context length, and system role, then package it into a local model with ollama create, avoiding re-passing parameters on every call.

Ollama's shortcomings are equally clear: it doesn't support batching. It's comfortable for single-request scenarios, but becomes a bottleneck as soon as concurrency picks up.

llama.cpp: Maximum Control

llama.cpp requires you to compile it yourself, but this buys you the greatest freedom:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1   # NVIDIA GPU

**GGUF and k-quant quantization**: GGUF is llama.cpp's flagship model file format, designed specifically for CPU/hybrid inference. It packs weights, vocabulary, and hyperparameters into a single file and supports memory mapping (mmap) to speed up loading and reduce memory pressure. The K in Q4_K_M refers to k-quant, a block-wise quantization scheme: weights are grouped into blocks, each represented with different bit widths and scale/minimum parameters, preserving higher precision for sensitive layers. The M suffix stands for medium, a quality-size compromise among the S (small), M, and L (large) tiers. Compared to the earlier Q4_0, Q4_K_M delivers better perplexity with fewer bits, making it the community's de facto standard. Choosing a quantization tier is essentially a trade-off among VRAM, speed, and output quality.

**Speculative Decoding** works on the principle of "small model drafts, large model verifies." During inference, a much smaller draft model first generates several candidate tokens in parallel in a single forward pass (say 4–8), then the main model verifies the entire segment at once. If the candidates are accepted by the main model, the per-token generation overhead is saved; rejected portions fall back and are regenerated. Since verification is parallel while generation is serial, as long as the draft model's acceptance rate is high enough, you can achieve 2–3x speedup overall, with an output distribution identical to using the main model alone—no quality loss. Draft models are typically smaller versions from the same family or specially trained distilled models. The cost is an extra chunk of VRAM, so on edge devices with tight VRAM you'll want to weigh whether to enable it.
# or make LLAMA_METAL=1  # Apple Silicon

It supports converting Safetensors models from Hugging Face into GGUF and manually selecting the quantization type. The quantization tier directly determines the balance between size and quality. Below, using a 7B model as an example:

Type	Size (7B)	Quality	Speed
Q2_K	~2.8 GB	Low	Fastest
Q3_K_M	~3.3 GB	Medium	Fast
Q4_K_M	~3.8 GB	Good	Balanced
Q5_K_M	~4.7 GB	Better	Slower
Q8_0	~7.2 GB	Near fp16	Slowest

Q4_K_M is the default answer for the vast majority of scenarios—it strikes the best balance of size, quality, and speed. When starting the service, you can specify the number of GPU offload layers, context length, and enable Flash Attention acceleration:

./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 -fa

It also has an advanced feature worth noting: Speculative Decoding. Pair a small draft model with a large model to significantly accelerate generation. Configure it by specifying both the main model and the -md draft model. This mechanism is already relatively mature in llama.cpp and is a practical means of boosting efficiency in edge scenarios.

vLLM: Production-Grade Throughput

vLLM's positioning is clear—born for concurrency. Installation requires CUDA 12.1+ and Python 3.10+, and also supports ROCm (AMD GPUs):

pip install vllm

When starting the service, --gpu-memory-utilization controls the VRAM usage ratio, and --max-model-len limits the context length. For multi-GPU scenarios, use --tensor-parallel-size for tensor parallelism, allowing a 70B model to be split across two cards. For quantization, AWQ and GPTQ are natively supported—just specify --quantization awq.

The performance gap is an order of magnitude. Under the same model (Llama-3.1-8B), batch size 32, and context 2048:

• vLLM (PagedAttention): ~450 tok/s
• llama.cpp (batch size 32): ~280 tok/s
• Ollama (no batching): ~120 tok/s

It should be noted that this data comes from a single source's test environment, and actual performance will vary with hardware and load. But the trend is stable: vLLM has a clear advantage under high concurrency, and Ollama was never positioned for throughput.

Performance Tuning Essentials

Each engine has its own tuning levers. For llama.cpp, set thread count to the number of physical cores, use -b and -ub to control batch size, and Flash Attention requires Hopper/Ada architecture or Apple Silicon support. vLLM boosts batching capacity via --max-num-batched-tokens and --max-num-seqs; --enable-chunked-prefill suits long contexts, while --enable-prefix-caching reuses shared prefixes, yielding notable gains for scenarios with fixed system prompts. Ollama is mainly controlled through environment variables, such as OLLAMA_NUM_PARALLEL for concurrency, OLLAMA_MAX_LOADED_MODELS for how many models stay resident in VRAM, and OLLAMA_FLASH_ATTENTION=1 to enable acceleration.

Batch size and concurrency are the key variables that differentiate the performance curves of the three engines. llama.cpp's -b (logical batch size) and -ub (physical micro-batch size) determine how many tokens are processed per forward pass—larger values improve GPU utilization but raise VRAM usage accordingly; vLLM's --max-num-seqs limits the number of simultaneously running sequences, and --max-num-batched-tokens limits the total tokens per batch—together they constrain the balance between VRAM and throughput. Two concepts must be distinguished: static batching processes a fixed set of requests at once and must wait for the slowest one to finish; continuous batching dynamically adds new requests, keeping the GPU busy almost constantly. Ollama has no true batching by default and can only mitigate this with OLLAMA_NUM_PARALLEL opening multiple parallel slots, but each slot still runs inference independently, so the gap versus vLLM widens further under high concurrency.

Common Troubleshooting

VRAM overflow is the most common problem. The fix path is to reduce GPU layers, switch to a smaller quantization, or offload some layers to CPU. When CPU inference is slow, first confirm you're using Q4_K_M and have BLAS enabled. Model loading failures are usually related to GGUF compatibility—try regenerating with llama.cpp's conversion script. If API connections are refused, check the port and firewall; Ollama requires setting OLLAMA_HOST=0.0.0.0 to be accessible externally. When output quality is poor, raise the temperature, switch to a larger model, or increase quantization precision.

The Cost Equation and Decision Path

The economics of local deployment need to be viewed over time. A used Ollama + RTX 3090 setup costs about $800 in hardware and about $15/month in electricity; llama.cpp + RTX 4090 is about $1,600 with $25/month in electricity; vLLM + dual A100 80GB is about $30,000 with about $500/month in hosting. Compared to cloud API usage of $50 to $500 per month, a local GPU's payback period is roughly 6 to 18 months. For teams with high data privacy requirements and stable usage, local deployment offers more obvious value; for scenarios with volatile usage, cloud APIs are more flexible.

The selection logic can be summed up in one sentence: choose Ollama if you just want to chat and test quickly; choose llama.cpp if you want to embed into applications, build edge devices, or customize quantization; choose vLLM if you need to serve many users and pursue high throughput; if you're on an Apple Silicon Mac, Ollama or llama.cpp's Metal backend are both suitable choices.

Conclusion

Local large models are already production-ready. The boundaries of the three engines are clear: Ollama wins on developer experience, llama.cpp wins on control and edge adaptability, and vLLM wins on serving scale. The fact that they share an OpenAI-compatible API means you can start with Ollama to validate ideas, then smoothly migrate to vLLM once concurrency pressure rises—without rewriting business code. Next, you can build on this to add RAG, function calling, or fine-tune your own specialized model with LoRA.

One-line Summary

A guide comparing the positioning, performance, cost, and selection of three local LLM inference engines.

Paragraph Summary

This article systematically compares three mainstream inference engines for local LLM deployment: Ollama, llama.cpp, and vLLM. Ollama excels at out-of-the-box usability, ideal for local chat and rapid validation; llama.cpp offers complete control from quantization type to GPU offload layers, the top choice for edge and embedded scenarios; vLLM achieves production-grade throughput via PagedAttention and continuous batching. The article provides VRAM estimation rules of thumb, installation and startup commands for each engine, a quantization tier selection table, performance tuning parameters, and common troubleshooting methods. In a high-concurrency test on the same model, vLLM reached ~450 tok/s, llama.cpp ~280 tok/s, and Ollama ~120 tok/s. All three provide OpenAI-compatible APIs, allowing zero-code switching, with a payback period of roughly 6 to 18 months.