Local AI Inference Engine Selection Guide: A Five-Layer Architecture Breakdown

Comparing Ollama, llama.cpp, vLLM, and LM Studio as four versions of the same tool is a fundamental category error. They are, respectively, an application, an API manager, a low-level execution runtime, and a datacenter service scheduler. If you're choosing an inference engine based on a "tokens per second" screenshot from social media, you're measuring the wrong bottleneck entirely.

This article draws on a deep-dive by Bilibili creator RepoChad to systematically break down the five layers of the local AI inference stack and offer selection recommendations for different use cases.

The Five Layers of the Local AI Inference Stack

To understand why local models run slowly, you first need to decompose the inference stack into five distinct layers:

• Layer 1 — Model Format & Quantization: e.g., GGUF, EXL3, FP8, or raw safetensors.
• Layer 2 — Execution Runtime & Compute Kernels: The low-level engine that actually performs matrix multiplication on hardware.
• Layer 3 — Serving Engine & Scheduler: Handles continuous batching, memory allocation, and KV cache paging under concurrent requests.
• Layer 4 — Local Daemon & Model Lifecycle Management: Handles weight downloading, endpoint configuration, and VRAM residency management.
• Layer 5 — Application Workbench: Provides a graphical interface, local document retrieval, and tool configuration.

The key insight: when someone claims "Tool A is better than Tool B," they're often comparing two completely different layers of the inference stack. Understanding this is a prerequisite for rational tool selection.

The independence of these five layers is critical. Layer 1 formats determine how weights are encoded and compressed — GGUF is the unified container format for the llama.cpp ecosystem, supporting multiple quantization precisions (Q4_K_M, Q8_0, etc.); EXL3 is an efficient quantization format native to the ExLlamaV2 engine; FP8 is a low-precision floating-point standard targeting datacenter GPUs. Quantization is fundamentally a precision-for-memory trade-off: a 70B parameter model stored in FP16 requires roughly 140GB, while quantizing to 4-bit compresses it to around 35GB — at the cost of varying degrees of accuracy loss. Layer 2 compute kernels determine actual throughput on specific hardware — the same model format can run several times faster or slower depending on the kernel implementation. That's why "tokens per second" screenshots are meaningless without specifying hardware environment, quantization level, and runtime version.

llama.cpp: The Baseline Runtime for Low-Level Control

llama.cpp is an MIT-licensed C/C++ reference runtime centered on the GGUF format. Its primary strengths are portability and fine-grained hardware control: it runs on Apple Silicon via Metal, modern x86 processors, and GPUs running CUDA, ROCm, or Vulkan.

More importantly, llama.cpp is the standard for partial offloading. If you only have an RTX 3060 12GB, you can load 20 layers of a model into VRAM and let the remaining layers overflow to system RAM. Its server binary now also supports continuous batching, speculative decoding, and OpenAI-compatible endpoints.

That said, its continuous batching is designed for light loads — suited for single-user heavy workloads rather than high-concurrency multi-user saturation scenarios, and paged KV cache implementation is still under active development. If you want maximum low-level control over thread allocation and tensor splitting on consumer hardware, llama.cpp is your baseline.

Ollama: A Clean and Efficient Model Lifecycle Manager

A common misconception in developer forums is that Ollama is just a wrapper around llama.cpp. That description is long outdated. While it started there and retains a GGUF-compatible runner, Ollama has since built its own first-party multimodal engine for vision models and runs natively via the MLX path on Apple Silicon.

Ollama is fundamentally a model lifecycle manager, offering a clean CLI, hosted manifests, and an API listening on port 11434. In current versions, its default context window automatically scales with hardware:

• VRAM below 24GB: defaults to 4,000 tokens
• VRAM 24–48GB: expands to 32,000 tokens
• VRAM above 48GB: allocates up to 256,000 tokens

Ollama's trade-off is reduced low-level visibility — you don't get the direct parameter knobs that llama.cpp offers, and memory usage for parallel requests scales linearly with concurrency multiplied by active context length.

LM Studio: A Feature-Rich Desktop Workbench

LM Studio operates at Layer 5 — it's a desktop workbench with a visual interface, featuring built-in model search, local document retrieval, MCP (Model Context Protocol), and a headless daemon called LMStudio.

Under the hood, LM Studio uses a hot-swappable runtime — bundling llama.cpp for GGUF execution and MLX for Apple Silicon. As a result, inference speed on consumer hardware largely tracks the underlying runtime's performance, and the parallel request engine defaults to 4-way prediction on llama.cpp.

The most critical distinction here is the license: while the bundled runtimes are open source, the LM Studio desktop application itself is proprietary software. It permits local development and internal business use, but the terms restrict redistribution and SaaS usage.

vLLM and SGLang: The Two Pillars of Production Serving

vLLM sits at the other end of the spectrum — an Apache 2.0 production-grade serving engine built to maximize throughput across multiple concurrent streams. Its core innovation is PagedAttention, which manages the KV cache like virtual memory pages in an operating system, eliminating physical memory fragmentation. It supports chunked prefill, prefix caching, tensor parallelism, and multi-node GPU clusters.

It's worth emphasizing: vLLM is not designed for desktop offloading. While you can technically run GGUF models via external plugins, the official documentation treats GGUF support as "experimental and unoptimized." Testing vLLM on an RTX 3070 with GGUF for single-user workloads will be significantly slower than llama.cpp — you're bypassing its optimized paths while still paying the scheduler's memory overhead.

In production, vLLM faces direct competition from SGLang. SGLang uses RadixAttention, maintaining a radix tree of KV caches across requests to enable automatic prefix caching — making it especially fast for multi-turn agent conversations, complex tool-calling loops, and structured JSON generation. On enterprise Linux clusters with modern NVIDIA or AMD accelerators, SGLang and vLLM together define the true production frontier.

Additionally, on NVIDIA hardware where you can pin driver and CUDA versions, TensorRT-LLM offers custom kernel optimizations and in-flight batching — but it requires a rigid build matrix and lacks the plug-and-play experience of open-source frameworks.

The core insight behind PagedAttention comes from OS virtual memory design. Traditional inference frameworks pre-allocate contiguous maximum-length VRAM blocks for KV caches when requests arrive — even when actual sequences are far shorter, this memory can't be reused by other requests, causing severe internal fragmentation. PagedAttention splits the KV cache into fixed-size non-contiguous memory pages, dynamically allocated by the scheduler, theoretically raising VRAM utilization from under 40% to over 90% — directly amplifying the number of concurrent users serveable on the same hardware. SGLang's RadixAttention goes further: it reuses KV caches for shared prefixes (e.g., system prompts, few-shot examples) across requests via a radix tree. When multiple users share the same system prompt, that portion is computed only once and cached permanently, significantly reducing time-to-first-token. These technologies offer almost no perceptible advantage in single-user local testing, but the difference is dramatically amplified under tens to thousands of concurrent requests in production.

Edge Inference in the MoE Era: FreeToken and Colibri

The most significant recent architectural advances are concentrated on running massive-scale MoE (Mixture of Experts) models on consumer hardware. When a MoE model has hundreds of billions of total parameters, stuffing the entire weight matrix into consumer VRAM is simply impossible.

FreeToken is an Apache 2.0-licensed edge-native MoE serving engine. Its design keeps non-expert weights on the GPU, stores the full expert weight pool in system host RAM, and uses remaining VRAM as a dynamic LRU expert cache. When an expert cache miss occurs, FreeToken measures PCIe bus bandwidth in real time and splits computation — routing some missing expert traffic over the bus to the GPU while executing the rest directly on the CPU.

According to its technical paper, on an RTX 5090 system, Qwen3-235B-A3B achieves 77–83 tokens/s, and DeepSeek-related models reach 22–25 tokens/s on agent trajectories; even running a 35B parameter model on an 8GB RTX 4060 laptop shows respectable performance. However, the current CLI requires Linux x86_64, an NVIDIA GPU, and CUDA 13, and the supported model list is fairly limited.

Another experimental project, Colibri, is written in pure C with zero runtime dependencies and introduces a three-tier memory model: hot experts reside in VRAM, warm experts in system RAM, and cold experts are streamed directly from NVMe SSD. It can load ultra-large models like GLM with only 25GB of RAM — but you must distinguish between physical capacity and interaction latency. Once a prompt routes through a cache miss, time-to-first-token and decode speed drop to the physical limits of the storage read queue. This is excellent memory tiering engineering, but it's decidedly not "zero-latency magic."

The key property of MoE (Mixture of Experts) architecture is sparse activation: the model has a large number of parallel "expert" feed-forward networks, but only a few are selected and activated by the router on each forward pass. Take DeepSeek-V3 as an example — it has roughly 671B total parameters, but only about 37B are activated per token. This means MoE models have far less "compute" than dense models of equivalent parameter count, but still require loading all expert weights in terms of "memory footprint." The edge inference strategies of FreeToken and Colibri fundamentally exploit MoE sparsity — since only a few experts are used at a time, cold experts can be placed in slower storage and hot experts swapped in on demand. This strategy is completely ineffective for dense models (like LLaMA architectures), where all weights across every layer must participate in the computation of every token.

Inference Engine Selection Recommendations by Use Case

Taken together, the selection logic is actually quite clear:

• Want a desktop experience with a clean GUI → Use LM Studio
• Need a local API daemon that integrates with dev tools and manages models like packages → Use Ollama
• Want full control over GGUF quantization on mixed CPU/GPU consumer hardware → Use llama.cpp
• Building automated systems serving concurrent users, running on dedicated accelerators → Deploy SGLang or vLLM
• Experimenting with massive sparse MoE architectures on modern consumer NVIDIA cards → Watch FreeToken and Colibri closely

Ultimately, there's no single "best" local AI inference engine — only the question of which layer best matches your use case. Understanding the five-layer architecture is far more valuable than chasing any benchmark screenshot.