Magnitude vs Ollama: Hardware-Tuned Kernels Cut Inference Latency in Half

The same Llama-3.1-8B model, a different inference engine, and throughput jumps from 18.2 tok/s to 38.7 tok/s on an M2 Max, and from 52.1 tok/s to 119.3 tok/s on an RTX 4090 — no overclocking involved. It's about handing kernel compilation to the runtime. Magnitude uses JIT to tune for your chip on the spot at first load, reclaiming the performance that Ollama's generic kernels leave on the table.

Ollama's Architectural Bottleneck: One Set of Kernels to Rule Them All

Ollama's convenience comes from precompiled GGML kernels built for generic x86/ARM configurations, with no distinction for a specific chip's SIMD width, cache hierarchy, or memory bandwidth characteristics. The M2 Max, the RTX 4090, and the AMD 7950X all run the same matrix multiplication routines.

The consequence: nominally similar hardware configurations produce wildly different time-to-first-token (TTFT) and tokens-per-second. It gets trickier with long contexts: past 8K, the static memory allocator can't adapt to the actual VRAM/RAM split, and KV-cache thrashing can drop throughput by roughly 40%. According to the original profiling data, at 8K context Ollama spends 22% of actual wall time in cudaMemcpy calls.

Magnitude's Approach: Tune on the Spot at First Run

Magnitude (GitHub: magnitudedev/magnitude) does the opposite: the first time it loads a model, it benchmarks matrix operations, attention mechanisms, and memory transfer patterns directly on your silicon, then compiles kernels that match that configuration.

Benchmark Setup

All tests used Llama-3.1-8B-Instruct (Q4_K_M quantization) across three machines: an M2 Max (32GB unified memory, macOS 14.5), an RTX 4090 (24GB VRAM, Ubuntu 22.04 + CUDA 12.4), and an AMD 7950X (64GB DDR5, pure CPU, AVX-512). The test script measured cold-start latency, TTFT at 2K/4K/8K context, and sustained throughput over 512 generated tokens, taking the median of 5 runs to filter out outliers.

Results: 1.8x to 2.3x Speedup

Hardware	Engine	TTFT (8K)	Throughput (tok/s)
M2 Max	Ollama	420ms	18.2
M2 Max	Magnitude	210ms	38.7
RTX 4090	Ollama	180ms	52.1
RTX 4090	Magnitude	85ms	119.3
7950X (CPU)	Ollama	1840ms	6.8
7950X (CPU)	Magnitude	950ms	12.4

The GPU gap is the most dramatic: Magnitude's CUDA kernel auto-tuning activates the 4090's Tensor Cores for INT4 operations, a path Ollama's generic route never reaches. On Apple Silicon, the Metal shader compiler leverages unified memory to eliminate the CPU-GPU data transfers Ollama still performs. On the CPU side, AVX-512 tuning delivers nearly double the performance too, showing this approach isn't only meaningful for big-VRAM GPUs.

JIT (Just-In-Time) compilation in an inference engine means something different from runtime JIT in languages like Java: here it refers to generating and compiling specialized kernels at first model load, tailored to the current GPU/CPU's instruction set, register count, cache size, and memory bandwidth, rather than using the general-purpose binaries shipped in a release. A typical implementation first runs a set of micro-benchmarks (e.g., matrix multiplications with different tile sizes), then uses an autotuner to search for the optimal parameter combination, ultimately producing a kernel cache valid only for that hardware configuration. This also explains why first load takes 45 to 90 seconds — that time is spent mainly on compilation and verification, not on loading model weights.

Tensor Cores are dedicated matrix computation units NVIDIA introduced starting with the Volta architecture, capable of executing mixed-precision matrix multiply-add at far higher throughput than ordinary CUDA cores. But to use the Tensor Core path, matrix dimensions must align to specific tile sizes and data types must match (e.g., INT4/INT8 require specific packing and dequantization flows). Generic GGML kernels, to preserve cross-architecture portability, often fall back to generic FP16 or even FP32 paths; Magnitude's autotuning tries the INT4 Tensor Core path and adopts it when precision is acceptable — a key reason for the doubled throughput on the RTX 4090.

Root Cause Analysis: Static vs Adaptive Memory Layout

The main battleground for the performance gap isn't compute — it's data movement and memory layout.

The original article's nvprof profiling shows that during KV-cache updates at 8K context, Ollama spends over a fifth of its time in cudaMemcpy; after Magnitude fuses cache writes with attention computation (kernel fusion), that overhead drops to 4%.

The strategy on the M2 is even more clever: after detecting the 100GB/s memory bandwidth ceiling, Magnitude automatically switches to a lower-precision accumulator, doubling throughput with no precision loss. Decisions like this must be built on measured hardware characteristics — generic kernels have no way to judge them.

KV-cache is the mechanism in Transformer inference that caches historical Key/Value tensors, avoiding recomputation of the entire context with each new token generated. Its size grows linearly with context length, often occupying several GB of VRAM at 8K context, so how the cache is allocated, how it's moved from CPU memory to GPU, and how it's read during attention computation directly determine latency. Ollama's static allocator carves out a fixed region once at startup; when the actual sequence length doesn't match the reservation, it triggers extra copies or reallocation. Kernel fusion, by contrast, merges KV-cache writes and attention computation into the same CUDA kernel, reducing round trips of intermediate results through VRAM — the main source of the drop in cudaMemcpy share from 22% to 4%.

Production Configuration Example

Magnitude offers a declarative configuration similar to a Modelfile:

FROM llama-3.1-8b-instruct-q4km.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 8192
PARAMETER magnitude.cache_kernels true
PARAMETER magnitude.tune_on_load true

First load takes 45 to 90 seconds for kernel tuning — a one-time cost; subsequent startups use cached kernels with cold-start times on par with Ollama. Setting num_ctx explicitly to 8192 is about backing up your latency SLA rather than relying on auto-detection.

When to Stick with Ollama

Magnitude assumes stable hardware. If you're running in a VM with fluctuating CPU quotas, or sharing GPU slices via Kubernetes MIG, the tuned kernels may misjudge actual resources and underperform Ollama's conservative but robust defaults.

Scenarios with frequent model switching are also a poor fit: testing 20-plus models an hour means accumulated tuning overhead eats up the per-request gains.

The original article's deployment advice is quite pragmatic: for production services with fixed hardware and stable traffic patterns (customer service bots, internal document Q&A), a 2x latency reduction translates directly into half the instances or double the request capacity; for exploratory research and model evaluation pipelines, Ollama's zero-config startup is still the faster path to your first answer.

The full benchmark scripts and tuning logs are at github.com/magnitudedev/magnitude. The real takeaway from this comparison: competition among inference engines has shifted from "how many models do you support" to "how well do you understand my hardware."

One-line Summary

Magnitude uses JIT to compile hardware-specific kernels on the spot, boosting Llama-3.1-8B inference throughput by up to 2.3x.

Paragraph Summary

This article compares the local inference engines Magnitude and Ollama on Llama-3.1-8B (Q4_K_M). Ollama relies on precompiled generic GGML kernels that ignore specific chip characteristics, causing 22% of time to be spent in cudaMemcpy at 8K context and KV-cache thrashing that can drop throughput by ~40%. Magnitude benchmarks the target hardware at first load and compiles specialized kernels, measuring throughput rising from 18.2 to 38.7 tok/s on M2 Max and from 52.1 to 119.3 tok/s on RTX 4090, with 8K TTFT dropping from 420ms to 210ms and from 180ms to 85ms respectively. The gains come mainly from kernel fusion and adaptive memory layout. The cost is a one-time 45-90 second tuning overhead; Ollama remains the safer choice when hardware fluctuates or models switch frequently.