A Practical Guide to Deploying oMLX Local Inference on M5 Max 128GB

After Apple introduced the MLX framework for Apple Silicon in late 2023, the ecosystem around it has been expanding rapidly. oMLX is one of the most aggressively iterated products in this space: an LLM inference server built on top of MLX that fills in continuous batching, two-tier KV caching across RAM and SSD, and APIs compatible with both OpenAI and Anthropic.

For an M5 Max with 128 GB of unified memory, the point of this combination is that you can mount three to four large models simultaneously, compress the KV cache with TurboQuant 3.5-bit, and feed chat, agent, and IDE requests in parallel to a single machine. This article consolidates benchmark data from multiple sources and presents a validated configuration.

What oMLX Is and Why It's Worth the Effort

The core value of oMLX isn't that it's "yet another local inference tool" — it's that it fills the service-ification gap that has long existed in the Apple ecosystem. The MLX framework itself only solves the problem of running models on Apple Silicon, not the engineering problems of multi-model coexistence, request queuing, or moving the KV cache across memory tiers. oMLX fills in that layer.

The current version is 0.6.4 (Apache 2.0 license), and there are two installation paths: download the .dmg from GitHub Releases and drag it into Applications, after which the server starts as a background process with a menu bar icon; or go through Homebrew with brew tap jundot/omlx followed by brew install jundot/omlx/omlx, then bring up the service with omlx start and omlx serve --model-dir ~/models.

Opening http://localhost:8000/admin for the first time will ask for an API Key. By default this field is empty, which is fine as long as the server only listens on 127.0.0.1. Once you plan to expose it on the LAN or share it with others, you must go to Settings → Auth & Info and set a sufficiently long random string. This section supports configuring multiple keys at once.

Continuous batching is the core mechanism that distinguishes modern LLM inference servers from naive per-request inference. Traditional batching requires waiting for an entire batch of requests to finish generating before accepting the next batch, which causes short requests to be held up by long ones and leaves the GPU/ANE idling. Continuous batching checks which sequences have finished immediately after each decode step and dynamically inserts newly arrived requests into the freed slots, keeping each decode step's batch full. It's usually paired with prefix cache reuse and paged KV management (such as PagedAttention). oMLX supports it, which means that when Claude Code, a chat session, and an agent loop run concurrently, throughput isn't simply queued linearly — in benchmarks, 8x concurrency delivers a 3.28x speedup, far above the single-request baseline of 72.4 tok/s.

How to Tune Server Parameters on a 128 GB Machine

Settings → Global Settings is the entry point for all server configuration. On a 128 GB machine, only about a dozen decisions actually affect the experience.

For the network layer, keep Host as Localhost only (127.0.0.1) and Port at the default 8000.

Memory management is the key part. Set both Memory Limit (Total) and Memory Limit (Models Only) to Auto; oMLX subtracts what macOS reserves for itself, leaving roughly 110–114 GB for inference on a 128 GB machine. Set Hot Cache Limit to 10% — this is the RAM-tier staging KV cache; on models with TurboQuant enabled, community practice on 128 GB machines generally lands at 10%. Cold Cache Limit (SSD Cache) is also 10%, roughly 80 GB of SSD for cold tokens, which avoids filling the disk.

For concurrency and keep-alive, set Max Concurrent Requests to 16, which is ample for the single-user scenario of "an agent plus a chat session plus an IDE typing at the same time"; small teams sharing can push it to 32. Set Idle Timeout to None so models stay resident and hot, and the first token returns in well under a second rather than several seconds.

For generation defaults, set Max Context Window globally to 256000. The Qwen3.6 family is stable at long context, and with TurboQuant the actual memory holds up fine. Individual models can be lowered separately in Model Settings. Set Max Tokens to 64000, and Temperature to 1.0 for general use and 0.2–0.5 for code models.

The last item is Model Settings → Experimental Features → TurboQuant KV Cache, set to 3.5-bit on large dense models. vLLM's conclusion, validated in independent research, is that 3.5-bit matches full precision in quality while using about a quarter of the KV cache memory. On the M5 Max, this determines whether a 128k context "blows up memory" or "runs, and leaves room for other models too." Note that on 0.6.4 it cannot be stacked with VLM MTP, and in practice it does not speed up generation.

The KV cache (Key/Value Cache) is the intermediate state preserved during autoregressive Transformer inference: for each token generated, the model needs to look back at the attention Keys and Values of all previous tokens, and without caching this would require recomputation at a cost of O(n²). KV cache memory usage grows linearly with context length and layer count, and in long-context scenarios often exceeds the model weights themselves. The point of a two-tier cache (a hot RAM tier plus a cold SSD tier) is this: the hot tier keeps the most recently and frequently accessed sequence prefixes, while the cold tier persists evicted tokens, trading latency for capacity. This is precisely the engineering logic behind the "10% Hot Cache + 10% SSD Cold Cache" configuration — out of 110 GB of memory, the margin left beyond the weights is finely divided into these two blocks.

A Multi-Model Coexistence Selection List

128 GB is enough to simultaneously mount a large primary model, a fast assistant, a vision model, an embedding model, and a reranker. Download them all by pasting Hugging Face repository addresses into Models → Downloader.

According to feedback from users running Claude Code on oMLX (Diego R. Baquero's gist is the source of this configuration set), Unsloth's Qwen 3.6 35B-A3B series MoE (about 3B active parameters per token) is currently the most stable choice. The 8-bit version runs on 128 GB with no pressure at all:

• Primary (chat + code + reasoning): unsloth/Qwen3.6-35B-A3B-MLX-8bit, about 37.7 GB, an all-rounder at MoE 35B / 3B active that coexists with other models without squeezing them
• A more compressed alternative: unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit, about 21.6 GB, at the cost of a quality tier, but you can mount two primaries at once
• Dense reasoning: Mistral-Large-2-123B-Instruct-mlx-4bit (about 70 GB) or Llama-3.3-70B-Instruct-mlx-4bit (about 40 GB), stepping in when you need a single deep reasoning pass and a small-activation MoE isn't enough
• Fast assistant: Qwen3-14B-Instruct-mlx-4bit, about 8 GB, for cheap tasks, parsing, and summarization in agents
• Vision: Qwen2.5-VL-32B-Instruct-mlx-4bit, about 18 GB, covering OCR, image captioning, and multimodal reasoning
• Embedding: BGE-M3-mlx, about 1.2 GB, a three-in-one of dense + sparse + multi-vector
• Reranking: ModernBERT-base-mlx, about 150 MB, for closing the loop on a decent RAG pipeline

With the 8-bit Qwen3.6 35B-A3B as the primary, plus the 14B assistant, VL-32B, embedding, and reranker, comfortable concurrent usage is around 65–70 GB. Stacking on a Mistral Large 2 123B for dense reasoning brings the nominal figure to about 135 GB. The LRU policy will move idle models to SSD, so as long as not all models are being called at high frequency simultaneously, this coexistence is feasible. It's recommended to pin the primary model from the Model Manager to prevent it from being swapped out.

Connecting Claude Code to a Local Endpoint

oMLX exposes an Anthropic-compatible API at http://127.0.0.1:8000, with current version routes under /v1: POST /v1/messages and POST /v1/messages/count_tokens. Claude Code recognizes ANTHROPIC_BASE_URL, so pointing the CLI at your local machine is just a matter of exporting environment variables.

After 0.5.0 there's a shortcut that handles everything with one command: omlx launch claude. It sets ANTHROPIC_BASE_URL to point at your server, writes the configured key into ANTHROPIC_AUTH_TOKEN (filling in omlx when no key is set), clears ANTHROPIC_API_KEY to force the base URL, raises API_TIMEOUT_MS to 3,000,000, and disables non-essential traffic. It also sets CLAUDE_CODE_MAX_CONTEXT_TOKENS and CLAUDE_CODE_AUTO_COMPACT_WINDOW according to the engine's actual capabilities, and enforces a minimum window of 48,000 tokens, refusing to start if the model can't reach it. Codex, OpenCode, OpenClaw^Ad, Hermes, and Pi all have corresponding subcommands.

The manual approach still works and suits scenarios requiring fine control over each variable. First turn off the attribution header in ~/.claude/settings.json: {"env": {"CLAUDE_CODE_ATTRIBUTION_HEADER": "0"}}. Then export this set of variables in your startup script: ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, the three ANTHROPIC_DEFAULT_OPUS_MODEL / SONNET / HAIKU_MODEL, plus ANTHROPIC_DEFAULT_MODEL, API_TIMEOUT_MS=600000, CLAUDE_CODE_USE_BEDROCK=0, DISABLE_NONESSENTIAL_TRAFFIC=1, and finally launch with claude --bare.

--bare skips hooks, LSP, plugin sync, and auto-memory, compressing Claude Code's system prompt to about 1,795 tokens (thousands when fully loaded). This is a reasonable default when working offline against a local model: every token in the system prompt is bandwidth your Mac is burning for nothing. Drop the flag when switching back to api.anthropic.com.

One limitation that needs to be stated clearly: Claude Code is tuned for Claude's own tool-calling format and response patterns. Putting a non-Claude model behind ANTHROPIC_BASE_URL works for autocomplete and reasoning, but tool-call reliability and agent loops degrade noticeably. It's suitable for offline work, keeping sensitive data off the network, or serving as a fallback when api.anthropic.com is rate-limited — not as a one-to-one replacement for Claude Opus 4.7.

Cross-Machine Use: Connecting from a MacBook to a Mac Studio

If oMLX runs on a Mac Studio and you want to use Claude Code from a MacBook, SSH port forwarding is the bridge: ssh -L 8000:localhost:8000 user@mac-studio. Once connected, 127.0.0.1:8000 on your laptop points at the Studio's oMLX.

Save the variable-exporting portion as claude-local.sh, chmod +x it, and run it inside the SSH session. The default API key is an empty string; if you set one in Settings → Auth & Info, substitute it. You can add --dangerously-skip-permissions to the script to skip interactive permission confirmations that are meaningless in automated pipelines.

Benchmarks: The Gap Between Older and Newer Versions

First, a set of reference numbers: M5 Max 40-core / 128 GB, oMLX 0.3.8, measured in May 2026. Qwen 3.6 35B-A3B 8-bit decodes at 65–80 tok/s in short context, the model on this machine that most changes daily experience. pp1024/tg128 gives a 577 ms first token, 72.4 tok/s decode, 38.5 GB peak memory; pp4096/tg128 decodes at 83.1 tok/s; by pp32768/tg128 it drops to 15.2 tok/s. Under continuous batching, the 1x baseline is 72.4 tok/s, while 8x concurrency reaches 237.7 tok/s, a 3.28x speedup.

Llama 3.3 70B 4-bit decodes at 14–18 tok/s. Mistral Large 2 123B 4-bit with TurboQuant 3.5-bit KV cache decodes at 8–11 tok/s, and the truly critical figure is peak memory of about 74 GB at 128k context — FP16 KV would blow straight past 128 GB. dasroot.net's long-context experiments with a 104B dense model recorded the same peak.

vLLM measured a TurboQuant KV compression ratio of 4.41x in May 2026. This number determines whether you "have 128k context but can't fit it" or "can fit it and still mount two more models." At 4-bit and 3.5-bit, quality is close to full precision; at 3-bit you start to feel it in code and long-text reasoning.

Advanced Tuning on 0.6.4: Speculative Decoding

The biggest speed lever on 0.6.4 is speculative decoding, and which method to choose depends on the model. Measured through the API at temperature 0.6: Qwen3.8-27B oQ4e with Lightning MTP goes from 21.1 to 39.7 tok/s, and Qwen3.6-35B-A3B 8-bit with VLM MTP goes from 79.7 to 117.5 tok/s. Overall, coding scenarios speed up 1.19x to 1.88x, and at 15,000-token context it's 1.07x to 1.47x.

oMLX 0.6.4 offers three methods, and only one can be chosen per model:

Lightning MTP uses the multi-token prediction head built into the weights, and is only available if the mtp.* tensors were preserved during conversion. VLM MTP uses an external drafter under 2 GB and only takes effect when no other requests are running on that model. DFlash uses z-lab's block-diffusion drafter, proposing up to 16 tokens at a time, serving requests serially and maintaining its own prefix cache. With all three methods, the large model verifies each proposed token, so the output comes from the large model rather than the drafter.

Measured combinations per model: Qwen3.6-35B-A3B 8-bit with mlx-community/Qwen3.6-35B-A3B-MTP-bf16 (1.7 GB) using VLM MTP goes from 79.7 → 117.5 tok/s in coding and 84.0 → 90.7 tok/s at 15k context; Qwen3.8-27B oQ4e with Jundot/Qwen3.8-27B-oQ4e-mtp (17.0 GB) using Lightning MTP goes from 21.1 → 39.7 and 21.4 → 27.2; Gemma 4 12B 8-bit with mlx-community/gemma-4-12B-it-assistant-bf16 (0.88 GB) using VLM MTP goes from 27.4 → 50.5 and 26.6 → 39.2.

If the Qwen model's settings page shows the warning Config declares MTP layers but the weight files contain neither mtp. tensors nor native nextn layers*, it means the MTP head was stripped during conversion. mlx-lm's default converter does exactly this, and both mlx-community's Qwen3.6-35B-A3B and lmstudio-community's Qwen3.8-27B are affected. Two ways out: download a build that preserves the MTP head (such as the oQ4e-mtp series published by the oMLX author), or use VLM MTP with an external drafter.

DFlash only pays off with short prompts. In the built-in benchmarks, Qwen3.6-35B-A3B 8-bit goes from 92 to 112 tok/s at 1k prompts, but drops from 78 to 17 tok/s at 32k prompts. Qwen3.8-27B 4-bit drops from 23.5 to 6.3 at 32k. If you use Claude Code (which sends tens of thousands of tokens of context each turn), turn it off.

Qwen ANE Prompt Processing (offloading part of prefill to the Apple Neural Engine) shows inconsistent gains in practice. On Qwen3.8-27B oQ4e, prefill for an 8,193-token prompt goes from 452 to 510 tok/s, a 12.8% gain. But paired with Lightning MTP the gain disappears: prefill at 4k is 505 vs 499, and at 32k it's 422 vs 471 (actually slower). The cost is that model loading goes from 16.0 GB to 20.7 GB, and load time from 2 seconds to 20 seconds. It's recommended to turn this off on this machine.

TurboQuant KV does not speed up generation. On Qwen3.8-27B oQ4e with Lightning MTP plus 4-bit TurboQuant, peak memory at 32k prompt is unchanged (24.4 GiB), while decode actually drops from 35.4 to 30.3 tok/s. In these hybrid models only the full-attention layers retain a KV cache, so there isn't much to compress in the first place. It saves memory, not time.

Speculative decoding works like this: a cheap small drafter model pre-"guesses" the next several tokens, then the large model verifies this batch of candidates in parallel in one pass. Accepted tokens are equivalent to tokens the large model generated directly, and at the first rejected position you roll back and start over from there. Because verifying k tokens with the large model takes far less time than generating k tokens serially, this speeds things up as long as the acceptance rate is high enough. MTP (Multi-Token Prediction) heads are prediction modules built into the weights during training, so the drafter is part of the model itself; external drafters like DFlash are separate small models that need extra loading. The key constraint is that all candidate tokens ultimately go through verification by the large model, so output quality is unaffected by the drafter — only speed is.

Measured Numbers on 0.6.4

With the configuration above, this M5 Max generates 1.19 to 1.88 times the tokens per second of an untuned setup in coding scenarios. With a 15,000-token context it's 1.07 to 1.47 times. With no acceleration at all, Qwen3.6-35B-A3B 8-bit sustains 77.9 tok/s at 32k prompt, whereas on 0.3.8 it was only 15.2 tok/s.

That gap is too large to be a measurement artifact. Same machine, same model family, with only the version going from 0.3.8 to 0.6.4, 32k long-context decode goes from 15.2 to 77.9 tok/s, 1k prompt from 72.4 to 92.1, and first token from 577 ms down to 298 ms.

There are two test methodologies. Through the API, streaming requests go to /v1/chat/completions, temperature 0.6, reasoning disabled. In the short test each configuration answers three coding tasks (Python, TypeScript, Go) twice each, with 768-token answers; in the long test it answers two questions about roughly 15,100 tokens of source code, with 512-token answers. Timing is done client-side. The built-in benchmark is started from POST /admin/api/bench/start, with each configuration repeated three times and the median taken, prompt lengths of 1,025 / 4,097 / 16,385 / 32,769, generating 256 tokens, corpus code_python, always greedy sampling.

Every request carries a unique prefix, so none of them cheated via prefix caching. The tests ran continuously for about three hours with other applications open as usual. The built-in benchmark recorded macOS thermal pressure levels, with 194 of 242 measurements at level three of five, Heavy. Only the first batch of gpt-oss ran on a cold machine, and its 16k prefill dropped from 3,140 tok/s cold to 1,755 tok/s hot. So the absolute numbers should be read as "hot Mac" numbers, and the relative gains before and after tuning are more reliable than the absolute values.

What Changed from 0.3.8 to 0.6.4

0.5.0 (July 10, 2026) added native depth-k speculative decoding for Qwen3.6-27B, Qwen3.6-35B-A3B, DeepSeek^Ad-V4-Flash, and GLM-5.2. The project measured on M3 Ultra that Qwen3.6-35B-A3B goes from 89.6 to 140.4 tok/s (1.57x), and Qwen3.6-27B from 35.0 to 55.1 tok/s.

The same version also added custom compute kernels for DeepSeek V4, Qwen3.5/3.6, and GLM-5.2, with prefill gains measured on M3 Ultra of 45% (DeepSeek-V4-Flash at 64k context, 313.3 → 455.8 tok/s), 33% (Qwen3.6-27B), and 99% (GLM-5.2 at 32k). For the M5 series, what matters is that scheduling is NAX-aware, specifically to avoid regressions on M5, so it's worth verifying by measurement rather than assuming.

oMLX also no longer relies solely on the weights you download: it brings its own quantization scheme. 0.5.0 added oQe, a pipeline with activation importance calibration and MoE expert coverage tracking. By the project's measurements, oQ4e has higher average accuracy than oQ4 at the same on-disk size — 83.88% vs 82.83% on Qwen3.6-35B-A3B, and 73.09% vs 70.01% on Qwen3.5-9B.

The 0.6.3 branch optimized the prefill path on the Neural Engine. Bank compiled memory on this front dropped from 35.8 GB to 4.7 GB, and an optional persistent compilation cache cut new process startup time by 53% to 66%. 0.6.4 pushed further, improving Qwen3.8-Flash-Next prompt processing by 33.5% and reducing total request time by 24.2% at 32k (again M3 Ultra data).

On the API side, /v1/rerank, /v1/responses, and /v1/mcp/tools were added, so by mounting an embedding model and a reranker together, the whole retrieval pipeline can run on the same server. Authentication also changed from "a bunch of equivalent keys" to three permission levels: the master key can call the API and access the dashboard; sub-keys can only call the API and can't access the dashboard or change settings; and dashboard session tokens live in signed cookies. To distribute access to multiple applications, issue sub-keys.

There's also Profiles (saving named setting bundles around a single model, exposed as model:profile, occupying no extra memory), Aliases (renaming models at the API level), and experimental distributed inference (splitting model layers across multiple Macs according to each machine's memory, over SSH, taking into account whether the link is Thunderbolt or 10GbE).

A Few Common Pitfalls

The default hot_cache_max_size in the config file is "0", which effectively turns off the RAM hot cache. The memory limit key is now named memory_guard_tier (four levels: safe / balanced / aggressive / custom), and custom values go in memory_guard_custom_ceiling_gb, with the default ceiling being system memory minus 8 GB.

The kernel memory limit needs to be adjusted manually. macOS limits the memory the GPU can lock by default, and the dashboard shows a red warning when it's below what oMLX can use. Run sudo sysctl iogpu.wired_limit_mb=124518 (on a 128 GB machine, memory minus 5%) and apply it when restarting oMLX. This value is lost after a Mac restart.

Clicking "Save Settings" unloads all loaded models, even if you never touched a cache-related field, because the dashboard re-sends every field.

Set Prefill Priority to Speed. When memory is tight, Max Context throttles prefill to 32 tokens per step to fit the prompt, while Speed keeps full speed and rejects prompts that don't fit. Turn on Hide Helper Models to keep drafters out of /v1/models where clients might mistakenly select them. Keep Max Concurrent Requests between 8 and 16, and don't drop it to 1 — the scheduler won't accept new requests once the number of pending cache writes reaches this limit.

Keep Burst Decode at Balanced. Aggressive accumulates tokens for up to 200 ms before sending (versus 100 ms), so clients see the first token later.

iogpu.wired_limit_mb is the kernel parameter macOS uses to limit the amount of lockable physical memory for the GPU/ANE. The default is conservative, to prevent a single process from monopolizing memory and destabilizing the system, but for local LLM inference — which genuinely needs a large amount of resident VRAM — it becomes a bottleneck instead. Raising it is equivalent to telling macOS "I'm willing to take that risk." The sysctl setting is a runtime parameter and isn't written to persistent config, so you must re-run it after every Mac restart, or wrap it in a startup script. Setting it to memory minus 5% (about 124518 MB) on a 128 GB machine is the community rule of thumb: it leaves enough margin for the system while giving oMLX enough room to lock all weights and the KV cache in resident memory, avoiding swaps to SSD that would slow down decoding.

Sources

• Diego R. Baquero's gist "Running Claude Code with a local LLM": the source of the base recipe for this article's setup (TurboQuant 3.5-bit, 64k context, attribution header off, Qwen 3.6 35B-A3B family)
• vLLM's blog on the TurboQuant comprehensive analysis: independent validation of the conclusion that 3.5-bit matches full precision in quality
• omlx.ai/benchmarks: a public leaderboard with real submissions on M5 Max
• dasroot.net's M5 Max long-context and memory fragmentation experiments: another detailed experiment on M5 Max + TurboQuant long context

Originally published on jacar.es; benchmark numbers are updated with each oMLX release.