oMLX
oMLX is a local large language model inference engine built on Apple's MLX framework, designed specifically for Mac devices equipped with Apple Silicon chips. It enables users to run large language models on local devices, fully utilizing the unified memory architecture and hardware acceleration of Apple Silicon. It supports inference acceleration techniques such as multi-token prediction (MTP) to improve local inference efficiency, and targets developers and users requiring private, offline large model inference on the Mac platform.
Timeline (last 90 days)
After oMLX 0.5.0, the 'omlx launch claude' command is provided, which can set ANTHROPIC_BASE_URL to point to the server, write the configured key into ANTHROPIC_AUTH_TOKEN (filling in 'omlx' when no key is set), clear ANTHROPIC_API_KEY to force going through the base URL, and set API_TIMEOUT_MS.
oMLX is an LLM inference service built on top of MLX, supporting continuous batching, two-tier memory+SSD KV caching, and compatibility with the OpenAI and Anthropic APIs.
The current version of oMLX is 0.6.4 and it uses the Apache 2.0 license.
oMLX can be installed via the .dmg from GitHub Releases, or via Homebrew (brew tap jundot/omlx and brew install jundot/omlx/omlx).
oMLX's admin interface is at http://localhost:8000/admin and requires an API Key; the default field is empty, and it is safe when the service listens only on 127.0.0.1, but if exposed to the LAN you need to set a random string in Settings → Auth & Info, and multiple keys are supported.
oMLX delivers a 3.28x speedup at 8x concurrency, with a single-request baseline of 72.4 tok/s.
On a 128 GB machine, when oMLX's memory limit is set to Auto, roughly 110-114 GB is left for inference.
On models with TurboQuant enabled, the community's actual runs generally set the Hot Cache Limit to 10% on 128 GB machines.
On oMLX 0.6.4, TurboQuant KV Cache cannot be stacked with VLM MTP, and in actual testing it does not bring generation speedup.
oMLX exposes an Anthropic-compatible API at http://127.0.0.1:8000, with routes under /v1 in the current version, including POST /v1/messages and POST /v1/messages/count_tokens.
10 more timeline events
All Facts (19)
Continue is hosted on GitHub and uses the Apache 2.0 license.
60%UnverifiedAfter oMLX 0.5.0, the 'omlx launch claude' command is provided, which can set ANTHROPIC_BASE_URL to point to the server, write the configured key into ANTHROPIC_AUTH_TOKEN (filling in 'omlx' when no key is set), clear ANTHROPIC_API_KEY to force going through the base URL, and set API_TIMEOUT_MS.
50%UnverifiedThe current version of oMLX is 0.6.4 and it uses the Apache 2.0 license.
50%UnverifiedoMLX can be installed via the .dmg from GitHub Releases, or via Homebrew (brew tap jundot/omlx and brew install jundot/omlx/omlx).
50%UnverifiedoMLX's admin interface is at http://localhost:8000/admin and requires an API Key; the default field is empty, and it is safe when the service listens only on 127.0.0.1, but if exposed to the LAN you need to set a random string in Settings → Auth & Info, and multiple keys are supported.
50%UnverifiedoMLX delivers a 3.28x speedup at 8x concurrency, with a single-request baseline of 72.4 tok/s.
50%UnverifiedOn a 128 GB machine, when oMLX's memory limit is set to Auto, roughly 110-114 GB is left for inference.
50%UnverifiedOn models with TurboQuant enabled, the community's actual runs generally set the Hot Cache Limit to 10% on 128 GB machines.
50%UnverifiedOn oMLX 0.6.4, TurboQuant KV Cache cannot be stacked with VLM MTP, and in actual testing it does not bring generation speedup.
50%UnverifiedoMLX exposes an Anthropic-compatible API at http://127.0.0.1:8000, with routes under /v1 in the current version, including POST /v1/messages and POST /v1/messages/count_tokens.
50%UnverifiedoMLX is an LLM inference service built on top of MLX, supporting continuous batching, two-tier memory+SSD KV caching, and compatibility with the OpenAI and Anthropic APIs.
50%Unverified在oMLX中启用JSON schema约束解码后，Qwen3.8-27B的30个回答中0个无法解析，但语法约束与MTP无法同时运行，请求退回普通解码，速度降至约25 tokens/秒。
50%UnverifiedoMLX 0.7.0的内存守卫会拒绝同时加载内存占用总和投影超过动态上限的模型（例：投影38.24GB超过34.51GB上限，返回507错误），并会在加载一个模型时驱逐空闲的另一个模型，导致正在进行请求丢失。
50%UnverifiedoMLX employs Continuous Batching technology, re-evaluating batch composition at every decoding step.
50%UnverifiedoMLX uses a two-tier RAM + SSD KV cache architecture that keeps active entries in RAM and offloads inactive ones to SSD, with the cache persisting across application restarts.
50%UnverifiedoMLX is a native Swift application, not built on Electron.
50%UnverifiedoMLX is an open-source tool that turns a Mac into a local LLM inference server, running persistently in the menu bar.
50%UnverifiedoMLX claims to compress AI agent wait times from approximately 90 seconds to around 5 seconds, a speedup of nearly 18x.
50%UnverifiedoMLX supports multiple model types including text, vision, OCR, embedding, and reranking.
50%