MLX-serve Achieves Million-Token Local Inference

An Inference Engine Built for Real Long-Context Workloads

A developer posted on Reddit about adding Qwen3.8-Flash-Next model support to MLX-serve, successfully running a 1 million (1M) token context window locally on an Apple M5 Max with 128GB of memory. What makes this achievement stand out is its focus on real deep-context workloads rather than simple performance benchmarks.

What a Context Window This Large Actually Means

The context window refers to the maximum number of tokens a model can process at once. One token corresponds to roughly 0.75 English words or 0.5 Chinese characters, so 1M tokens equates to approximately 750,000 English words or 500,000 Chinese characters — the equivalent of several full-length novels or an entire mid-sized codebase. In real-world scenarios, long-context capability is critical: analyzing a complete code repository can require hundreds of thousands of tokens, and tasks like legal document review, academic literature surveys, and long-form creative writing all depend on deep context. However, industry benchmarks often use short contexts (2k–8k tokens) and low temperature sampling (temperature < 0.5, producing more deterministic output) to showcase speed, which fundamentally diverges from production scenarios that require creative output (temperature = 1.0, high randomness) over long contexts.

The developer explicitly stated that this engine isn't designed to show off impressive tok/s numbers on ultra-short contexts. Instead, it's built to reliably handle deep-context tasks spanning a million tokens under temperature=1.0 sampling conditions. This stands in stark contrast to most local inference benchmarks, which are typically run with short contexts and low temperature sampling — a significant gap from real production use cases.

Core Performance Results

Based on the developer's demo video (approximately 760k context) and measured data, the performance is solid:

• Generation speed: Prose/text generation tasks sustain approximately 40 tok/s, coding tasks reach approximately 75 tok/s, and this speed holds throughout the entire 1M context without significant degradation
• Quantization strategy: Mixed-precision quantization with 8-bit for dense layers, 4-bit for expert layers, and 8-bit quantization for the KV cache
• Memory usage: Peak memory at full 1M context load is approximately 117GB, requiring the system parameter iogpu.wired_limit_mb to be set to 120000

Quantization Technical Details

Quantization is a technique that converts model weights from high precision (e.g., 16-bit floating point) to lower precision (e.g., 8-bit or 4-bit integers), significantly reducing memory footprint and computational load. The mixed-precision strategy used in this project reflects a deep understanding of the model architecture: Dense Layers are the computation paths that every token passes through, so 8-bit quantization is used to maintain baseline quality; Expert Layers are the conditionally activated components in the MoE (Mixture of Experts) architecture where only a subset of experts participate in computation, allowing for more aggressive 4-bit quantization. The KV Cache (Key-Value Cache) stores historical states from the attention mechanism and consumes enormous memory in long-context scenarios — a 1M token KV cache can reach tens of gigabytes unquantized, and 8-bit quantization compresses it to a manageable size.

The developer specifically emphasized that the mixed 4-8 bit quantization strategy effectively compresses memory usage while preserving model quality. This is crucial for long-context scenarios, where accumulated quantization errors can lead to significant degradation in output quality.

The iogpu.wired_limit_mb Parameter

macOS by default limits the amount of wired memory the GPU can use, preventing GPU processes from exhausting system resources and causing crashes. iogpu.wired_limit_mb is a kernel parameter that controls this upper limit (in MB). During inference, this project requires the GPU to access 117GB of data, far exceeding the default limit, so the parameter must be set to 120000 (approximately 117GB) for it to run properly. The typical method involves editing /Library/Preferences/SystemConfiguration/com.apple.Boot.plist to add boot arguments, which requires administrator privileges and takes effect after a restart. This is the double-edged sword of Apple's unified memory architecture: while the GPU can access all system memory, the system also needs protective mechanisms to prevent runaway processes. Regular users should adjust this parameter with caution, as setting it too high may cause system instability.

Real-World Validation: A Custom Monitoring Plugin

To verify the engine's usability in actual workflows, the developer had Qwen build an MLX Serve Monitor plugin under deep context and integrated it into the Opencode2 application. This closed-loop demonstration of "using the model to develop tools, then using those tools to monitor the model" is far more convincing than raw performance benchmarks.

Launch Configuration and Technical Details

The project code and model weights are fully open-source. For users looking to reproduce the results, the developer provided launch parameters for single-concurrency operation:

--model ./llm/models/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit \\\\
--host 127.0.0.1 \\\\
--port 11234 \\\\
--ctx-size 1048576 \\\\
--kv-quant 8 \\\\
--max-tokens 64000 \\\\
--mtp \\\\
--prefix-cache-mem 10GB \\\\
--prefix-cache-entries 1 \\\\
--ssm-checkpoint-max 16 \\\\
--metrics

Several engineering highlights are evident from the configuration:

• --ctx-size 1048576 explicitly sets the 1M token context length
• --kv-quant 8 enables 8-bit KV cache quantization, the key mechanism for controlling memory usage in long-context scenarios
• --mtp enables Multi-Token Prediction, boosting generation throughput
• --prefix-cache-mem 10GB and --ssm-checkpoint-max 16 involve prefix caching and state checkpointing mechanisms for reducing redundant computation and maintaining response speed during long sessions

Multi-Token Prediction

Traditional autoregressive language models predict only the next token at each step, requiring n forward passes to generate n tokens. Multi-Token Prediction (MTP) is an inference optimization technique that adds multiple prediction heads to the model, enabling a single forward pass to predict the next k tokens simultaneously (typically k=2–4). If the predictions are accurate, k-1 computation steps can be skipped entirely, significantly boosting throughput. This technique is especially effective for high-determinism tasks like code generation — code syntax structures are relatively fixed, making subsequent tokens highly predictable. The project's 75 tok/s for coding tasks versus 40 tok/s for prose is partly attributable to MTP's advantage in code scenarios. It's worth noting that MTP improves throughput rather than latency — time-to-first-token remains unchanged, but overall generation speed is faster.

Prefix Caching and State Checkpointing

Prefix Caching leverages the fact that system prompts and conversation history typically remain unchanged across turns in long conversations: the KV cache for these fixed prefixes is saved and reused in subsequent turns, avoiding redundant computation. --prefix-cache-mem 10GB allocates 10GB of memory for storing these caches. State Checkpointing (SSM Checkpoint) periodically saves snapshots of the model's inference state, allowing recovery from the nearest checkpoint rather than recomputing from scratch when a session is interrupted or needs to backtrack. --ssm-checkpoint-max 16 limits the maximum number of checkpoints to 16, balancing memory usage with recovery granularity. Both mechanisms are essential for long-session scenarios: processing 1M tokens can take several minutes or longer, and without caching and checkpoints, every new input would require reprocessing the entire context — resulting in a terrible user experience.

Related Resources

• Engine code: github.com/ddalcu/mlx-serve
• Model weights: huggingface.co/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit
• Opencode2 plugin: github.com/beamivalice/opencode2-mlx-serve

Technical Value and Practical Challenges

MLX and Apple's Unified Memory Architecture

MLX is a machine learning framework released by Apple in December 2023, optimized specifically for Apple silicon, similar to Google's JAX. Its defining feature is full utilization of Apple silicon's Unified Memory Architecture (UMA) — where CPU, GPU, and Neural Engine share the same physical memory, eliminating the data copy overhead between CPU memory and GPU VRAM found in traditional architectures. This gives Mac devices a unique advantage for large model workloads: 128GB of unified memory can be fully utilized by the inference engine, whereas in traditional GPU setups, even if the system has 128GB of RAM, GPU VRAM is typically limited to 24GB, creating a severe bottleneck. MLX-serve is a community-developed inference serving layer built on top of MLX, providing an OpenAI API-compatible interface.

This project further validates Apple's unified memory architecture as a unique advantage for local large model inference. 128GB of unified memory enables consumer-grade devices to handle inference workloads on the order of 117GB — something nearly impossible with traditional discrete GPU VRAM. MLX, as Apple's official machine learning framework, combined with the community-driven serving layer, is progressively closing the gap in local long-context inference capabilities.

The developer candidly noted that the project "probably has bugs everywhere" and that testing every use case isn't feasible, calling on users to actively provide feedback. This reminds us that million-token local inference is still in its early exploration phase, and real-world stability and edge case handling still require community collaboration to refine.

For developers with high-end Macs who need to process extremely long documents, large codebases, or deep conversations, this solution offers a fully local option — no need to upload sensitive data to the cloud to perform million-token-scale inference on a single machine.

Key Takeaways