Ollama v0.40.0-rc1 Released: MLX Tokenizer Aligned with Publisher Semantics

Ollama has released v0.40.0-rc1, a release candidate focused on aligning the MLX backend tokenizer behavior with the original tokenization semantics defined by model publishers. While this may seem like a low-level implementation detail, it directly affects whether local inference results match the official model's expected output.

Core Update: Aligning with Publisher Tokenizer Semantics

The primary commit in this release candidate is titled "mlx: match publisher tokenizer semantics" (PR #18779). The goal is to make Ollama's tokenization behavior under the MLX backend strictly reproduce the processing pipeline as defined by the original model publishers.

Specifically, this update covers several critical areas:

• Pretokenizer stage order: Executes preprocessing steps in the order defined by the publisher;
• Split behavior: Ensures consistent text splitting logic;
• Unicode boundary handling: Guarantees correct boundary detection for multilingual and special character scenarios;
• Added-token normalization: Applies consistent handling to custom special tokens;
• Ranked BPE merges: Follows the priority ordering during BPE merge operations.

The update also specifically addresses two edge cases: empty added tokens, and behavior when Metaspace receives empty input — ensuring stable and predictable results even under these extreme inputs.

BPE (Byte Pair Encoding) is one of the most widely used tokenization algorithms in modern large language models. The core idea is to start from individual bytes or characters, then repeatedly merge the most frequently occurring adjacent symbol pairs in the training corpus into new symbols, until the target vocabulary size is reached. This process generates a "merge priority table," where each merge rule has a corresponding rank — a lower rank means the merge was executed earlier during training. During inference, tokenization must strictly follow the same ranked order; otherwise, the same text may be split into different token sequences, causing the model to "see" an input distribution that deviates from training. Metaspace is a special pretokenization technique common in the SentencePiece ecosystem. It replaces spaces between words with a special underscore character (▁), allowing the model to perceive word boundaries at the token level. Handling empty input for Metaspace requires additional protective logic.

Why Tokenization Consistency Matters

The tokenizer is one of the most overlooked yet impactful components in the LLM inference pipeline. A model is trained using specific tokenization rules to split text into tokens. If the tokenization approach during inference differs from training — even subtly, such as in Unicode boundary handling or special token processing — the input may be split into a different token sequence, ultimately degrading output quality.

For users running the MLX backend on Apple Silicon, this kind of consistency issue is especially sensitive. MLX is Apple's machine learning framework optimized for its chips, and Ollama leverages it to deliver efficient local inference on Mac devices. Only when tokenization semantics strictly match the publisher's definition can locally deployed models truly replicate their official behavior — avoiding the hard-to-debug problem of "same prompt, different results."

MLX is an open-source machine learning framework released by Apple in late 2023, designed specifically for the Unified Memory Architecture of Apple Silicon (M-series chips). Similar to how CUDA serves NVIDIA GPUs, MLX allows the CPU, GPU, and Neural Engine to share a single memory pool, eliminating the overhead of copying data between processors and enabling efficient local LLM inference on Mac devices. By integrating the MLX backend, Ollama allows users to run billion-parameter language models smoothly on a MacBook or Mac Studio without any cloud services. Because the MLX backend has its own independent low-level implementation path, its tokenization logic must be maintained separately from other backends (such as llama.cpp) — which is precisely the background for this dedicated alignment effort.

Test Coverage: Shared Reference Cases Across Go and Python

Also notable in this update is the investment in test engineering. The developers introduced shared reference test cases between Go and Python, using already-published tokenizers as the ground truth, automatically fetching missing models and failing on errors. This approach ensures that tests cover real-world scenarios rather than relying solely on manually constructed samples.

Building on this, the update also adds targeted regression tests focused on three error-prone areas:

1. Configuration precedence: Validates the resolution order when multiple configuration sources are present;
2. Byte fallback: Handles characters that cannot be directly covered by the vocabulary;
3. Parallel encoding: Ensures correctness of tokenization results under multithreaded or concurrent conditions.

Cross-language shared reference cases enable continuous cross-validation between the Go implementation and the Python reference implementation — a significant engineering investment for ensuring long-term tokenizer stability and preventing future refactors from silently introducing behavioral drift.

Byte fallback is a mechanism for handling characters absent from the vocabulary — typically rare Unicode characters, emoji, or other special symbols. When the tokenizer encounters a character not found in its vocabulary, it falls back to decomposing it into its UTF-8 byte sequence, mapping each byte to a special token such as <0xE4>. This mechanism ensures the tokenizer never crashes on arbitrary input, but different implementations may have subtle differences in when byte fallback is triggered and how byte tokens are encoded. This is especially prone to inconsistency in multilingual scenarios (particularly for non-Latin scripts like Chinese and Japanese) and with special symbols — which is why it was designated as a focal point for this round of regression testing.

What This Means for Users

As a release candidate, v0.40.0-rc1 is primarily aimed at users and developers who want to validate changes early. If you're running models on Apple Silicon via the MLX backend, this update should address tokenization inconsistencies you may have encountered, bringing local inference results closer to the model's official behavior.

This release was tagged by project maintainer dhiltgen with a verified GitHub signature. Since it remains an RC version, production users are advised to evaluate carefully and wait for the stable release before upgrading.

Overall, this is a classic "unglamorous but critical" under-the-hood refinement — it brings no flashy new features, but it meaningfully improves the reliability and reproducibility of local LLM inference.