Ollama v0.40.4 Pre-release: llama.cpp Upgrade and MLX Optimizations

Ollama has released the v0.40.4 pre-release version. Although this update contains only a small number of commits, it involves inference engine upgrades, server-side routing optimizations, and improvements to MLX state management on the Apple Silicon platform. For developers who rely on Ollama for local model deployment, these changes are worth paying attention to.

Core Update: llama.cpp and EmbeddingGemma 2 Support

The most critical change in this update is the upgrade of llama.cpp to build b11521, submitted by jmorganca. This upgrade is primarily aimed at decision models and EmbeddingGemma 2 support.

As a core component of Ollama's underlying inference engine, llama.cpp's version iterations directly affect model loading efficiency and inference performance. Advancing the underlying engine to a newer build means Ollama can be compatible with more newly released or upcoming model architectures. The addition of EmbeddingGemma 2 is particularly noteworthy—embedding models play a foundational role in RAG (Retrieval-Augmented Generation) and semantic search scenarios, and the completeness of Ollama's support for such models will directly impact the development experience for downstream applications.

llama.cpp is an LLM inference framework written in C/C++, initiated by Georgi Gerganov, with the core goal of enabling LLMs to run efficiently on consumer-grade hardware (laptops, desktops, Raspberry Pi, etc.) without relying on expensive GPU clusters. It achieves cross-platform acceleration through the GGUF quantization format, optimized kernels for different CPU instruction sets (AVX, NEON, etc.), and backends such as Metal and CUDA. Ollama itself does not build inference capabilities from scratch but uses llama.cpp as its underlying engine, so its version number (e.g., b in b11521 represents build) directly affects the model architectures, quantization types, and inference performance that Ollama supports. The community often treats llama.cpp's build number as the de facto standard for "which new models can run."

EmbeddingGemma 2 belongs to the Google Gemma series of text embedding models, specifically designed to map text into dense vectors rather than generative dialogue. EmbeddingGemma focuses on small size, multilingual support, and on-device deployment, making it suitable for generating semantic vectors locally. Its key difference from generative models is that the output is a fixed-dimension vector rather than a token sequence, so it serves as an "indexing layer" in tasks such as RAG, semantic retrieval, clustering, and deduplication. Ollama's support for such models means developers can complete the two steps of "vectorization + generation" within the same local toolchain, without needing to deploy a separate embedding service.

Server-side Optimization: Reusing Verified Manifests and Loaded Models

dhiltgen's server-side changes focus on performance optimization of inference routing—reusing verified manifests and already-loaded models.

This change addresses a practical problem: in multi-request scenarios, if every inference request re-verifies the model manifest or reloads the model, it introduces unnecessary latency and resource consumption. Through caching and reuse mechanisms, Ollama can respond faster to consecutive inference requests, which is especially important for high-frequency invocation scenarios in production environments. Model loading is often one of the most time-consuming steps in the local inference pipeline, and reducing redundant loading can directly improve the user's waiting experience.

Manifest in Ollama refers to the file listing that describes a model, containing metadata such as model layers, weight digests, quantization formats, templates, and parameters. It is typically stored in JSON format and addressed by content hash. Before loading a model each time, Ollama needs to verify whether the manifest is complete and whether the corresponding blobs already exist in the local cache—a process involving file reads and hash comparisons.

In high-concurrency or consecutive request scenarios, if every request goes through the full process of "read manifest → verify digest → load weights," the accumulated I/O and parsing overhead can be considerable. dhiltgen's change allows the server to cache verified manifests and model instances already resident in memory, so subsequent requests can reuse them directly. This is a typical inference service optimization approach: amortizing the "startup cost" from every request across the process lifecycle. The trade-off is the need for more careful memory management strategies, such as deciding when to unload idle models under VRAM/memory pressure, otherwise long-term residency could exhaust resources.

MLX-related Improvements: Continuous Refinement on the Apple Silicon Platform

This update includes three changes directly related to MLX, demonstrating Ollama's continued investment in the Apple Silicon platform.

State compression after speculative rollback (submitted by dhiltgen): After a rollback occurs in speculative decoding, the recovered recurrent state is compressed. Speculative decoding is a technique to accelerate inference by having a small model rapidly generate drafts that are then verified by a larger model, but the state must be rolled back when verification fails. This change makes state management after rollback more compact, helping to reduce memory usage.

MLX / MLX-C version upgrades (submitted by dhiltgen): Synchronizing the dependency versions of MLX and MLX-C to ensure consistency between the underlying framework and upper-layer implementation.

Although these two changes involve deep technical details, they both point in the same direction—making Ollama's inference on Mac devices more stable and efficient.

MLX is an open-source array computing and deep learning framework introduced by Apple's Machine Learning Research team, designed specifically for Apple Silicon's unified memory architecture, with Python, C++ (MLX-C), and Swift interfaces. Unlike the CUDA ecosystem, MLX uses a lazy computation graph and a unified memory model, where CPU and GPU can share the same memory block, avoiding repeated data copying between devices, thus achieving better energy efficiency and latency performance for local inference on Mac.

Speculative decoding is a technique that accelerates generation without changing the output distribution: first, a small and fast "draft model" generates several candidate tokens at once, and then the target large model verifies them all in parallel in a single pass. If all pass, it is equivalent to replacing multiple large-model forward passes with one; if rejected at a certain position, the subsequent accepted tokens must be discarded and the state rolled back to before the rejection point. Rollback involves truncation and reconstruction of caches such as KV Cache and recurrent state. Improper handling can cause memory fragmentation or usage inflation. Ollama's compression of post-rollback state this time is precisely aimed at reducing the extra overhead of this acceleration path.

Launch Experience: Codex Model Selector Integration

ParthSareen's launch change incorporates installed models into Codex's /model selector. This is a developer experience improvement that allows users in the Codex environment to directly see locally installed Ollama models and switch to them without additional configuration.

Although this change is small, it reduces friction between toolchains. For developers using both Ollama and Codex, the model discovery and selection process becomes smoother.

Codex here refers to OpenAI's coding agent/command-line tool (Codex CLI), which supports connecting to different model backends to perform code generation, editing, and terminal tasks. Its /model is a slash command in interactive sessions used to list and switch currently available models.

The significance of Ollama's integration with Codex lies in connecting the two toolchains of "local models" and "coding agents." Previously, if users wanted to use local models pulled by Ollama in Codex, they often had to manually configure parameters such as base URL and model name; this change directly exposes installed Ollama models in the /model selector, effectively letting Codex automatically discover the local model list. For developers, this lowers the barrier to using coding agents in privacy-sensitive or offline scenarios—code does not need to leave the local machine while still enjoying agent-style automated editing. This also reflects Ollama's expansion from a simple "model runner" to a "local AI toolchain hub."

Observations on Version Iteration Rhythm

From v0.40.3 to v0.40.4, three commits and four contributors—this pre-release version's changes are relatively restrained. But on closer inspection, each change has a clear direction: following up on the underlying engine, server-side performance optimization, platform-specific improvements, and developer tool integration.

As a local large model running tool, Ollama's update rhythm reflects the team's continuous investment in multi-platform support (especially the MLX path on Apple Silicon) and inference efficiency. This time, the support for EmbeddingGemma 2 is worth noting for developers building RAG applications, while the server-side manifest reuse optimization may demonstrate value in long-running services.

Pre-release versions usually mean the official release will follow soon, and developers following Ollama can watch for the subsequent official release.

One-line Summary

Ollama v0.40.4 pre-release: upgrades llama.cpp to support EmbeddingGemma 2, optimizes server-side and MLX.

Paragraph Summary

Ollama has released the v0.40.4 pre-release version. Although it contains only 3 commits involving 4 contributors, the direction of changes is clear. The core is upgrading the underlying inference engine llama.cpp to b11521, adding support for decision models and the EmbeddingGemma 2 embedding model, the latter being especially important for RAG and semantic search developers. On the server side, dhiltgen optimized inference routing, reusing verified manifests and already-loaded models to reduce redundant loading overhead in multi-request scenarios. On Apple Silicon, there are three MLX-related improvements, including state compression after speculative decoding rollback and MLX/MLX-C version synchronization. In addition, Codex's /model selector begins incorporating locally installed models, improving toolchain connectivity. Overall, this reflects Ollama's continuous investment in multi-platform support and inference efficiency.