Ollama v0.40.4-rc1: Adding Kev Decision Scoring to MLX

Ollama has released the v0.40.4-rc1 pre-release, with the core change being the introduction of Kev decision scoring capability in the MLX backend. This update was tagged and committed by maintainer dhiltgen, with a signature verification badge on GitHub. For developers following local inference frameworks and the Apple Silicon ecosystem, this change is worth noting.

What Changed in This Version

According to the commit message (#18976), this change mainly involves two things: first, enabling MLX to load Kev models along with their FP32 decision head and release metadata; second, supporting System One scoring on top of shared-state caching, while adding Kev-specific resource limits.

From the wording, this is not a simple version number bump, but rather the integration of a decision scoring mechanism into the MLX inference path. A "decision head" typically refers to an additional output structure attached beyond feature extraction, used to produce a judgment or score; and "System One scoring" suggests this mechanism may be related to fast, intuitive judgment processes.

FP32 Decision Head: A decision head is an additional output layer beyond the model's backbone network, typically receiving feature vectors extracted by the backbone and outputting a judgment result or score. Loading the decision head at FP32 (32-bit floating point) precision means this portion of weights is not quantized or compressed, and its numerical representation range and precision are closer to the training state. In contrast, the backbone is typically quantized to INT8, INT4, or other low-bit formats to save memory and speed up inference. The reason for preserving FP32 precision specifically for the decision head is that scoring output is more sensitive to numerical perturbations: low-precision quantization can easily cause score ranking shifts, and ranking accuracy is precisely the core metric for filtering and candidate ranking tasks. The cost is that this portion of weights occupies more VRAM and requires FP32 compute support during inference.

What the Combination of MLX and Kev Means

MLX is Apple's array and machine learning framework designed for Apple Silicon, and has already seen considerable adoption in local inference scenarios. Integrating Kev decision scoring into MLX essentially enables local devices to run models with decision output capabilities without relying on remote services.

This has direct value for two types of scenarios. The first is applications that need to do fast screening or scoring on-device, such as content filtering and candidate ranking. The second is scenarios sensitive to latency and privacy, where completing scoring locally can eliminate network round-trips and avoid data leaving the device.

The MLX framework is a machine learning array framework open-sourced by Apple in late 2023, specifically designed for the unified memory architecture of M-series chips. Compared to frameworks like PyTorch, MLX's core features are lazy computation and a unified memory model—CPU and GPU share the same physical memory, avoiding the frequent data copy overhead in traditional inference. This is particularly important for local inference: model weights need not be moved between devices after loading, and the boundary between VRAM and memory is blurred. Ollama's choice of MLX as one of its backends reflects its native efficiency on Apple Silicon. It should be noted that MLX's operator coverage and ecosystem maturity still lag behind the CUDA ecosystem, and certain model structures may require additional adaptation work.

Shared-State Caching and Kev-Specific Limits

The "shared-state caching" mentioned in the commit is an engineering detail, but its impact is significant. Caching shared state avoids redundant computation, especially when the scoring interface is called consecutively, significantly reducing overhead. However, caching also brings consistency challenges, so the commit simultaneously adds Kev-specific limits, using quotas or boundary conditions to constrain resource usage, preventing cache bloat or a single request from monopolizing too many resources.

Such limits are not uncommon in local inference frameworks, but binding them to a specific model family (Kev) indicates that the maintainers have clear expectations for managing resource behavior.

System One scoring borrows its naming from the dual-process theory in cognitive psychology: System One refers to fast, automatic, intuitive judgments, while System Two is slow processing requiring reasoning and deliberation. In the model context, System One scoring typically refers to a fast score produced by a single forward pass, without multi-step reasoning, self-reflection, or chain-of-thought. This type of scoring has low latency and small compute requirements, making it suitable for high-frequency calls or as an initial screening layer for large-scale filtering. It is a complementary concept to the "decision head": the decision head provides the output structure, while System One describes the evaluation method. Understanding this helps determine what tasks this capability suits—it is low-latency fast judgment, not complex decision-making requiring deep reasoning.

What Developers Should Note

This is an rc version, i.e., release candidate, meaning APIs or behavior may still change before the official release. If your project depends on stable interfaces, it is advisable to wait for the official release before upgrading. If you are evaluating on-device decision scoring solutions, you can do prototype validation on this version, focusing on two things: whether the accuracy and VRAM overhead brought by the FP32 decision head are acceptable, and whether shared-state caching actually hits under your call patterns.

Another signal worth noting is that Ollama is incorporating more fine-grained model capabilities (not just text generation, but also decision scoring) into a unified local inference framework. For teams building on-device intelligent applications, this is a reusable infrastructure direction.

Release Candidate (rc) is a stage in the software release process, referring to a build where features are essentially frozen and preparing for official release but still undergoing final validation. It sits between beta and stable: compared to beta, rc typically no longer introduces new features; compared to stable, rc still allows rollback or behavior changes if serious issues are found. Therefore, rc versions are not suitable for direct production use, especially for projects depending on stable API contracts. The reasonable approach is to use rc in isolated environments for compatibility testing and feature prototype validation, feed issues back to maintainers, and upgrade after the official release. Ollama's rc tags are applied by maintainers with signature verification, indicating the build source is trustworthy, but this is separate from interface stability.

Summary

The core value of v0.40.4-rc1 lies in bringing Kev decision scoring into MLX, along with caching and resource limits. The change itself is not large, but it points to a clear trend: local inference frameworks are moving from "can run models" to "can run models with specific capabilities."

One-line Summary

Ollama v0.40.4-rc1 introduces Kev decision scoring in the MLX backend, along with caching and resource limits.

Paragraph Summary

Ollama released the v0.40.4-rc1 pre-release, with the core change being the integration of Kev decision scoring capability into the MLX backend. The commit (#18976) enables MLX to load Kev models along with their FP32 decision head and release metadata, supports System One scoring on top of shared-state caching, and adds Kev-specific resource limits to constrain cache bloat and resource monopolization. This means local devices can directly run models with decision output capabilities, suitable for on-device fast screening, candidate ranking, and scenarios sensitive to latency and privacy. As it is an rc version, interfaces and behavior may still be adjusted; prototype validation is recommended, focusing on the accuracy and VRAM overhead of the FP32 decision head, as well as cache hit rates. Overall the change is modest, but it points to the trend of local inference frameworks moving from "can run models" to "can run models with specific capabilities."