Ollama v0.40.5 Released: MLX Metal Kernels Ported to CUDA

Ollama has released the v0.40.5 pre-release version, with the core change being the porting of MLX's Metal custom kernels to the CUDA platform. It also updates the model recommendation documentation, improves custom tool call support in OpenAI Responses history, and enhances cloud payment error messages.

Core Update: MLX Metal Kernels Ported to CUDA

The most noteworthy technical change in this update is PR #18657 submitted by developer @dhiltgen—porting MLX's Metal custom kernels to CUDA. MLX is Apple's machine learning framework, and its Metal kernels are specifically optimized for Apple Silicon. Migrating this computational logic to CUDA means Ollama is extending inference acceleration capabilities originally tied to the Apple ecosystem to the NVIDIA GPU platform.

For Ollama users with NVIDIA GPUs, this is expected to bring more efficient local model inference performance, especially when running quantized models. The MLX framework itself is known for efficient inference under a unified memory architecture, and its kernel design philosophy being reused on CUDA may bring an optimization path different from traditional CUDA implementations.

The same contributor also submitted an MLX version number upgrade (PR #18993), indicating that MLX-related modules are still actively iterating.

MLX is an open-source machine learning framework launched by Apple in late 2023, designed specifically for Apple Silicon's Unified Memory Architecture. Unlike traditional GPU inference frameworks, MLX's computational graph and memory management assume that the CPU and GPU share the same physical memory, thereby avoiding frequent data copy overhead. Its Metal custom kernels are low-level computation units written in Metal Shading Language, optimized at the instruction level for the GPU cores of M-series chips, and are commonly used for inference hotspots such as quantized matrix multiplication and attention mechanisms.

CUDA, on the other hand, is NVIDIA's parallel computing platform and programming model, whose kernels are written in CUDA C/C++ and run on GPUs with dedicated VRAM. Porting Metal kernels to CUDA is not a simple syntax translation: the data residency assumptions under the unified memory model, the mapping of thread groups to thread blocks, and the strategies for utilizing registers and shared memory during quantization and dequantization all need to be redesigned. The value of Ollama's move lies in bringing MLX's kernel approaches validated on low-bit quantized inference into the NVIDIA ecosystem, potentially forming an optimization path distinct from mainstream implementations such as cuBLAS/FlashAttention.

Developer Experience Improvements

For the OpenAI-compatible interface, PR #18911 allows accepting custom tool calls in Responses history. For developers building Agent applications through Ollama's OpenAI-compatible layer, this means that the context passing of tool calls in multi-turn conversations is more complete, reducing the hassle of manually assembling history.

On the documentation side, PR #18981 updated model recommendations and standardized language, reducing confusion for new users when choosing models.

The Ollama desktop app optimized cloud payment error messages (PR #18986), allowing users to more clearly identify the cause when encountering payment-related issues. This PR is also the first contribution from new contributor @willietran.

The OpenAI Responses API is a newer stateful conversation interface from OpenAI. Unlike traditional Chat Completions, it maintains conversation history on the server side, and the client only needs to submit new input. The "custom tool calls" within it refer to the model outputting structured function call requests (tool calls), which are executed externally and then returned with results. In Agent scenarios, a single round of conversation often includes multiple tool calls and result round-trips, and the history must fully preserve the pairing relationships between these calls and outputs so the model can correctly understand the context. Ollama's OpenAI-compatible layer previously had incomplete support for such history items, requiring developers to manually concatenate or trim them. PR #18911 allows the compatibility layer to directly accept and pass through custom tool call records, thereby simplifying state management for Agent applications.

Version Nature and Impact

v0.40.5 is currently a pre-release, tagged v0.40.5-rc1, with commit hash 00bef57, automatically released by GitHub Actions and accompanied by a GPG verification signature. The official version has not yet been released, and production environments are advised to wait for the stable version.

Judging from the update contents, this version leans toward expanding low-level inference capabilities and refining developer interfaces, rather than adding end-user-facing features. If the MLX-to-CUDA port validates smoothly, it may affect Ollama's inference efficiency performance on non-Apple hardware in the future, and is worth continued attention.

One-line Summary

Ollama v0.40.5 pre-release ports MLX Metal kernels to CUDA and improves developer interfaces.

Paragraph Summary

Ollama has released the v0.40.5-rc1 pre-release version, with the core change being PR #18657 submitted by contributor @dhiltgen, which ports the Metal custom kernels of Apple's MLX framework to CUDA. This means that inference acceleration capabilities originally tied to Apple Silicon are beginning to extend to NVIDIA GPUs, potentially bringing different optimization paths for quantized model inference. On the developer experience side, the OpenAI-compatible interface now supports accepting custom tool calls in Responses history, improving context passing in multi-turn Agent conversations; documentation updated model recommendations and standardized language; and the desktop app optimized cloud payment error messages. This version leans toward low-level inference capabilities and developer interface refinement rather than new end-user features. The official version has not yet been released, and production environments are advised to wait for the stable version.