TensorFold: A Cross-Platform LLM Inference Engine Written in Zig

TensorFold, written in Zig, unifies the Metal, CUDA, and Vulkan backends to tackle hardware fragmentation in local LLM inference.
TensorFold is an open-source LLM inference engine written in Zig that supports Metal, CUDA, and Vulkan backends, aiming to solve hardware fragmentation in local LLM inference. Zig's lack of garbage collection, explicit memory management, and compile-time specialization make it well suited to inference scenarios that demand low overhead and cross-platform compilation. By separating backend abstraction from operator implementation, the project keeps upper-level scheduling logic unified while swapping kernels per platform, reducing deployment costs on heterogeneous hardware. The project now has 1,818 stars and 385 forks, and community interest continues to grow, though the complexity of maintaining multiple backends means its maturity will still take time to prove.
A New Choice for Cross-Platform Inference: What Is TensorFold?

Running large language models locally used to mean choosing within the CUDA ecosystem—hardware other than NVIDIA GPUs rarely got an equivalent experience. The open-source project TensorFold on GitHub aims to change that: it positions itself as an LLM inference engine for Metal, CUDA, and Vulkan, written in the Zig language. It has currently earned 1,818 stars and 385 forks, with 58 new stars in a single day, indicating that community interest is on the rise.
Judging from its technology stack choices, TensorFold's ambitions are hardly subtle. CUDA covers NVIDIA data-center and consumer GPUs, Metal corresponds to Apple Silicon's unified memory architecture, and Vulkan extends its reach to AMD, Intel, and some mobile GPUs. Together, the three cover almost all mainstream local inference hardware. Unifying these three backends in a single engine means developers no longer need to maintain multiple sets of inference code for different platforms, and the fragmentation problem in model deployment can hopefully be consolidated.
Metal is the low-level graphics and compute API Apple provides for its own GPUs (including the integrated GPUs in Apple Silicon chips). Compared with general-purpose solutions like OpenCL, it gives developers more direct control over the hardware, making it the preferred choice for high-performance computing on macOS/iOS. CUDA is NVIDIA's parallel computing platform and programming model, and thanks to its mature toolchain and broad framework support, it has long dominated GPU inference. Vulkan, meanwhile, is a cross-platform graphics and compute API maintained by the Khronos Group, supporting Windows, Linux, Android, and other systems, and covering a variety of GPUs from AMD, Intel, Qualcomm, and others. The three correspond to different hardware ecosystems, and TensorFold's simultaneous support for all of them means it is trying to use a single engine to connect the previously fragmented landscape of local inference hardware.
Why Write an Inference Engine in Zig?
Zig is not a gimmick in this project but an engineering decision that closely matches its goals. Inference engines are extremely sensitive to memory layout, VRAM allocation, and kernel scheduling, and any implicit overhead in a runtime layer gets amplified across every token generation. Zig has no garbage collection, offers explicit memory management and good interoperability with the C ABI, and makes it easy to call each platform's driver interfaces and compute libraries directly.
At the same time, Zig's compile-time capabilities allow the same source code to be conditionally compiled for different backends, and combined with comptime for operator specialization, they reduce runtime dispatch costs. For a project that needs to interface with three vastly different APIs—Metal, CUDA, and Vulkan—this "one codebase, multiple compilation targets" capability is cleaner than introducing yet another abstraction runtime layer.
Zig is a systems-level programming language whose design emphasizes simplicity, explicit control, and seamless interoperability with the C ecosystem. It has no garbage collector, and all memory allocation must be managed explicitly by the developer, which avoids implicit runtime overhead—crucial for performance-sensitive inference engines. Zig's comptime mechanism allows code to be executed at compile time, enabling generic programming and operator specialization—for example, generating dedicated matrix multiplication kernels for different backends and reducing runtime branching. In addition, Zig can directly include C headers and call C libraries, making it very natural to interface with the official driver APIs of CUDA, Metal, and Vulkan. Together, these features form the technical rationale for TensorFold's choice of Zig.
The Significance of a Three-Backend Architecture

The cross-platform challenge of LLM inference lies not in model loading but in operator implementation. Core operators such as matrix multiplication, attention mechanisms, and normalization have very different efficient implementations across backends. TensorFold separates backend abstraction from operator implementation, which in theory allows the upper-level inference scheduling logic to remain unified while the underlying kernels are swapped per platform.
This design is especially appealing to several groups of users. Mac users have long been limited by how deeply inference frameworks support Metal, while the Vulkan backend gives non-NVIDIA GPUs on Windows and Linux a chance to participate in local inference. For teams that need to deploy models to heterogeneous edge devices, a unified engine can significantly reduce adaptation costs. However, multiple backends also mean multiplied maintenance burdens—operator coverage, numerical consistency, and performance tuning for each backend must be tracked independently, which is the most common long-term challenge for projects of this kind.
In LLM inference, operators refer to the basic computational units that make up a model's computation graph, such as matrix multiplication (GEMM), Softmax, LayerNorm, and the QKV projections in attention mechanisms. Different hardware backends implement the same operator in vastly different ways for efficiency: CUDA may use cuBLAS or custom kernels, Metal requires Metal Performance Shaders or hand-written Metal shaders, and Vulkan relies on compute pipelines and SPIR-V shaders. Therefore, a cross-platform inference engine must implement and optimize these operators separately for each backend while keeping the upper-level scheduling logic unified. TensorFold separates backend abstraction from operator implementation precisely to cope with this complexity, but it also means that operator coverage, numerical precision, and performance tuning for each backend must be maintained independently, becoming a key challenge for the project's long-term development.
Project Status and Use Cases
Judging from community data, TensorFold has moved past the "personal experiment" stage and entered the range of open-source projects with real attention. Its 1,818 stars and 385 forks indicate that developers are already trying it out or building on it. It should be clear, however, that early-stage inference engines of this kind usually have uneven backend maturity: the CUDA ecosystem has the most complete toolchain, while the depth of Metal and Vulkan implementations often depends on the maintainers' investment.
Those who may want to follow TensorFold include researchers who want to get models running on new hardware, application developers who need a cross-platform inference solution, and learners who want to study how inference engines are implemented internally. If you are looking for out-of-the-box production deployment, mature commercial frameworks remain the safer choice; but if you value platform freedom and are willing to participate in building an early-stage project, TensorFold offers a direction worth tracking.
Conclusion
The value of TensorFold is not that it is "yet another inference engine," but that it uses Zig to bring the Metal, CUDA, and Vulkan backends into a single engineering framework, addressing the long-standing hardware fragmentation problem of local LLM inference. The project's current community traction shows that there is real demand for this direction, while the complexity of maintaining multiple backends means its maturity will take time to prove. For developers focused on cross-platform AI infrastructure, this is an open-source project worth adding to the watchlist.
Related articles

Why Microsoft CEO Nadella Still Backs Nvidia After Jensen Huang's Departure
Microsoft CEO Nadella publicly expresses confidence in Nvidia interim CEO Colette Kress. This article analyzes Nvidia's power transition after Jensen Huang's departure, Microsoft's compute dependency, and the trust logic of the AI supply chain.

Nadella: Critical Infrastructure Is 'Unpatchable' as AI Cybersecurity Arms Race Escalates
Microsoft CEO Nadella addresses AI cyberattack fears, saying much critical infrastructure is unpatchable, downplaying the Outlook outage, and analyzing AI's dual role in cybersecurity.

Musk Backs Jensen Huang: Firing Him Is Like Cutting Jordan
Musk says Jensen Huang is loyal to the U.S. and likens firing him to cutting Michael Jordan—highlighting politicization anxieties in the AI chip industry and defending talent value.