Inductor
The backend component of the PyTorch 2.0 compilation stack, responsible for lowering computation graphs into efficient Triton GPU kernels or C++ code for CPU execution. It constitutes the final execution layer through which torch.compile achieves performance acceleration.
Timeline (last 90 days)
Inductor iterates quickly, and its interfaces and internal IR change from time to time, so hardware vendors' adaptation layers need to keep following up, which constitutes a kind of maintenance cost.
When generating kernels, Inductor by default translates the computation graph into Triton code, and falls back to the C++/OpenMP path when the target hardware does not support Triton or is not suitable for it.
In its adaptation for Spyre, IBM did Spyre-specific scheduling and memory hierarchy adaptation on top of Inductor, needing only to map the intermediate representation output by Inductor to the Spyre instruction set.
Inductor is becoming the de facto intermediate layer of the AI chip ecosystem, reducing the complexity of chip-to-framework adaptation from O(M×N) to close to O(M+N).
With the continuous investment of Microsoft, Intel, AMD and IBM, the quality of Inductor's general-purpose code generation on non-NVIDIA hardware can already cover a considerable portion of training and inference workloads.
As a general-purpose backend, in workloads that are extremely latency-sensitive or highly customized at the operator level, Inductor's general passes may be inferior to specialized compilers, and IBM needs to retain a handwritten optimization channel for Spyre's key operators.
The IBM Spyre AI accelerator switched its torch.compile backend from the in-house sendnn compilation stack to the open-source community-maintained Inductor.
Vendors such as Intel and AMD also adopt the strategy of plugging into the Inductor extension mechanism and doing hardware-specific optimizations through custom backend interfaces, rather than maintaining parallel compilation pipelines.
TorchInductor is PyTorch's default backend compiler, capable of lowering computation graphs into Triton GPU kernels or C++/OpenMP CPU code.
In PyTorch 2.14, NVGEMM brings CUTLASS kernels generated by CuTeDSL into the Inductor compilation backend.
3 more timeline events
All Facts (13)
Inductor iterates quickly, and its interfaces and internal IR change from time to time, so hardware vendors' adaptation layers need to keep following up, which constitutes a kind of maintenance cost.
50%UnverifiedWhen generating kernels, Inductor by default translates the computation graph into Triton code, and falls back to the C++/OpenMP path when the target hardware does not support Triton or is not suitable for it.
50%UnverifiedIn its adaptation for Spyre, IBM did Spyre-specific scheduling and memory hierarchy adaptation on top of Inductor, needing only to map the intermediate representation output by Inductor to the Spyre instruction set.
50%UnverifiedInductor is becoming the de facto intermediate layer of the AI chip ecosystem, reducing the complexity of chip-to-framework adaptation from O(M×N) to close to O(M+N).
50%UnverifiedWith the continuous investment of Microsoft, Intel, AMD and IBM, the quality of Inductor's general-purpose code generation on non-NVIDIA hardware can already cover a considerable portion of training and inference workloads.
50%UnverifiedAs a general-purpose backend, in workloads that are extremely latency-sensitive or highly customized at the operator level, Inductor's general passes may be inferior to specialized compilers, and IBM needs to retain a handwritten optimization channel for Spyre's key operators.
50%UnverifiedThe IBM Spyre AI accelerator switched its torch.compile backend from the in-house sendnn compilation stack to the open-source community-maintained Inductor.
50%UnverifiedVendors such as Intel and AMD also adopt the strategy of plugging into the Inductor extension mechanism and doing hardware-specific optimizations through custom backend interfaces, rather than maintaining parallel compilation pipelines.
50%UnverifiedTorchInductor is PyTorch's default backend compiler, capable of lowering computation graphs into Triton GPU kernels or C++/OpenMP CPU code.
50%UnverifiedIn PyTorch 2.14, NVGEMM brings CUTLASS kernels generated by CuTeDSL into the Inductor compilation backend.
50%UnverifiedInductor is the default compilation backend for PyTorch 2.x, responsible for lowering computation graphs into efficient low-level code such as Triton or C++.
50%UnverifiedThe Inductor backend lowers computation graphs to efficient Triton GPU kernels or C++ code for CPU.
50%UnverifiedBringing CUTLASS kernels into Inductor is expected to deliver substantial performance gains for GEMM-intensive workloads such as Transformers and large language model training and inference.
50%