torch.compile

The torch.compile feature introduced in PyTorch 2.0 improves inference and training speed through graph capture and compilation optimization while maintaining the ease of use of dynamic graphs.

torch.compile delivers an average training speedup of 30% to 2x on mainstream models and can be enabled with just a single decorator call.

torch.compile is a core compilation feature introduced in PyTorch 2.0, built on TorchDynamo (bytecode capture) and TorchInductor (code generation backend).

torch.compile is a just-in-time compilation (JIT) mechanism introduced in PyTorch 2.0, and its workflow has three steps: TorchDynamo captures the computation graph at the bytecode level, AOTAutograd handles the forward and backward graph structures, and finally it is handed to the backend compiler to generate executable code.

The IBM Spyre AI accelerator switched its torch.compile backend from the in-house sendnn compilation stack to the open-source community-maintained Inductor.

TorchInductor is PyTorch's default backend compiler, capable of lowering computation graphs into Triton GPU kernels or C++/OpenMP CPU code.

The torch.compile backend of PyTorch 2.0 extensively uses kernels generated by Triton.

TorchScript and torch.compile are used for model compilation and acceleration

The fullgraph mode of torch.compile captures and optimizes the entire model computation graph as a single complete unit.

torch.compile relies on the TorchDynamo bytecode interception mechanism and the TorchInductor code generation backend.

In fullgraph mode, operations that rely on Python dynamic control flow or call external C++ extensions without FakeTensor registration will cause a graph break, causing compilation to fail.

vLLM's internal implementation changes will make it incompatible with torch.compile in fullgraph mode.

TorchInductor is the default backend for torch.compile and relies heavily on Triton as an intermediate layer for code generation when producing fused operators.

torch.compile places higher demands on handling scenarios such as dynamic shapes and graph breaks.

PyTorch's torch.compile, IREE, and the compiler stacks of numerous AI chip vendors are all built on top of MLIR.

Users can obtain compilation acceleration by calling torch.compile(model) before the model, which typically delivers inference or training speed improvements of 10%–200% on mainstream GPUs.

Training uses batch size 256, 400,000 steps, EMA 0.9999, linear learning rate decay over the final quarter, and torch.compile achieves a 2.4× speedup over eager mode.

One of the design goals of torch.compile is to maintain semantic compatibility with eager mode, so developers only need to add a decorator around the model to enable compilation optimizations.

torch.compile takes effect with just a single-line decorator added before a model or function, significantly lowering the barrier to performance optimization compared to the previously required manual use of torch.jit.script or torch.jit.trace to trigger compilation optimization.

In large model training scenarios, the collaborative optimization of torch.compile and FSDP (Fully Sharded Data Parallel) is one of the most active research directions in the community.