ExecuTorch Lands on Arm Cortex-M: Running PyTorch Models Directly on MCUs

The Dilemma of Traditional MCU Deployment: The Format Conversion Trap

In embedded AI development, deploying a trained PyTorch model to an Arm Cortex-M microcontroller (MCU) has always been a headache. Developers typically have to go through a tedious format conversion process—exporting the model from PyTorch to ONNX, then converting it again through TensorFlow Lite for Microcontrollers or another inference engine. This process not only introduces extra toolchain maintenance costs, but more problematically, it creates cross-framework numerical mismatches: the same model produces slightly different results when inferred under different frameworks, making debugging extremely time-consuming.

According to the PyTorch official blog, this situation is now changing. The introduction of ExecuTorch aims to completely eliminate the model format conversion step, allowing PyTorch models to run on MCUs in a more native way.

ONNX (Open Neural Network Exchange) is an open model exchange format jointly initiated by Microsoft, Meta, and other vendors, with the goal of allowing models trained in different frameworks (PyTorch, TensorFlow, MXNet, etc.) to interoperate. It is essentially a framework-neutral specification for operators and graph structures. In actual deployment pipelines, ONNX is often used as an "intermediate language": PyTorch first exports the model as an ONNX graph, which is then parsed and converted by the target inference framework (such as TensorFlow Lite Micro). The problem is that this "translated twice" process involves differences in operator definitions, data types, rounding strategies, and other details. Any inconsistency can cause numerical drift, and errors are often hidden deep in the pipeline and difficult to locate.

TensorFlow Lite for Microcontrollers (TFLite Micro) is Google's inference engine for MCU scenarios and has long been the de facto standard deployment solution on Cortex-M. It provides a streamlined interpreter, fixed-point/quantized operator libraries, and static memory planning capabilities, without depending on the full C++ standard library. TFLite Micro itself also accepts TFLite format input, so migrating from PyTorch typically requires a multi-hop conversion of "PyTorch→ONNX→TFLite→TFLite Micro," increasing both toolchain complexity and the risk of numerical errors.

How ExecuTorch Bypasses Format Conversion

ExecuTorch is a native on-device inference runtime within the PyTorch ecosystem. Unlike the traditional three-stage process of "training framework→intermediate format→deployment framework," ExecuTorch's design philosophy is: after exporting from PyTorch, it goes directly into a lightweight runtime environment, with no need for intermediate layers such as ONNX. For resource-constrained CPUs like Arm Cortex-M, this means a significantly simplified toolchain—developers no longer have to maintain build systems for two or more frameworks simultaneously.

More critically, there is numerical consistency. Because cross-framework conversion is bypassed, the behavioral difference between training and inference is minimized. The official blog points out that this solves the long-standing pain point of "cross-framework numerical mismatch" that has plagued embedded developers—previously requiring layer-by-layer output comparisons and investigation of operator implementation differences, this problem is now avoided in principle.

ExecuTorch is the on-device inference runtime officially launched by PyTorch in 2023, covering phones, embedded devices, and even MCUs. Its core idea is to introduce a unified intermediate representation, EXIR (Export IR): models are exported from the PyTorch frontend to EXIR via torch.export, then packaged in the .pte (PyTorch ExecuTorch) format and handed to the runtime for direct execution. Because training and deployment share the same IR system, operator semantics, data types, and rounding rules are consistent, avoiding cross-framework translation errors in principle.

ExecuTorch adopts a layered runtime design: representation layer (graph definition), operator layer (Kernel), and backend layer (Backend Delegate). For specific hardware, subgraphs can be lowered to optimized backends via Delegate, such as operator libraries customized for Cortex-M. This architecture makes it easy to trim the runtime size for the target platform, which is key to its ability to reach down to MCUs.

Technical Challenges and Implementation Path on Cortex-M

The Arm Cortex-M series CPUs are characterized by extremely small memory (typically tens to hundreds of KB of RAM), low clock speeds, no operating system or only an RTOS, and a lack of standard C++ runtime support. Squeezing PyTorch models into such an environment requires ExecuTorch to do extensive trimming and adaptation at the runtime level.

Based on information released by PyTorch officially, the project team has built dedicated operator libraries and memory management solutions for Cortex-M, ensuring that the inference process does not depend on dynamic memory allocation or complex STL features. This means models can allocate memory statically and execute deterministically—a basic requirement for hard real-time embedded scenarios. For teams already familiar with the PyTorch training workflow, the path from cloud models to MCU firmware is significantly shortened.

Static memory allocation is a key constraint for MCU inference. Cortex-M typically has only tens to hundreds of KB of SRAM and no MMU; heap allocation easily causes fragmentation and introduces nondeterministic latency, so the inference runtime must use a pre-planned memory pool (arena). Through memory planning, ExecuTorch calculates the lifetime and offset of each tensor before execution, allowing all intermediate results to reuse the same static buffer and avoiding runtime malloc/free.

Deterministic execution means that given the same input, the inference latency, memory usage, and output results are completely consistent and do not vary with external factors such as heap state or task scheduling. This is a non-negotiable requirement in hard real-time scenarios such as industrial control, motor drives, and medical sensing. It requires: no dynamic memory, no recursion, no undefined behavior, and bounded operator latency. ExecuTorch's adaptation for Cortex-M is precisely centered on these constraints, trimming the operator library and avoiding dependence on the full STL and exception mechanisms as much as possible.

Practical Impact on Embedded AI Developers

The most direct change is workflow convergence. In the past, a typical MCU AI project required: PyTorch training→export to ONNX→convert to TFLite Micro→hand-write C++ inference glue code→debug numerical errors. Now, in theory, it can be: PyTorch training→ExecuTorch export→run on Cortex-M. The toolchain shrinks from three or four to one or two, and the burden of version alignment is correspondingly reduced.

Another implicit benefit is model iteration speed. When training and deployment use the same IR (intermediate representation) system, the loop of modifying the model structure, re-exporting, and flashing for testing is noticeably accelerated. For small vision or sensor fusion models that require frequent tuning, this efficiency gain cannot be ignored.

Of course, the hardware ceiling of Cortex-M still exists. Model size, operator support scope, and inference latency are all limited by the chip itself. ExecuTorch solves the "deployment path" problem, not the "insufficient compute" problem. Developers still need to trim models according to the SRAM and Flash capacity of the specific MCU.

Ecosystem Significance and Follow-up Observations

Bringing ExecuTorch to Arm Cortex-M is an important step in PyTorch's continued penetration into the embedded side. Arm's dominance in the MCU market means this adaptation will cover a vast number of IoT devices, wearables, and industrial sensor nodes. For the embedded community that has long relied on TensorFlow Lite Micro, there is now a native option from the PyTorch ecosystem, and competition will drive the maturation of the entire on-device inference toolchain.

What is worth watching next is operator coverage and the completeness of the community toolchain. If ExecuTorch can cover the common CNN and lightweight Transformer operators used on Cortex-M and provide stable quantization support, its adoption rate on MCUs is expected to rise rapidly. For embedded teams currently selecting a solution, this is a window worth evaluating and trying.

Operator coverage is one of the core metrics for evaluating the maturity of an on-device inference framework. As long as a model uses any operator not supported by the runtime, the entire graph cannot be deployed, so coverage directly determines the boundary of usability. For Cortex-M scenarios, the key is covering the common operator sets required by CNN backbones (convolution, depthwise separable convolution, pooling, normalization) and lightweight Transformers (attention, LayerNorm, GELU).

Quantization support likewise determines feasibility on the MCU side. MCU compute is limited, and int8 or int16 quantization is usually required to achieve acceptable latency and memory usage. A complete quantization pipeline includes post-training quantization (PTQ), quantization-aware training (QAT), and the corresponding fixed-point operators at runtime. If quantization accuracy loss is uncontrollable, it will bring debugging burdens similar to "cross-framework numerical mismatch." Therefore, a stable and reproducible quantization toolchain is one of the key variables determining whether ExecuTorch can be widely adopted on MCUs.