ExecuTorch

ExecuTorch supports Arm Cortex-M microcontrollers, allowing PyTorch models to run on MCUs without conversion through intermediate formats such as ONNX.

ExecuTorch solves the "deployment path" problem, not the "insufficient compute" problem; model size, operator support scope and inference latency are still limited by the chip itself.

Through memory planning, ExecuTorch computes the lifetime and offset of each tensor before execution, letting intermediate results reuse the same static buffer and avoiding runtime malloc/free.

ExecuTorch is the on-device inference runtime officially launched by PyTorch in 2023, covering phones, embedded devices and even MCUs.

ExecuTorch introduces a unified intermediate representation, EXIR; models are exported to EXIR via torch.export, then packaged in the .pte format and handed to the runtime for execution.

ExecuTorch adopts a layered runtime design, including the representation layer, the operator layer and the backend layer (Backend Delegate).

Google's MediaPipe LLM Inference, Meta's ExecuTorch, and llama.cpp with Android support are mature mobile inference frameworks capable of mapping model computations to a phone's GPU or NPU for execution.

On a structured output task involving a store audit checklist, ExecuTorch (Llama 3.2 1B, schema stuffed into the prompt) produced only 2 valid outputs out of 20 calls, while Apple Foundation Models (guided generation) produced valid outputs for all 20 calls.

Apple Foundation Models constrains model output at the framework level via guided generation to necessarily conform to a given type structure, whereas ExecuTorch has no corresponding capability and can only write the schema into the prompt.

ExecuTorch 1B on a simulator requires a 1.14 GB download (6 minutes 12 seconds), approximately 5 seconds to load on each launch, approximately 907 MB of in-app memory during generation, a generation speed of 43 tokens/s, and a median time-to-first-token of 579 ms for subsequent calls.

Do not trust availability flags self-reported by engines; make one real single-token probe call and cache the result.

React Native 0.87 has removed RCTCxxBridge, requiring a patch to use RCTTurboModuleWithJSIBindings for injecting JSI bindings before ExecuTorch-related builds can compile.

ExecuTorch 0.10 reports anonymous download analytics by default, and setTelemetryEnabled(false) must be called to disable it.

The core design goal of ExecuTorch is to export trained PyTorch models into a format that can run efficiently on resource-constrained devices, covering scenarios such as smartphones, microcontrollers, and even wearable devices.

The ExecuTorch workflow consists of three phases: exporting the model computation graph with torch.export, performing quantization and optimization through compiler passes, and finally packaging into the .pte format to run on the target device.

The discussion is intended to cover three groups of people: those new to ExecuTorch, experienced practitioners, and contributors who aspire to create learning content.

After successfully running through the official ExecuTorch examples and tutorials, developers often find it difficult to smoothly transition to real-world application development.

Every link in the ExecuTorch model deployment pipeline may encounter issues such as unsupported operators, precision loss, or insufficient performance.

As a core supplier of edge computing chips, Arm has an incentive to participate in advancing the ExecuTorch learning ecosystem in order to lower the barrier for developers deploying AI models on its platform.

ExecuTorch is an on-device inference runtime framework launched by Meta, built on the Export mechanism of PyTorch 2.0.