Apple Neural Engine

The Neural Engine in Apple's custom chips is an NPU optimized specifically for matrix multiplication and convolution operations, and offers higher energy efficiency than general-purpose CPUs and GPUs on specific AI inference tasks.

苹果M系列芯片中的Neural Engine本质上属于NPU

The ANE is essentially a dedicated accelerator optimized for convolution and matrix operations.

Apple does not guarantee the stability of any low-level ANE interfaces, and the implementation details of the ANE may change substantially across chip generations.

Running the Whisper model on iOS devices requires converting the model to Core ML or using inference frameworks such as whisper.cpp, and leveraging the Apple Neural Engine for acceleration.

ExecuTorch places particular emphasis on extremely low memory footprint, predictable latency, and the ability to adapt to diverse hardware backends such as Arm CPU/GPU, Apple Neural Engine, and Qualcomm DSP.

Through reverse engineering communities (such as the Asahi Linux team and researchers like @dougallj), it is known that the ANE prefers specific tensor dimension multiples (e.g., channel counts must be multiples of 16 or 64) as well as memory layout formats such as NHWC.

When an operator in a model does not meet ANE constraints, the Core ML runtime falls back to executing that operator on the GPU or CPU, thereby consuming significant memory bandwidth and introducing synchronization latency.

Apple has never publicly disclosed the full specifications of the ANE.

The core optimization approaches for reclaiming ANE bandwidth include aligning data layouts, operator fusion and selection, and reducing memory round-trips.

When deploying models on Apple platforms, one should not focus solely on floating-point compute (FLOPS) metrics but also pay attention to memory bandwidth as a key performance determinant.

Through appropriate model architecture design and data layout optimization, it is possible to recover approximately 50 GB/s of effective memory bandwidth under specific workloads.

The ANE has strict requirements on data layout and typically employs specific tiling strategies to maximize on-chip cache utilization.

Unlike the general programmability of GPUs, the ANE more closely resembles a fixed-function dataflow engine—extremely efficient for supported operators, but requiring fallback to the CPU or GPU for unsupported operations.

The ANE's "all-or-nothing" characteristic is the fundamental reason why many models cannot fully utilize the ANE.

Choosing ANE-friendly operator combinations, prioritizing half-precision floating point, and avoiding frequent data transfers between devices can significantly affect inference latency and power consumption.

Apple has consistently lacked publicly available technical documentation for the ANE.

Reverse engineering results are better suited as a reference for understanding and optimization, rather than as a foundation for building production dependencies.

The ANE of the A17 Pro chip delivers 35 TOPS of computing power, with power consumption approximately 1/10 that of a GPU performing the same task.

Based on Apple's consistent strategy, the likelihood of fully opening ANE documentation in the near term is low.