Core ML

Apple Intelligence是苹果在WWDC 2024上发布的系统级AI框架

The Apple Core ML framework allows developers to convert models trained in PyTorch or TensorFlow to the .mlmodel format for local execution, leveraging parallel computation across the GPU and Neural Engine.

在Apple Silicon设备上，借助Core ML转换工具和ANE加速，Whisper medium模型可实现接近实时的转录速度

Apple has simultaneous control over chip design, the operating system, and application frameworks (such as Core ML), enabling full-stack optimization from model compression to hardware scheduling.

Apple introduced an enhanced Speech framework in macOS 14, along with optimized Core ML conversions for the Whisper model.

Apple's Metal Performance Shaders and Core ML frameworks can offload inference to the GPU and neural engine, reducing CPU usage.

Running the Whisper model on iOS devices requires converting the model to Core ML or using inference frameworks such as whisper.cpp, and leveraging the Apple Neural Engine for acceleration.

Achieving 13-second transcription of one hour of audio on a MacBook Air usually involves technical approaches such as model selection (the Whisper series or optimized versions), inference acceleration (Metal acceleration, Core ML conversion, etc.) and precision trade-offs (quantization such as int8, int4).

ml-stable-diffusion is Apple's officially open-sourced conversion tool library, used to convert PyTorch weights of the Stable Diffusion series into Core ML format, enabling models to invoke the iPhone's neural engine for inference.

Anvil's image generation feature is called Anvil Dream, an SDXL Lightning variant converted to Core ML format via Apple's ml-stable-diffusion, generating 1024x1024 images on the neural engine.

The test audio processing path is the same as microphone capture: FLAC is first decoded via AVAudioFile, then resampled with AVAudioConverter to 16 kHz mono Float32, and the audio is read into memory in advance so file I/O is not counted in the timing; only the transcribe stage is timed, and model download and Core ML preparation are outside the timing region.

The iOS export uses Core ML 8-bit weights, with a model size of 26MB and 99.96% agreement with fp32 labels, and in the keyboard it actually uses the CPU compute unit, taking about 4ms per sentence in a release build; the Android version uses ONNX opset 17 with int8 dynamic quantization, 27MB, 99.1% agreement, and likewise about 4ms per sentence on CPU.

One of the advantages of Apple's Neural Engine lies in its deep integration with the Core ML framework, enabling developers to more efficiently offload model inference to dedicated hardware.

The core technical approach for on-device AI typically relies on lightweight model frameworks such as ONNX Runtime, TensorFlow Lite, or Apple's Core ML.

Core ML allows developers to convert models trained in formats such as ONNX and PyTorch into a format that can perform efficient inference on Apple devices without requiring network requests.

The mainstream solutions for local inference include llama.cpp, Ollama, ONNX Runtime, and Apple's Core ML.

Common on-device model deployment solutions include frameworks such as Core ML and llama.cpp, which can compress model weights to anywhere from a few hundred MB to several GB.

The MLComputeUnits option provided by Apple only allows coarse-grained specification of the preferred compute backend and cannot force individual operators to run on the ANE.

When an operator in a model does not meet ANE constraints, the Core ML runtime falls back to executing that operator on the GPU or CPU, thereby consuming significant memory bandwidth and introducing synchronization latency.

The Core ML Instrument in Instruments can observe the execution of each layer in a model, but with limited granularity.