Dual On-Device Models on iPhone: How a Chat Model and an Image Model Coexist

Packing both a large language model and an image generation model into a single iPhone — the biggest enemy isn't compute power, it's memory. Developer Nathan shared his experience building Anvil on the DEV community, a fully on-device private AI assistant for iPhone, attempting to answer an increasingly practical question: just how far can on-device AI go?

Two Models, One Phone

Anvil's architecture is straightforward: the chat side runs Gemma 4 E2B via LiteRT LM, while the image side is Anvil Dream — an SDXL Lightning variant converted to Core ML format through Apple's ml-stable-diffusion, generating 1024x1024 images on the Neural Engine.

Neither model is weak on its own, but the problem is they can't reside in memory simultaneously. Nathan's solution is runtime switching: dynamically load the corresponding model based on the user's current action, and unload it when done. This is an engineering compromise that on-device AI applications can't avoid.

iPhone's unified memory architecture performs well when running a single model, but the combined weights and intermediate activations of two models quickly eat up available memory. Nathan notes this is also why Anvil requires an iPhone 15 Pro or newer — older devices simply can't handle the memory demands.

LiteRT LM is Google's on-device large model inference runtime, formerly known as TensorFlow Lite (TFLite), later renamed and extended by the Google AI Edge team to support generative AI workloads. It applies operator-level optimizations for mobile CPUs, GPUs, and NPUs, supports weight quantization (int8, int4) to compress model size, and provides inference acceleration mechanisms such as streaming output and KV Cache reuse. Compared to calling cloud APIs directly, LiteRT LM lets developers complete token generation locally on-device, avoiding network round-trip latency and data exposure. Its limitations stem from phone memory bandwidth and thermal constraints — usable model sizes and context lengths are far smaller than cloud deployments.

ml-stable-diffusion is Apple's official open-source conversion tool library, used to convert PyTorch weights from the Stable Diffusion series into Core ML format, enabling models to invoke the iPhone's Neural Engine for inference. The conversion process involves operator mapping, weight precision selection (such as float16), and model partitioning to fit the Neural Engine's memory constraints. The converted Core ML models integrate deeply with Swift applications, leveraging Apple Silicon's unified memory architecture to reduce data copy overhead. This is the key prerequisite that enables SDXL Lightning to generate 1024x1024 images on iPhone.

Privacy-First Design Choices

Anvil's positioning is clear: a private assistant that doesn't depend on the cloud. After downloading models over WiFi for the first time, chat, voice, note processing, task management, photo and document operations all happen locally on the phone. There's no account system, and web search and iCloud backup are both optional and kept isolated from on-device chat data.

This design sacrifices some convenience — for example, it can't call GPT-4-level cloud models — but in exchange, you get the certainty that data never leaves the device. For scenarios involving sensitive information, this trade-off may well be worth it.

That said, on-device small models still show a clear gap compared to cloud large models on complex reasoning tasks. Gemma 4 E2B is sufficient for lightweight conversation and instruction following, but its capability boundaries become apparent when facing multi-step logical reasoning or long-context understanding.

The E2B in Gemma 4 E2B stands for Effective 2 Billion, meaning an effective parameter count of approximately 2 billion. This naming comes from Google's labeling approach for the Gemma series of small models, emphasizing the model's equivalent scale under specific inference configurations — actual parameters may be slightly higher depending on architecture design. The 2-billion-parameter level is on the larger side for mobile deployment, typically requiring int4 quantization before deployment, with a file size of about 1-2 GB and even higher memory usage at runtime. Such models are suitable for short conversations, summarization, and simple instruction following, but on tasks requiring multi-step reasoning chains or precise factual recall, the gap with 70B-level cloud models is significant.

The Real Constraints of On-Device AI

Nathan actively solicited tester feedback on speed, battery life, and usability in his post — which itself illustrates the core challenges facing on-device AI.

Memory is a hard constraint. The inability for two models to coexist means users will experience loading delays when switching between features. This experience barely exists in cloud solutions, but on-device it's a physical limitation.

Heat and battery life are hidden costs. Continuous Neural Engine inference generates heat, and battery consumption far exceeds ordinary apps. For users who need to use AI features frequently, this could become a daily-use obstacle.

Model updates are an operational challenge. Cloud models can be iterated and upgraded at any time, while on-device models require users to re-download hundreds of MB or even GB-scale weight files.

Unified Memory Architecture (UMA) is one of the core designs of Apple Silicon: the CPU, GPU, and Neural Engine share the same physical memory pool, eliminating the need for explicit data copies between chips. This architecture shows clear advantages when running a single model, reducing data transfer latency and redundant memory usage. But when two large models reside simultaneously, they share the same memory ceiling — the iPhone 15 Pro's 8 GB must accommodate the system, apps, model weights, and intermediate activations all at once, leaving perhaps only 3-4 GB for AI models in practice. This is the fundamental reason Anvil must switch at runtime rather than load simultaneously, and the fundamental bottleneck preventing older iPhones (6 GB or less) from running it.

Where Does On-Device AI Fit?

Anvil's attempt reveals a trend that's taking shape: on-device AI isn't meant to replace cloud AI, but to offer a different option in specific scenarios.

Scenarios suited for on-device include: personal data processing that's extremely privacy-sensitive, environments with unstable or unavailable networks, and simple interactions requiring low-latency responses. Unsuitable scenarios include: complex reasoning, large-scale knowledge retrieval, and applications requiring frequent model capability updates.

From an engineering perspective, the pattern Anvil demonstrates — multiple specialized small models switched on demand — may be more pragmatic than trying to run an all-in-one large model on a phone. Gemma handles text understanding and generation, SDXL Lightning handles images, each doing its part, with a scheduling layer coordinating resources.

The cost of this architecture is fragmentation in user experience, but what you get in return is the ability to provide multiple AI capabilities within phone memory limits. For developers wanting to build on-device AI applications, Anvil's practice offers a sample worth referencing.

Anvil is currently in public testing on TestFlight, and beta testers can use Anvil Pro for free.

One-line Summary

Developer Nathan shares Anvil's approach: running Gemma and SDXL dual models on iPhone via runtime switching to build a fully local private AI assistant.

Paragraph Summary

Anvil is a private AI assistant running entirely on iPhone, using Gemma 4 E2B with LiteRT LM for chat and an SDXL Lightning variant converted to Core ML via ml-stable-diffusion for images. Since the two models can't reside in memory simultaneously, the developer adopted a runtime on-demand loading and unloading strategy, with iPhone 15 Pro as the minimum requirement. The app has no account system — chat, voice, notes, and photo processing all happen locally, with web search and iCloud backup optional and data-isolated. The article also reveals three real constraints facing on-device AI: memory is a hard limit, heat and battery life are hidden costs, and model updates are an operational challenge. The conclusion is that on-device AI doesn't replace the cloud, but offers a differentiated choice in privacy-sensitive, offline, and low-latency simple interaction scenarios.