Maple-Preview: 20B Ternary MoE Model Achieves 120 tok/s On-Device Inference on iPhone

When Large Models Meet Smartphones: A Counter-Intuitive Technical Demo

On Hacker News, a project called Maple-Preview has caught the attention of the tech community. Its core value proposition directly addresses the pain points of current large model deployment: a Mixture of Experts (MoE) model with 20 billion (20B) parameters, using ternary quantization, running on an iPhone at 120 tokens per second (120 tok/s).

For those familiar with local large model deployment, these numbers together create an inherent tension. A 20B-level model typically requires tens of gigabytes of VRAM or memory, while smartphones have extremely limited resources. Running a model of this scale on a mobile device at 120 tok/s inference speed (far exceeding human reading speed) relies on a combination of several key technologies.

Breaking Down Three Core Technologies: Ternary Quantization, MoE, and On-Device Inference

Ternary Quantization: Compressing Weights to the Extreme

Traditional large model weights are typically stored in 16-bit (FP16) or 32-bit floating point numbers. Even after INT8/INT4 quantization, each parameter still requires at least 4 bits. Ternary quantization means each weight has only three possible states—typically -1, 0, and +1.

This approach is closely related to the widely-discussed BitNet and 1.58-bit LLM research in recent years. The theoretical foundation of ternary quantization can be traced back to the BitNet paper published by Microsoft Research in 2023, followed by BitNet b1.58 which further demonstrated the feasibility of 1.58-bit (i.e., ternary) quantization for large language models. Unlike traditional post-training quantization methods (such as GPTQ and AWQ), ternary quantization typically requires Quantization-Aware Training (QAT) during the training phase, allowing the model to learn to adapt to extremely low-bit weight representations. From an information theory perspective, the entropy of three states is log₂(3) ≈ 1.58 bits, which is why ternary quantization is also called 1.58-bit quantization. This extreme compression is feasible partly because large model weight distributions are inherently sparse—many weight values are close to zero and can be safely quantized to 0 without seriously degrading model output quality.

Ternary representation theoretically requires only about 1.58 bits per parameter, achieving nearly 10x storage compression compared to FP16. This is the prerequisite for fitting a 20B model into phone memory: roughly estimated with ternary encoding, 20B parameters require only about 4GB of storage, falling within the acceptable range for high-end iPhones.

More importantly, ternary weights allow matrix multiplication to be replaced with addition and subtraction instead of floating-point multiplication, dramatically reducing computational energy consumption—critical for battery-powered, thermally-constrained mobile devices.

MoE Architecture: Saving Compute Through Sparse Activation

The second key concept is MoE (Mixture of Experts). 20B is the model's total parameter count, but the essence of MoE lies in "sparse activation": each inference only activates a small subset of expert networks rather than engaging all parameters.

MoE architecture is not a new concept—it was proposed by Jacobs et al. back in 1991—but it truly came to life in the large model era starting with Google's Switch Transformer (2021). Subsequently, Mixtral 8x7B (released by Mistral AI) became one of the most influential MoE models in the open-source community, demonstrating that MoE can significantly reduce inference compute costs while maintaining high-quality outputs. MoE models typically include a Router/Gate network that dynamically determines which experts to activate at each layer for each token. A typical configuration like "top-2 out of 8" means only 2 out of 8 experts are activated each time, resulting in actual computation of roughly one-quarter that of a dense model.

This means that although the total model size is 20B, the active parameters actually invoked in a single forward pass may be only a few billion. This explains why it can achieve 120 tok/s throughput on a compute-limited phone—the actual inference workload is far smaller than its nominal scale. However, MoE also introduces engineering challenges: all expert parameters need to reside in memory (even when not activated), placing higher demands on memory bandwidth and capacity. This is precisely why ternary quantization is particularly important for MoE models—it dramatically reduces the total parameter volume that must remain in memory. The combination of MoE and ternary quantization is essentially a dual optimization: "trading storage for compute, then trading sparsity for storage."

120 tok/s: What Mobile Inference Speed Really Means

120 tok/s is a remarkably impressive speed. For reference, comfortable human reading speed is approximately 3-5 words per second. This throughput means the model can generate fluent long-form text in near real-time, providing practical feasibility for on-device AI applications (such as offline assistants and privacy-sensitive scenarios).

To understand why this speed is noteworthy on mobile devices, one needs to understand iPhone hardware capabilities. Apple introduced the Neural Engine starting with the A11 Bionic chip, and by the A17 Pro and M-series chips, NPU performance has reached 35+ TOPS (trillion operations per second). Apple's Unified Memory Architecture (UMA) means CPU, GPU, and NPU share the same memory pool, eliminating data transfer overhead—extremely advantageous for large model inference. However, the bottleneck for on-device inference is often not peak compute but memory bandwidth—the iPhone 15 Pro has approximately 100 GB/s memory bandwidth, far below the multi-TB/s bandwidth of data center GPUs. This means that even if the model fits in memory, bandwidth can severely limit token generation speed. Ternary quantization, by compressing each parameter to 1.58 bits, dramatically reduces the amount of data that needs to be read from memory per inference, thereby minimizing the impact of this core bandwidth bottleneck—this is the key engineering logic behind achieving 120 tok/s.

The Value and Use Cases of On-Device AI Deployment

Why go through all this effort to squeeze large models into phones? The answer lies in the unique value of on-device inference:

• Privacy protection: Data never leaves the device—all computation happens locally, which is critically important for sensitive scenarios in healthcare, finance, and more.
• Offline availability: AI capabilities work without network access, eliminating dependence on servers and bandwidth.
• Cost and latency: Eliminates cloud API call costs while removing network round-trip latency.

On-device AI deployment has become a competitive focal point for tech giants. Apple announced Apple Intelligence at WWDC 2024, emphasizing a device-first AI strategy; Google launched Gemini Nano specifically designed for mobile; Qualcomm's Snapdragon 8 Gen 3 and MediaTek's Dimensity 9300 both feature NPUs specifically optimized for on-device large model inference. In the open-source community, the llama.cpp project has made local inference of quantized models possible across various hardware, while MLC-LLM focuses on mobile compilation optimization. The core challenge of this race is: how to deliver AI experiences as close to cloud-quality as possible on mobile chips that consume one-thousandth the power of data center GPUs.

Maple-Preview's demo, to some extent, validates the feasibility of the "large parameter model + extreme quantization + sparse architecture" technical approach on mobile devices. It's not simply cramming a small model into a phone, but attempting to preserve the capability ceiling of large models on mobile.

Maintaining a Prudent Perspective

As a "Show HN" preview project, we should maintain a degree of caution. At the time of discussion, the post had only 12 upvotes and 4 comments, with no large-scale community validation yet. Several questions deserve attention:

First, model quality and precision loss. While ternary quantization offers impressive compression ratios, extreme quantization typically introduces accuracy degradation. How this 20B ternary model performs on real-world tasks, and whether it can match full-precision models of equivalent scale, requires more benchmark testing to prove. Academic research shows that while quantization-aware training can partially compensate for precision loss, perceivable gaps still exist between extremely low-bit models and full-precision models on complex reasoning, mathematical computation, and similar tasks. Whether the model has been specifically optimized for certain tasks, and where its general capability boundaries lie, are critical dimensions for evaluating its practical value.

Second, speed measurement conditions. At what prompt length, on which iPhone model, and under what battery and thermal conditions was 120 tok/s measured? Whether this speed can be sustained in long-context scenarios is an unavoidable practical question for on-device deployment. Mobile devices trigger thermal throttling under sustained high loads, causing chip frequency reduction and potentially significant drops in inference speed. Additionally, generation-phase speed is typically much higher than the prefill phase (processing prompt)—if 120 tok/s refers only to peak decoding speed, the actual user experience may differ from expectations.

Third, reproducibility and open-source availability. As a preview version, its degree of open-sourcing, usability, and whether it can be easily verified by third parties will determine whether this is a solid engineering achievement or a demo under ideal conditions.

Running Large Models on Small Devices: The Technology Trend of On-Device AI

Regardless of Maple-Preview's ultimate maturity, the direction it represents is clear: ternary/low-bit quantization + MoE sparse architecture is becoming the mainstream technology combination for on-device large models. From BitNet to various mobile inference frameworks, the industry is systematically tackling the challenge of "running large models on small devices."

Behind this trend is the convergence of multiple forces: chip manufacturers' continued investment in NPU compute, academic breakthroughs in ultra-low-bit quantization theory, and users' growing demand for privacy and offline AI capabilities. In the future, we may see more model architectures "designed for on-device from the start"—not training large models first and then compressing them, but considering low-bit, sparse activation, and limited memory bandwidth constraints from the very beginning of architecture design.

When a phone can locally run a 20B-level model, the form and boundaries of AI applications will be redefined. These seemingly "counter-intuitive" technical demonstrations are often harbingers of industry transformation.

Key Takeaways