[KongchangAI]
· 2 min read· 1,297 words

MNN-Pocket-0.3B Technical Report: 173MB On-Device Model Achieves 466 tokens/s Inference

MNN-Pocket-0.3B Technical Report: 173MB On-Device Model Achieves 466 tokens/s Inference

Alibaba's MNN team open-sourced a 300M-parameter on-device model at only 173MB in INT4, hitting 466 tokens/s on iPad Pro M5.

Alibaba's MNN team released the MNN-Pocket-0.3B technical report, showing a complete engineering path for on-device large models. It's a 300M-parameter dense model that, after INT4 quantization, deploys at just 173MB, supports up to 32K-token context, and reaches 466 tokens/s decoding on the iPad Pro M5 Metal backend. The methodology emphasizes co-design of architecture, data, training pipeline, and inference—including hardware-aware selection and dependency-aware vocabulary pruning. Pre-training covers ~4.1T Chinese/English tokens; post-training has three stages: instruction fine-tuning, multi-teacher on-policy distillation, and QAT. It claims state-of-the-art results at its size, even beating some 0.5B models on instruction following; weights and MNN package are open-sourced.

The biggest obstacle to deploying large models on-device has always been memory and compute. The MNN-Pocket-0.3B technical report released by Alibaba's MNN team offers a valuable approach: using a 300M-parameter dense model, compressing the deployment footprint to 173MB with INT4 quantization, achieving up to 466 tokens/s decoding throughput on an iPad Pro M5 (Metal), while supporting contexts up to 32K tokens. Both the model weights and the MNN deployment package are open-sourced.

MNN-Pocket-0.3B Technical Report Overview

Co-Design of Architecture and Data

The core methodology of MNN-Pocket-0.3B is the co-design of architecture, data, training pipeline, and inference system, rather than training a model first and then figuring out how to fit it onto a device.

At the architecture level, the team adopted a hardware-aware selection strategy, deriving the model structure from the actual memory and compute constraints of target devices. One notable technical point is dependency-aware vocabulary pruning — vocabulary size directly determines the parameter count of the embedding and output layers, which has an especially significant impact on on-device models. The pruning accounts for dependencies between tokens, preserving model capability as much as possible while compressing size.

This "derive model design from deployment constraints" approach stands in sharp contrast to the cloud-model path of "build big first, then distill," and is much closer to the real needs of on-device scenarios.

The core idea of dependency-aware vocabulary pruning: in multilingual or large-vocabulary models, the parameter count of the embedding and output layers is proportional to vocabulary size, and these two layers can account for a considerable proportion of a 300M-parameter model. If pruning simply removes low-frequency words by frequency, it easily breaks semantically tightly-linked token clusters (for example, combinatorial dependencies among digits, punctuation, and subword fragments). The dependency-aware method first counts co-occurrence and substitution relationships of tokens in the corpus, keeps or prunes strongly dependent tokens as a whole, and then remaps vocabulary indices, thereby compressing the embedding matrix while reducing the impact on downstream tasks. For on-device models, reducing the vocabulary from the hundred-thousand level to the tens-of-thousands level can often directly save tens of MB of weights — a highly cost-effective compression technique.

Four-Stage Pre-Training: 4.1 Trillion Tokens

On the pre-training side, MNN-Pocket-0.3B went through a four-stage process covering about 4.1 trillion high-quality Chinese and English tokens. For a model at the 300M-parameter level, this data scale is quite substantial — training a small model with a large volume of data is currently the mainstream approach to improving small-model capability.

The report states that the model thereby acquired broad knowledge and reasoning abilities. Judging from subsequent evaluation results, Chinese and English knowledge and reasoning are indeed its relatively standout areas, which should be directly related to the mix design of the pre-training data.

Three-Stage Post-Training and Quantization Awareness

After pre-training comes a three-stage post-training pipeline:

  1. Instruction fine-tuning: aligning instruction-following ability, which is the foundation of on-device assistant-type applications.
  2. Multi-teacher on-policy distillation: using multiple stronger teacher models to guide the student model, in an on-policy manner so that the student receives corrections on trajectories it generates itself, which is typically more effective than offline distillation.
  3. Quantization-aware training (QAT): directly preparing for INT4 deployment. For the 173MB INT4 deployment footprint to be feasible, the QAT step is indispensable.

One detail worth noting: quantization is not a remedial measure before deployment, but a formal part of the training pipeline. This "consider quantization during training" approach is becoming standard practice for on-device models.

Multi-teacher on-policy distillation differs from traditional offline distillation in the source of data: offline distillation typically uses answers generated by teacher models on a fixed dataset as supervision signals, and the student merely imitates; on-policy distillation has the student model first generate its own responses to prompts, and then multiple teacher models provide token-by-token probability distributions or preference corrections on these "student trajectories," so the student can correct the mistakes it would actually make, rather than only learning the teacher's ideal-state outputs. Running multiple teachers in parallel can also fuse the strengths of different teachers across different capability dimensions (such as math, code, and dialogue), mitigating the problem of a single teacher's preferences being amplified. The cost is the need to call teacher models online, which noticeably raises training infrastructure and cost.

Hardware-Level Optimization of the Inference Engine

Beyond the model-side work, the MNN inference engine is optimized for specific hardware, achieving fast, memory-efficient inference and long-context handling. The 466 tokens/s decoding speed is a concrete number measured on the Metal backend of an iPad Pro M5, showing that these optimizations are not just generalities.

The combination of a 32K context and a 173MB deployment footprint means that memory-constrained devices such as smart glasses and phones can also handle scenarios like long-document processing and multi-turn dialogue. The report explicitly lists smartphones and smart glasses as target devices.

The key difference between quantization-aware training (QAT) and post-training quantization (PTQ) is that QAT simulates INT4 rounding and truncation errors during the training forward pass, letting weights adapt to low-precision representation as they are updated; backpropagation typically uses a straight-through estimator (STE) to bypass the non-differentiable quantization operators. Compared with PTQ, which directly quantizes already-trained weights, QAT can significantly reduce accuracy loss at 4-bit, which is especially important for small models like the 300M-parameter one whose capacity is already tight. The cost is a more complex training pipeline: quantization ranges need to be calibrated, outliers handled, and scheduling coordinated with the distillation stage — which is why on-device models incorporate quantization into the formal training pipeline.

Evaluation Performance: State of the Art Among Small Models

The report states that MNN-Pocket-0.3B reaches state-of-the-art levels among models of the same size, performing especially well in Chinese and English knowledge and reasoning, mathematics, programming, and instruction following. It consistently outperforms open-source models of the same scale, and on instruction-following benchmarks even surpasses some 0.5B-parameter models.

Smaller parameters yet better performance — behind this is the efficiency advantage brought by co-design: fewer parameters for equal capability means less memory usage and faster inference, which is exactly the trade-off most valued in on-device scenarios.

Open Source and Deployment

The team released MNN-Pocket-0.3B as open weights and provides an out-of-the-box MNN deployment package. For application developers, this means they can integrate directly on-device without having to carry out model conversion and quantization themselves.

Judging from the information disclosed in the technical report, the value of this model lies not in topping some leaderboard, but in demonstrating a complete engineering path for on-device large models: hardware-aware design, large-scale small-model training, distillation and quantization-aware post-training, and inference engine optimization — four links that interlock. For teams hoping to fit language models into phones or smart glasses, this report itself may be more instructive than the model weights.

One-line Summary

Alibaba's MNN team open-sourced a 300M-parameter on-device model that is only 173MB in INT4 and achieves 466 tokens/s decoding on an iPad Pro M5.

Paragraph Summary

Alibaba's MNN team released the MNN-Pocket-0.3B technical report, demonstrating a complete engineering path for on-device large models. The model is a 300M-parameter dense language model that, after INT4 quantization, has a deployment footprint of only 173MB, supports contexts up to 32K tokens, and achieves up to 466 tokens/s decoding throughput on the Metal backend of an iPad Pro M5. Methodologically, it emphasizes co-design of architecture, data, training pipeline, and inference system, including hardware-aware selection and dependency-aware vocabulary pruning; pre-training covers about 4.1 trillion Chinese and English tokens, and post-training consists of three stages: instruction fine-tuning, multi-teacher on-policy distillation, and quantization-aware training. Evaluations claim it reaches state-of-the-art levels among models of the same size in Chinese and English knowledge reasoning, mathematics, programming, and instruction following, even surpassing some 0.5B models on instruction-following benchmarks. Both the weights and the MNN deployment package are open-sourced.

#MNN-Pocket-0.3B#on-device model#INT4 quantization#quantization-aware training#multi-teacher distillation#edge AI inference
Share:

Related articles