MiniCPM-V 4.7 35B-A3B: The First Open-Source MoE Multimodal On-Device Model

OpenBMB has released MiniCPM-V 4.7, the newest member of the MiniCPM-V family, introducing its first MoE architecture variant — 35B-A3B. With this model, running MoE large models on phones has moved from proof-of-concept to a new stage.

Why On-Device MoE Deployment Has Always Been Hard

MoE (Mixture of Experts) architectures offer the advantage of stronger capabilities with fewer activated parameters, but their demands on memory and scheduling make on-device deployment tricky. The dynamic nature of expert routing means weights must switch frequently during inference, which easily causes stuttering or slowdowns on resource-constrained mobile devices.

MiniCPM-V 4.7 35B-A3B offers a systematic solution to this tension. Officially described as "the first open-source MoE model to achieve lossless, smooth on-device inference," its core support comes from a combination of techniques: Lookahead Expert Scheduling (LES), fine-grained layer-wise quantization, and DSpark speculative sampling integration. Together, these achieve a decoding speed of 13+ tokens/s on iPhone 17 Pro Max — an impressive figure for a 35B-class MoE model.

MoE (Mixture of Experts) is a sparse architecture that splits a single dense network into multiple "expert" sub-networks. During inference, each token passes only through a small number of experts selected by a router, so the total parameter count can be very large while the parameters actually activated per forward pass (i.e., activated parameters, such as the 3B in 35B-A3B) remain small. This "large capacity, low activation" property is theoretically ideal for on-device use — but the cost is that all weights must still reside in memory, and routing results change dynamically token by token, so the expert weights to load differ at each step, making memory access randomized and prone to bandwidth bottlenecks and scheduling jitter.

A New Answer to the Performance-Efficiency Trade-off

The arms race in model capabilities has been going on for a long time, but the evaluation criteria for on-device scenarios are shifting. On resource-constrained devices, simply piling on parameters is meaningless; what truly matters is "capability density" — how much capability can be released within a limited compute budget.

In Instruct mode, the 35B-A3B variant achieves a higher overall score than Qwen3.6 35B-A3B with less inference time. The official claim is that it "advances the performance-efficiency Pareto frontier of multimodal large models."

What's worth noting about this comparison isn't who's higher or lower, but the trend it reveals: competition in on-device multimodal models has shifted from "can it be done" to "is it cost-effective enough."

Naming like "35B-A3B" refers to a total parameter count of 35B with roughly 3B activated per token. The "A" stands for Activated, measuring the parameter scale actually involved in computation during a single inference. The opposite is a dense model — for example, a 35B dense model must use all 35B parameters to generate each token. Thus an A3B model's per-step computation is close to that of a 3B small model, yet thanks to its sparse expert pool, its expressive capacity can still approach the 35B level. This is also why the core challenge of on-device MoE isn't "compute" but "memory access" — what determines speed is often the bandwidth for feeding expert weights into compute units, not the floating-point operations of the matrix multiplications themselves.

Architectural Optimization in Visual Encoding

On the vision side, MiniCPM-V 4.7 adopts early compression inside the ViT from LLaVA-UHD v4. Simply put, visual tokens are compressed early within the encoder, reducing the computation of subsequent layers.

More practically, it supports 4x and 16x mixed visual token compression. Developers can switch flexibly based on task needs — using a low compression ratio to preserve detail when fine-grained recognition is required, or a high compression ratio to reduce cost when speed is the priority. This adjustability is far friendlier in real deployment scenarios than a fixed compression strategy.

The model continues the MiniCPM-V family's single-image, multi-image, and video understanding capabilities, with foundational abilities enhanced over the previous generation.

ViT (Vision Transformer) is a visual encoder that slices images into patches and models them as sequences; the resulting string of visual tokens is fed into the language model. The higher the resolution, the more patches, and the longer the visual token sequence — subsequent attention and MLP computation grows quadratically or linearly with sequence length, becoming the main overhead source for on-device multimodality. So-called early compression inside the ViT means merging or discarding redundant tokens at shallow layers of the ViT rather than waiting for the encoder to output all tokens before filtering, letting deeper layers compute directly on a shorter sequence. A 4x/16x compression ratio refers to the factor by which visual tokens are compressed — the higher the factor, the more compute saved, but the greater the loss of fine-grained information.

Platform and Chip Coverage

An on-device model's deployment capability largely depends on ecosystem coverage. MiniCPM-V 4.7 supports the three major mobile platforms — iOS, Android, and HarmonyOS — while also offering open-source on-device deployment solutions.

At the chip level, it provides day-0 support for Rockchip, Houmo, Intel, AMD, and Arm platforms. Day-0 support means these chip platforms can run the model on launch day, without waiting for adaptation. For teams building on-device products, this detail saves considerable engineering time.

Developer Toolchain

For inference frameworks, MiniCPM-V 4.7 is compatible with SGLang, vLLM, llama.cpp, and Ollama. Fine-tuning frameworks support SWIFT and LLaMA-Factory. Quantization formats cover GGUF, BNB, AWQ, and GPTQ, with newly added FP8 quantization support. The GGUF version has already been uploaded to Hugging Face for the community to download and use directly.

Officially, a Self-Replay Fine-Tuning method has also been released, aimed at expanding model capabilities at low cost while reducing forgetting of original abilities. This problem is common in fine-tuning practice — the model learns a new task, and performance on old tasks drops. Self-Replay's approach is to let the model replay its existing capabilities during fine-tuning, thereby maintaining stability.

The Next Step for On-Device Multimodality

The signal from MiniCPM-V 4.7 35B-A3B is fairly clear: on-device multimodal models are shifting from the "small parameters, good enough" mindset toward a "large-parameter MoE + system-level optimization" path. The 13+ tokens/s phone-side decoding speed, multi-chip day-0 support, and mixed compression strategy together form a complete solution aimed at real-world deployment.

The next points of interest: whether this LES + quantization + speculative sampling combination can be reused by other models, and where exactly the efficiency ceiling for on-device MoE inference lies.

"Day-0 support" is a timeliness commitment in the open-source model ecosystem: chip vendors or framework providers complete adaptation on the same day the model is released, so developers can download and run it on the corresponding hardware without waiting weeks or even months for porting and tuning. Its significance for on-device deployment is transferring adaptation cost from each product team to the model-and-chip collaborators. Related to this is speculative sampling, where a small, fast draft model guesses multiple tokens in succession, then the large model verifies them all in parallel at once; accepted tokens are kept directly, turning serial decoding into semi-parallel decoding with almost no change in output quality — a common technique for boosting tokens/s on-device.