Gemma 4 and Gemini Nano: Deconstructing the Architecture of On-Device Intelligence

Google's strategy for on-device AI is becoming clear: rather than trying to force cloud-scale models into phones, it's pursuing architectural innovation so that smaller models can deliver stronger capabilities on the same hardware. In a recent technical talk, Sarah, engineering lead of the Gemma modeling team, systematically broke down the design thinking behind Gemma 4 and Gemini Nano—from per-layer embeddings to quantization strategies, from multimodal capabilities to the AI Core integration path.

From Gemma 1 to Gemma 4: More Capability, Controlled Size

Sarah reviewed the evolution of the Gemma series: Gemma 1 introduced 2B and 7B text-only models, which were already extremely small at the time; Gemma 2 added coding capabilities and expanded to 27B; Gemma 3 added multimodality while going down to the 270M parameter range. Each generation grew in capability, but the goal remained the same—fitting single-device hardware: phones, laptops, GPUs.

Gemma 4's product matrix reflects these trade-offs: E2B and E4B target phones, the 12B unified model targets laptops (with the vision encoder removed to simplify deployment), and larger sizes come in both fully dense and mixture-of-experts (MoE) forms. Sarah emphasized that E2B and E4B are "small enough to run well on a phone"—an engineering challenge the team is proud of.

Context length is a notable upgrade in Gemma 4: support for up to 128K, and up to 256K for larger models. However, she cautioned that ML Kit caps context length, so developers should confirm their budget before expecting to run the full 128K.

Multimodality: Vision, Audio, and Video Advancing on Three Fronts

On vision, Gemma 4 uses variable aspect ratio processing—no longer squashing images into awkward squares, but preserving original proportions. This change keeps text and objects from being distorted, and Sarah called it "a massive improvement in model capability," with OCR performance improving especially noticeably. The model can also read charts, tables, and graphs, and perform translation of visual cues. The vision token budget is adjustable, letting developers balance high-resolution understanding against compute savings.

Audio is a newer modality, first appearing in Gemma 3n and significantly expanded in Gemma 4. It exists only in the smallest models (E2B, E4B, and 12B), supports 35+ languages out of the box, with pretraining coverage of 140+ languages—languages not on the list can achieve decent results through fine-tuning.

Video capabilities were introduced starting with Gemma 3n and further deepened in Gemma 4. Additionally, Gemma 4 adds agentic reasoning: planning, multi-step execution, tool calling, precise instruction following, and structured JSON output. Sarah particularly cautioned that if you integrate manually rather than using the ML Toolkit, be very careful with the chat template.

Benchmark Data: Small Models Punching Above Their Weight

The benchmark results Sarah presented were compelling: E2B comprehensively outperforms Gemma 3 27B—a much larger model—on reasoning, coding, and agentic tasks. On LiveCodeBench, Gemma 4 scores 44 versus Gemma 3 27B's 29; IEME and TAU2 (agentic evaluations) are likewise significantly higher. Moving up a few parameters, E4B consistently beats 27B on every benchmark.

What these numbers mean: from Gemma 3 to Gemma 4, capability density at the same parameter count has increased dramatically. And that is exactly the direction of progress on-device deployment needs most.

The Three Constraints of Mobile Hardware and Their Solutions

Sarah laid out the hardware gap between phones and servers bluntly: on memory, phones have 12–16GB versus roughly 80GB for servers; on bandwidth, phones have 50–80GB versus terabytes for servers; and power constraints are even stricter. Each technique she subsequently presented targets one of these constraints.

Per-Layer Embeddings (PLE): Where the "E" Comes From

Making a model smarter usually requires more parameters. Per-layer embeddings take a different approach: rather than piling parameters directly into the model body, they add an extra embedding table that is read at every layer. For a given input sequence, only the needed portion of the embeddings is loaded. The result—the nominal parameter count is far greater than 2B, but only about 2B is actually loaded into memory, which is what "effective 2B parameters" means. The embedding table typically resides in RAM rather than Flash, reducing both bandwidth pressure and memory footprint while retaining the capability of a larger model.

Per-Layer Embeddings (PLE) is a parameter-efficient architecture first introduced by Google in Gemma 3n. In a traditional Transformer, the embedding layer looks up the table only once at the input stage, whereas PLE additionally reads a shared embedding table at every layer—effectively giving each layer a lightweight "external memory." Because the embedding table is accessed sparsely—for a given token sequence, only the corresponding rows need to be loaded—it can substantially expand model capacity without adding dense compute. From an engineering perspective, PLE decouples "total parameters" from "active parameters": the nominal parameter count can far exceed 2B, but only about 2B actually resides in memory and participates in the forward pass. This is also the origin of the "E" (Effective) in the E2B and E4B names in Gemma 4.

Draft Models: The Speedup from Shared KV Cache

The draft model Gemma uses is multi-token prediction (MTP), a form of speculative decoding. A conventional autoregressive model produces one token at a time, whereas MTP predicts three to four tokens at once, which the main model then verifies—accepted ones are used directly.

The key difference from other draft models on the market: Gemma's draft model shares the base model's KV cache. Once E2B completes a full pre-fill and builds the cache, the draft model reuses it directly, eliminating the overhead of regenerating the pre-fill. The benefits Sarah cited: acceptance rates 40–60% higher than most draft models, 40% lower decoding latency, 30–40% lower energy consumption, and no code changes required.

Speculative Decoding is a technique for accelerating autoregressive generation without changing the output distribution. The core idea: use a smaller, faster "draft model" to guess multiple subsequent tokens at once, then have the main model verify those guesses in parallel; tokens accepted by the main model are used directly, while rejected ones are regenerated from the point of divergence. Because verification can be done in parallel, overall decoding latency drops significantly.

KV Cache refers to the key-value pairs of historical tokens stored during Transformer inference, avoiding recomputation of the entire context for each new token. In conventional speculative decoding setups, the draft model and main model each maintain separate KV caches, and the draft model needs its own pre-fill, incurring extra overhead. Gemma's MTP draft model directly reuses the KV cache already built by the main model, eliminating this redundant computation—which is the key reason its acceptance rate and latency gains outperform generic draft models.

Quantization: An Aggressive Path Tailored for Mobile Hardware

Quantization is not new, but the Gemma team did extra work specifically for mobile hardware. First is QAT (quantization-aware training), which accounts for quantization during training to help the model understand that parameters will be bucketed, thereby preserving quality—compressing from 16-bit to 4-bit while maintaining quality, with considerable memory savings.

E2B offers two quantization formats. Q4O is the general-purpose one, customizable and compatible with other quantization libraries; mobile quant is far more aggressive, and the team invested substantial research to make it optimal. It includes several targeted designs: static activation quantization eliminates the runtime overhead of dynamically computing ranges; per-channel quantization matches the structure of most mobile hardware; and targeted 2-bit quantization—normally 2-bit quantization suffers severe loss and is extremely hard to do well with PTQ, but the team uses QAT to know which parameters can be compressed to 2-bit without losing quality. Built-in quantization of embeddings and KV cache is also included. Together, this combination dramatically shrinks model size while significantly reducing active memory and bandwidth usage.

QAT (Quantization-Aware Training) and PTQ (Post-Training Quantization) are two complementary quantization approaches. PTQ directly compresses weights to lower bit-widths after training is complete—low cost, but precision loss is harder to control, and it is often unusable at extremely low bit-widths like 2-bit. QAT simulates the bucketing and rounding behavior of quantization during training, letting model parameters actively adapt to low-bit representations, thereby preserving quality when compressing from 16-bit to 4-bit or even partially to 2-bit.

Static activation quantization means pre-determining the quantization range (scale/zero-point) of activations before inference, so no dynamic statistics are needed at runtime, saving compute; per-channel quantization sets quantization parameters individually for each output channel, which is finer-grained than per-tensor quantization and better matches the multiply-accumulate array structure of mobile NPUs. The combination of the two is an important reason mobile quant can achieve low latency on phone hardware.

Two Integration Paths: AI Core and LiteRT

The first choice developers face is which path to take.

The AI Core path is the most hassle-free. Gemini Nano 4 is Gemma built and run through AI Core. Per-layer embeddings are handled properly by the runtime, so embedding parameters need not reside in RAM; MTP is built in; and mobile quant is used, with the runtime able to run at 2-bit precision. In the request flow, there is a safety check at both the input and output ends, forming a second layer of protection beyond the model itself. The model is shared across apps and updated automatically, making it suitable for scenarios where you're building multiple apps and don't want to manage packaged models yourself.

The LiteRT path offers complete control. The model is packaged with the app and does not share the on-device shared model, so memory, power, and resource management all become the developer's responsibility. In exchange, you get full control: you can use fine-tuned models and have more room for customization.

AI Core is a system-level inference service on the Android platform for on-device AI, where models are managed uniformly by the system, shared across apps, and updated with the system—developers call it via API without packaging model files themselves. LiteRT (formerly TensorFlow Lite) is Google's on-device inference runtime where the model is distributed with the app, and developers have full control over model version, memory layout, and hardware delegates. The trade-off between the two is essentially "system-managed" versus "app-owned": the former is hassle-free but offers limited customization, while the latter is flexible but requires managing memory and power yourself. Gemini Nano 4 is distributed through AI Core, whereas scenarios requiring fine-tuning or deep customization are better suited to LiteRT.

Two Live Demos

The first is a voice assistant running entirely on-device, using a VAD cascade: speech-to-text → Gemma → text-to-speech, fully streaming, running on E2B. The demo was conducted on a laptop, but Sarah emphasized that it's the same model as on a phone, and the entire pipeline can absolutely run on a phone. During the demo she deliberately interrupted and switched topics, and the model handled it smoothly throughout, with no server round-trips.

The second is a video capability demo set in an offline "wild camping" scenario, based on E4B. The model calls tools in the background and accesses various phone functions: recognizing and explaining a Spanish warning sign, solving a derivative, checking a calendar, and distinguishing whether a photo shows a cougar or a dog. The tone is natural, with a bit of humor.

Developers Can Get Started Right Away

Sarah closed with entry points: checkpoints are released on Hugging Face and Kaggle, model code has entered the Transformers codebase, and the ML Toolkit provides accompanying links. For a deeper dive into the ML Kit path, see David's session, while LiteRT development details are covered in the talks by Chitin and Ferguson.

For on-device AI developers, the signal from Gemma 4 is clear: model capability density is rising, quantization and inference optimization are moving toward mobile hardware, and integration paths are converging into two clear options—"hassle-free" and "full control." What truly determines the experience is now what you can build within this budget.

One-line Summary

With per-layer embeddings, a shared-KV-cache draft model, and mobile-tailored quantization, Gemma 4 lets small models punch above their weight against large models on phones.

Paragraph Summary

Google's on-device AI strategy has shifted toward architectural innovation rather than forcing large models onto devices. Gemma 4 offers phone-class models like E2B and E4B, with E2B comprehensively outperforming Gemma 3 27B on reasoning, coding, and agentic benchmarks. Key techniques include: per-layer embeddings (PLE), which make the nominal parameter count far exceed the roughly 2B actually loaded; an MTP draft model sharing the base model's KV cache, boosting acceptance rates by 40-60%, cutting decoding latency by 40%, and reducing energy consumption by 30-40%; and QAT combined with targeted 2-bit quantization to dramatically shrink size and active memory. For integration, AI Core provides an out-of-the-box shared path, while LiteRT offers a fully controllable path that supports fine-tuning.