Transformers v5.19.0 Released: EmbeddingGemma 2 Multimodal Embeddings and Breaking MoE Changes

Hugging Face Transformers has shipped its v5.19.0 release. The headline addition is Google's EmbeddingGemma 2 multimodal embedding model, alongside several breaking changes to Mixture-of-Experts (MoE) models, attention mechanisms, and Continuous Batching infrastructure. For engineers building production systems on this library, the upgrade brings new capabilities but also requires careful attention to API compatibility.

EmbeddingGemma 2: A Unified Cross-Modal Vector Space

The most anticipated new model in this release is EmbeddingGemma 2, a multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio, and video — individually or in combination — into a shared 768-dimensional vector space, enabling cross-modal retrieval, semantic similarity, clustering, and classification tasks.

The model employs Matryoshka Representation Learning, which means embeddings can be truncated to 512, 256, or 128 dimensions without significant loss of expressive quality. This is particularly useful for scenarios that require trading off storage costs against retrieval precision.

EmbeddingGemma 2 also offers configurable vision and video token budgets, and allows unused vision or audio towers to be disabled at load time to save memory. For users working exclusively with text or image-text scenarios, this flexible component pruning can substantially reduce deployment overhead.

Matryoshka Representation Learning (MRL) is an embedding training paradigm introduced by Kusupati et al. in 2022. The core idea is to simultaneously optimize embedding sub-vectors at multiple granularities during training — for example, the full 768 dimensions as well as truncated prefixes at 512, 256, and 128 — so that shorter sub-vectors also retain strong representational quality. This contrasts with traditional embedding models, which typically require additional PCA or quantization steps to reduce dimensionality and suffer noticeable semantic loss in the process. In practice, MRL means a single model can be reused across different resource constraints: full dimensionality for high-precision retrieval, lower dimensionality for large-scale coarse filtering to save storage and compute, with no need to retrain or deploy separate models. OpenAI's text-embedding-3 series uses a similar technique, making it one of the most well-known production deployments of MRL.

MoE Models Now Uniformly Return Router Logits

The breaking change most likely to affect developers is the standardized behavior across all MoE models that compute routing logits. When output_router_logits=True is set, these models now all return routing logits, following the established pattern of Qwen3-MoE.

Concretely, there is a router_logits recorder on the base model, the backbone outputs MoeModelOutputWithPast, and the model head outputs a MoE causal language model output class. Any code that relied on the old output structure or assumed routing logits were absent will need to be updated to read them from these new output classes.

While this change unifies API consistency, it does require adaptation in downstream code and is one of the more broadly impactful changes in this release.

Background on MoE routing: The core of a Mixture-of-Experts architecture is its routing mechanism. For each input token, the router computes a set of logits which, after a softmax, determine which expert sub-networks will process that token. Routing logits not only determine the computation path but are also essential for analyzing model load balancing and implementing auxiliary load-balancing losses — if some experts are overloaded while others sit idle, training becomes unstable and inference efficiency degrades, typically addressed by computing an additional balancing loss from the routing logits. Historically, different MoE models in the Transformers library (such as Mixtral, DeepSeek^Ad-MoE, and Qwen3-MoE) differed in whether they exposed routing logits and where they appeared in outputs, forcing downstream code to handle each model separately. Consolidating around MoeModelOutputWithPast and its companion head output classes simplifies cross-model reuse of training and analysis code.

Attention Mechanism and Continuous Batching Refactor

Several adjustments to the attention mechanism have been made in this release, centered on Continuous Batching (CB).

The primary change is the deprecation of the "paged|" prefix for SDPA and Flash Attention implementations. When using continuous batching, developers should now set the standard attention implementation directly (e.g., sdpa or flash_attention_2) rather than using paged|sdpa or paged|flash_attention_2. Accordingly, standard Flash and SDPA attention functions now natively support continuous batching, and "paged|..." implementations are redirected to them; eager mode still requires the "paged|eager" prefix.

Within continuous batching, cache updates for the index path and block-table path have been merged into a single call, which slightly changes the behavior of the cache update function. Custom code that called independent update paths should take note. Continuous batching has also become more device-agnostic, with new support added for XPU.

Another notable model behavior change affects Owlv2ForObjectDetection.embed_image_query. It now selects the query box with the highest objectness score following the original OWLv2 notebook, rather than using the OWL-ViT heuristic. As a result, image-guided query embeddings and detection results may differ from earlier versions.

Background on Continuous Batching: Continuous Batching is a key optimization technique in modern LLM inference servers (such as vLLM and TGI). Traditional static batching requires all requests in a batch to finish synchronously, meaning short requests must wait for long ones, leading to poor GPU utilization. Continuous batching allows new requests to be dynamically inserted and completed requests to be removed after each decoding step, dramatically improving throughput. It relies on Paged Attention under the hood — dividing the KV cache into fixed-size "blocks" managed through a block table with non-contiguous memory, analogous to operating system virtual memory paging, avoiding the memory waste of pre-allocating maximum-length caches for each sequence. This update internalizes the paged| prefix logic inside the standard attention implementations, meaning developers no longer need to be aware of paging details at the API level — the framework handles the memory management required for continuous batching automatically.

Parallelism and Cache Configuration Enhancements

On the distributed training front, Expert Parallelism (EP) has been significantly strengthened. The new release introduces a token-dispatch implementation, selectable via the new ep_dispatch_experts planning rule, which has become the default for Qwen3 MoE and Mellum. This improvement removes the restriction that EP size must equal TP size, making expert parallel deployments considerably more flexible. The Trainer has also been adapted for expert parallelism, and documentation now notes that PEFT adapters support tensor parallelism.

Several issues with quantized caches have been fixed: generate no longer modifies a user-provided cache_config, and QuantizedLayer.reorder_cache has been corrected. More significantly, per-layer cache configuration has been added — DynamicCache and StaticCache now initialize according to each layer's individual configuration (sliding window, attention chunk size, convolution state, and number of attention heads), better supporting heterogeneous model architectures.

Other Improvements and Upgrade Recommendations

This release also contains a large number of bug fixes and engineering improvements, including: migrating CI to Python 3.11, fixing DataCollatorForLanguageModeling ignoring seed=0, fixing multi-axis position ID chunked prefill for Qwen3-VL, and replacing matrix multiplication with multiplication for RoPE to improve efficiency. Deprecated pipelines now redirect, while removed pipelines raise exceptions directly.

For teams preparing to upgrade, the recommendation is to focus evaluation on three categories of changes: if you use MoE models and read routing logits, adapt to the new output classes; if you rely on continuous batching in production, remove "paged|" prefixes and test cache update behavior; and if you use OWLv2 for image-guided detection, verify result consistency. Overall, v5.19.0 expands multimodal capabilities while continuing to push the underlying architecture toward greater uniformity and efficiency.