Transformers v5.20.0 Released: Eight New Models and Export Quantization Support

Hugging Face's Transformers library has released v5.20.0, an update that makes substantive progress across four areas: model ecosystem, export quantization, parallel training, and inference performance. For developers who use inference and fine-tuning day to day, the most noteworthy items are the addition of eight new models and the arrival of native quantization in the export pipeline.

Eight new models span multimodality, OCR, and audio

Muse Spark Open: an 816B-parameter multimodal MoE

Muse Spark Open from Meta is a multimodal mixture-of-experts model that accepts text, image, audio, and video inputs and outputs text. Its total parameter count reaches 816B, with 42B activated per token, a context window of up to 1,048,576 tokens, and this window shared across modalities. The model is trained for agentic coding tasks, including multi-file editing, debugging, and tool calling. Official checkpoints are provided in both BF16 and NVFP4 formats.

Ling 3.0 Flash: native support for a hybrid attention architecture

InclusionAI's Ling 3.0 Flash adopts a hybrid MoE design in which the decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and per-head output gates, while the feed-forward blocks combine routed experts with shared experts. The model is now natively supported in Transformers with the loading class BailingHybridForCausalLM, so there is no need to set trust_remote_code=True.

Kimi Delta Attention (KDA) and Multi-head Latent Attention (MLA) are both attention variants designed to reduce the cost of long-context inference. The core idea of MLA is to compress the key-value projection into a low-dimensional latent vector, caching only that latent vector at inference time rather than the full K/V, which greatly compresses KV cache usage; it was popularized by the DeepSeek^Ad series. KDA, meanwhile, is an improvement along the linear/incremental attention route, avoiding explicit storage of the entire history of KV through an incrementally updatable state. The difference between the two lies in the trade-off between state update method and expressive capacity: MLA still retains modeling power close to standard attention, while KDA's memory and compute grow closer to linearly with sequence length. Mixing the two in a five-to-one grouping is intended to balance efficiency and quality on long sequences—most layers use the lightweight KDA, while a few layers use MLA to retain global modeling ability.

Kolibri 1: a reasoning MoE for German and English

Aleph Alpha's Kolibri 1 is an MoE reasoning model focused on German and English. Of its 50 layers, each layer uses sigmoid routing to assign each token to 6 of 384 experts, plus an additional shared expert. Architecturally, four sliding-window attention layers alternate with one full attention layer, and RoPE is applied only to the sliding-window layers.

MiniCPM-V 4.7: introducing canvas M-RoPE

OpenBMB's MiniCPM-V series continues to iterate. Version 4.7 retains the combination of a SigLIP vision encoder and a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes. Compared with 4.6, the main addition is canvas M-RoPE: instead of numbering visual tokens along a single 1-D sequence, each image is laid out on a 2-D canvas and each visual token is assigned a (temporal, height, width) position. This way, slices of the same image preserve their spatial relationships, and video frames preserve their temporal order.

M-RoPE (Multimodal Rotary Position Embedding) originates from the Qwen-VL series. Its approach is to split RoPE's positional dimensions into three groups—temporal, height, and width—numbering video frames, image rows, and image columns respectively, so that both the 2-D spatial relationships within a frame and the temporal relationships across frames can be correctly expressed by positional encoding. MiniCPM-V 4.7's canvas M-RoPE changes how images are numbered internally on top of this: the traditional approach flattens an image into a 1-D sequence and then numbers it, so once an image is split into multiple patches, the spatial adjacency between patches is lost in the 1-D numbering; by laying the image onto a 2-D canvas and assigning positions by (temporal, height, width), the patches still retain their original row-column relationships, and the relative distances between visual tokens match their true distances in the image. This is especially important for tasks requiring fine spatial localization (such as document question answering and region referencing), and it also lets video frames maintain a stable order on the timeline.

UnlimitedOcr: transcribing dozens of pages in a single forward pass

Baidu's UnlimitedOcr is a single 3B-parameter OCR model with a context length of 32,768 tokens. It is built on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This change keeps the KV cache constant during decoding, so the model can transcribe dozens of pages of content in a single forward pass. It supports a "gundam" mode (640x640 tiles with dynamic cropping) and a "base" mode (a single 1024x1024 view), and can output region detection results with bounding boxes.

Standard decoder attention's KV cache grows linearly with the number of generated tokens, so long-document transcription faces memory bloat and the overhead of page-by-page inference. Sliding Window Attention only lets each token attend to a recent fixed window, giving the cache an upper bound, but it loses historical information outside the window, making it hard to establish connections between pages beyond the window. Reference Sliding Window Attention (R-SWA) introduces a fixed reference position on top of the sliding window (usually an anchor or special token at the start of the sequence), allowing each token to attend both to nearby content and to continuously reference this stable global information. The result is that the KV cache stays a constant size during decoding without sacrificing cross-page context consistency, so the model can continuously transcribe dozens of pages in a single forward pass without needing multiple chunked calls.

UltraBERT: a long-sequence bidirectional encoder

UltraBERT is a family of bidirectional text encoders trained through a combination of masking ratios, pruning, and distillation, targeting classification and retrieval tasks. The released encoders support sequences up to 32,768 tokens and use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, as well as UltraBERT-Embed-150M for text embeddings.

MOSS-Transcribe-Diarize: end-to-end speaker diarization

MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model aimed at multi-speaker transcription, speaker diarization, and timestamping for long audio. It combines a Whisper-style encoder, 4x frame merging, a multimodal projector, and a Qwen3 language model, and can output joint transcription and diarization results in a single pass, with segments formatted as [start][S01]text[end]. It supports long audio through Whisper window chunking in the processor, supports prompt-based transcription with hotword biasing, and covers more than 50 languages.

Native quantization support in the exporter

The Transformers exporter now supports quantization directly at export time on Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, supports static quantization, dynamic quantization, and weight compression (INT8/INT4), and provides a configurable calibration pipeline. This means developers can more conveniently export optimized models for efficient inference on different backends.

Parallelization and generation performance improvements

On the distributed training and inference side, FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations. Expert-parallel dispatch, tensor parallel plans, static-cache generate(), and the byte counting of the cache allocator have all been fixed. The shared LM head in composite models now has its own TP plan entry, and enable_expert_parallel once again applies the EP plan separately.

On the generation side, this release fixes several issues with static-cache generate() under host-side inputs and tensor parallelism, speeds up Whisper generation on TPU by reducing device synchronization and recompilation, and fixes Whisper batch timestamp decoding for tensor and NumPy inputs. Generation parameter handling in pipelines has also been cleaned up, removing redundant warnings.

FSDP (Fully Sharded Data Parallel) is responsible for sharding model parameters, gradients, and optimizer states across cards, while TP (Tensor Parallel) splits a single weight matrix along a dimension and has multiple cards jointly complete the computation of the same layer. When the two are combined, the shape and sharding layout of a checkpoint depend on the specific world size, TP group size, and sharding strategy, so switching to a different parallel configuration often makes it impossible to directly load an old archive. This release lets FSDP × TP model and optimizer checkpoints be saved and loaded across different parallel configurations, meaning you can train on a small-scale topology and continue training or inference on another topology without having to reconvert the weights. Expert-parallel (EP) is a parallelism dimension specific to MoE, dispatching different experts to different devices; its interaction with TP plans is prone to dispatch errors, so it is necessary to establish a separate TP plan entry for the shared LM head and let enable_expert_parallel apply the EP plan independently, avoiding the two parallel strategies overriding each other.

Breaking change reminders

This is a release containing multiple breaking changes, so caution is needed before upgrading. Inference with EP currently has a speed regression, which the maintainers say is being fixed. Deprecated checks and methods in cache_utils.py have been removed, and code still calling these methods needs to switch to the current cache API. Accidental imports of other models have been removed from multiple model files, and the modular converter no longer adds such cross-model imports.

The minimum version of kernels has been raised to 0.17.2 to support use_fallback=KernelizeFallback.ALL, allowing fallback to the original implementation instead of raising an error when kernel loading fails. The newly added SubConfigSpec centralizes sub-config initialization, and custom or existing nested configs may need to declare sub-configs according to the new mechanism. The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens, so tokenization and encoding outputs involving these tokens may differ from before.

Kernel and cache fixes

Kernel support continues to expand in hardware compatibility: on ROCm, flash_attention_2 falls back to AITER kernels when flash-attn is not installed; T5-like models gain contiguous SDPA position bias masks, allowing them to reuse fused kernels again; and Hub rotary kernels are restricted to CUDA inference only because their backward pass produces incorrect gradients.

Several issues have been fixed in caching: static-cache generate() now correctly handles host-side nested inputs and tensor parallelism, offloaded cache prefetching no longer competes with offload copies, Phi3 LongRoPE cache receives the full prefix when rebuilding at context boundaries, and MiMo-V2-Flash's sliding window now counts query tokens.

For teams currently using Transformers, this release's new model coverage and export quantization capabilities are clear reasons to upgrade, but there are many breaking changes, so it is recommended to first verify compatibility of cache API calls and custom configs in an isolated environment.

One-line Summary

Transformers v5.20.0 adds eight models, native export quantization, and parallel fixes, but has many breaking changes.

Paragraph Summary

Hugging Face Transformers v5.20.0 advances across four areas: model ecosystem, export quantization, parallel training, and inference performance. It adds models including Muse Spark Open, Ling 3.0 Flash, Kolibri 1, MiniCPM-V 4.7, UnlimitedOcr, UltraBERT, and MOSS-Transcribe-Diarize, covering multimodal MoE, hybrid attention, German reasoning, document OCR, long-sequence encoding, and speaker diarization. The exporter supports quantization at export time on Dynamo, ONNX, OpenVINO, and ExecuTorch, unifying PT2E, ONNX Runtime, and NNCF. FSDP × TP checkpoints can be saved and loaded across parallel configurations, and static-cache generate() plus Whisper generation performance have been fixed. This version has many breaking changes, so upgrading requires first verifying the cache API and custom configs in isolation.