Transformers 5.17.0 Released: Six New Models and Unified RoPE Refactoring

Hugging Face Transformers has released version 5.17.0, with updates focused on new model integrations, standardization of visual rotary position embeddings (RoPE), generation performance improvements, and a wide range of quantization and caching fixes. For developers closely following the open-source LLM ecosystem, this release brings several architecturally ambitious new models alongside one breaking change that deserves careful attention.

Six New Models Land at Once

The most eye-catching part of this release is the expansion of the model library, spanning language, speech recognition, speech synthesis, and multimodal retrieval.

HYV4: A 780B-Parameter MoE Behemoth

Hy4-Preview is a 780B-parameter Mixture-of-Experts (MoE) language model that activates only 49B parameters per token. Each MoE layer contains 256 routed experts plus one always-active shared expert, with each token routed to 8 of them. The context window reaches 1 million tokens.

Its architecture combines four techniques: Multi-head Latent Attention (MLA), which compresses key-value pairs into low-rank latent representations; DeepSeek^Ad Sparse Attention (DSA), which uses a lightweight indexer to select top-k keys per query; gated MLA with learnable attention sinks, giving each head a sink logit that participates in softmax without contributing to the output value (similar to GPT-OSS); and Independent Hyperconnections (iHC), which replaces the traditional single residual path with multiple parallel residual streams. Notably, the implementation does not execute Multi-Token Prediction (MTP) layers, but the released weights retain these parameters so other runtimes can use them for speculative decoding.

Background — Mixture of Experts (MoE): MoE is a sparse activation architecture where the total parameter count is large, but only a subset of "expert" sub-networks are selected and activated by a router for each token, while the rest remain idle. This allows the model to achieve capacity close to a large dense model at compute costs closer to a small dense model. HYV4's 780B total parameters activate only 49B per token — an activation ratio of roughly 6% — dramatically reducing inference compute requirements compared to a dense model of equivalent size.

Multi-head Latent Attention (MLA), first introduced by DeepSeek-V2, compresses the KV cache from storing full K/V per head into a shared low-rank latent vector that is up-projected back during computation, significantly reducing GPU memory usage at inference time. Independent Hyperconnections (iHC) address gradient propagation in deep networks by replacing the single residual path with multiple parallel residual streams, allowing features at different granularities to flow independently without interfering with each other — a structural extension of the standard Transformer residual connection.

Three New Models in the Speech Domain

This release fills three gaps in the speech landscape at once: synthesis, recognition, and codec. VibeVoice is a framework that synthesizes high-fidelity, long-form multi-speaker speech using a "next-token diffusion" approach within an LLM architecture, making it particularly well-suited for content generation like podcasts and multi-speaker audiobooks.

Fun-ASR-Nano comes from Alibaba DAMO Academy's FunAudioLLM team. It is an 800M-parameter end-to-end speech recognition model that achieves SOTA results on Chinese, English, and Japanese ASR benchmarks while being far smaller than comparable models. It supports 7 Chinese dialects and 26 regional accents, with built-in hotword customization and native punctuation output, eliminating the need for a separate punctuation model.

Canary-1B-v2 is from NVIDIA — a fast, robust multilingual ASR and speech translation model that reuses the Parakeet Fast Conformer encoder and switches between transcription and translation tasks via decoder prompt prefixes. Also added is NeuCodec, a neural audio codec using Finite Scalar Quantization (FSQ) for a single-codebook design, maintaining transmission robustness at a low bitrate of 0.8kbps, taking 16kHz input and outputting at 24kHz.

KimiLinear and NeoMME

Kimi Linear is a hybrid linear attention architecture from Moonshot AI. Its core is Kimi Delta Attention (KDA) — built on Gated DeltaNet, it gives each key channel its own independent forget gate, allowing the recurrent state to decay per channel rather than per head. Most layers use KDA, with a full attention block reusing DeepSeek-V3 MLA inserted every four layers.

NeoMME is a family of 260M and 800M-parameter multimodal-native multilingual encoders from H Company. It processes multilingual text tokens and raw image patches simultaneously within a single bidirectional Transformer encoder, requiring no separately pretrained vision tower. The derived NeoMME-Retriever is fine-tuned specifically for visual document retrieval.

Background — Linear Attention: Linear attention is an approximate alternative to standard softmax attention that reduces computational complexity from quadratic to linear in sequence length via kernel function decomposition, at the cost of losing some precise global dependency modeling. Gated DeltaNet is a linear attention variant that introduces forget gates to control how much historical state is retained, addressing the problem of early linear attention models forgetting too quickly or failing to update over long sequences. Kimi Delta Attention refines this by bringing the forget gate down to the channel dimension of each key, giving the model differentiated memory decay rates for different semantic features — theoretically balancing long-range memory with local precision. The hybrid architecture (most layers using linear attention with periodic full attention blocks) is the mainstream approach for balancing efficiency and quality.

Breaking Change to Be Aware Of

This version includes one explicit breaking change: visual rotary position embeddings (2D/3D RoPE) have been unified into a centralized RoPE frequency computation module. This means that if your custom vision model relies on attention-layer-level or model-specific RoPE grid interleaving logic, you must migrate to the new modeling_rope_utils.py implementation. Teams maintaining custom vision models should assess the impact of this change before upgrading.

Background — RoPE Unification: Rotary Position Embedding (RoPE) encodes positional information into query and key vectors as rotation matrices, making attention scores naturally incorporate relative position relationships. It has become the standard positional encoding scheme for most modern LLMs. Extending RoPE to vision requires handling two-dimensional (row/column coordinates of images) and even three-dimensional (temporal axis of video) grid structures. Different models previously implemented their own frequency computation and coordinate interleaving logic, resulting in scattered, hard-to-maintain code. Unifying visual RoPE into modeling_rope_utils.py means all vision models now share a single validated frequency computation path, reducing the risk of precision discrepancies from implementation differences. The trade-off is that custom implementations relying on the old attention-layer-level logic must be migrated, or they will produce silent numerical errors or dimension mismatches.

Generation and Cache Performance Improvements

Improvements to the generation module address both performance and correctness. On the performance side, the most impactful change is stopping the synchronization of accelerators at every decoding step, which directly reduces per-step overhead. A fix was also applied to stop unconditionally downloading remote hub files during generation. For correctness, auto-compilation cache checking for encoder-decoder models has been added, past_key_values naming in AfMoE has been unified, and several flaky tests have been fixed.

On the caching side, a batch of fixes landed: a quantized cache bug in VibeVoice, VoxtralRealtime incorrectly rejecting non-static cache implementations, and a silent failure when paged attention was called without a cache — this now raises an exception explicitly. Documentation for ContinuousBatchingConfig usage has also been updated.

Ongoing Quantization and Kernel Improvements

Several specific quantization bugs were fixed: incorrect FP8 embedding handling for Qwen models, missing FP8 tensor parallel layer coverage, and unnecessary MXFP4 weight dequantization on XPU devices. AutoRound also gains support for 5/6/7-bit quantization.

On the kernels side, flexibility has been improved: nested FLA kernel imports now work when only fla-core is installed; a warning is now emitted when hub kernel functions silently fall back to slower pure PyTorch reference implementations; and standalone functions (such as RoPE) can now be registered in KernelConfig with the option to opt out of inheriting default mappings. A substantial number of bug fixes and test stability improvements also landed, covering device mismatches, gradient checkpointing, sliding window masks, and other details.

Background: FP8 is an 8-bit floating-point format (with E4M3 and E5M2 variants) that halves the storage and memory bandwidth requirements compared to FP16/BF16 for weights and activations — a key acceleration technique for large model inference and training on high-end GPUs like H100/H200. AutoRound is a post-training quantization (PTQ) method based on signed gradient descent that minimizes quantization error through small rounding adjustments to weights. Previously focused on 4-bit quantization, the new 5/6/7-bit support gives users finer-grained control over the precision-compression tradeoff. FLA (Flash Linear Attention) kernels are hardware-optimized implementations of linear attention — analogous to FlashAttention for standard attention — that reduce HBM access by reorganizing computation order. Allowing the use of related kernels with only the lightweight fla-core package installed lowers the dependency barrier significantly.

Summary

5.17.0 continues Transformers' characteristically rapid iteration pace: it brings cutting-edge large model architectures like HYV4 and Kimi Linear to the library, while also making practical additions in speech and multimodal domains. For everyday users, the most immediately noticeable improvements are faster inference from eliminating per-step accelerator synchronization and the fix preventing unconditional remote file downloads. For teams maintaining custom vision models, the RoPE unification refactor is the migration item that must be addressed before upgrading.