Transformers 5.18.0 Released: Four New Models for Multimodal AI and Speaker Diarization

Hugging Face Transformers has released version 5.18.0, pushed by maintainer vasqu. This update includes four new model architectures, six breaking changes, and a large number of bug fixes and performance improvements. For developers building production systems on top of Transformers, this is an iteration that balances new features with stability.

Four New Models: Spanning Speech, Multimodal, and Text Retrieval

The headline feature of this release is four new model additions, each targeting a distinct application domain.

Nemotron 3 Diarization: Streaming Speaker Diarization

Nemotron 3 Diarization is an open-weight streaming speaker diarization model focused on answering the question "who spoke when." It supports both streaming and offline inference, handles up to eight speakers, and outputs results ordered by the sequence in which each speaker first appears in the audio.

The model employs the Arrival-Order Speaker Cache (AOSC) and FIFO queue mechanism introduced for Streaming Sortformer. Notably, a single checkpoint supports multiple latency configurations ranging from 80 ms input buffers to 30.4-second offline-style buffers, with output frame resolution adjustable in multiples of 10 ms. Chunked inference removes any hard limit on audio duration, making it highly practical for real-world scenarios like live meeting transcription and call center recording analysis.

Speaker diarization is a classic challenge in speech processing — automatically annotating "who spoke when" without prior knowledge of speaker identities. Traditional pipelines typically involve three stages: voice activity detection (VAD), speaker embedding extraction, and clustering to merge segments from the same speaker. This cascaded approach suffers from accumulated errors across stages and struggles to support low-latency streaming. Sortformer is an end-to-end diarization framework proposed by NVIDIA that resolves output permutation ambiguity by enforcing an "arrival order" constraint. Where traditional methods rely on permutation-invariant training (PIT) to determine which output channel maps to which speaker, the AOSC mechanism instead assigns channels in the order speakers first appear in the audio. This fundamentally eliminates permutation ambiguity and ensures temporal consistency in streaming output.

NemotronH Omni: Joint-Reasoning Multimodal Model

NemotronH Omni is a multimodal reasoning model from NVIDIA that combines NemotronH's Mamba-Transformer hybrid language model with the RADIO vision encoder and an optional Parakeet audio encoder. Image and video patches are projected into the language model's embedding space via the RADIO tower and a pixel-shuffle MLP, occupying <image>/<video> context positions; audio segments are projected into <audio> positions in the same way.

The result is a single model capable of joint autoregressive reasoning over text, images, video, and audio. This unified multimodal architecture represents the mainstream direction of current frontier research, and the Mamba-Transformer hybrid backbone also delivers efficiency advantages for long-sequence processing.

Mamba is a sequence modeling architecture based on selective state space models (Selective SSM). Its key advantage is that inference computational complexity scales linearly with sequence length, compared to the quadratic complexity of standard Transformer self-attention. The Mamba-Transformer hybrid architecture interleaves Mamba layers with standard attention layers, aiming to combine Mamba's efficiency on long sequences with Transformer's strength in fine-grained local modeling. RADIO is a visual foundation model from NVIDIA that distills knowledge from multiple vision-language pretrained models (such as CLIP, DINOv2, and SAM) to produce visual representations that balance semantic alignment with fine-grained features, making it a suitable general-purpose vision encoder for multimodal systems. Pixel-shuffle MLP projection is a common resolution compression technique that rearranges spatially adjacent pixel patches along the channel dimension, reducing the number of tokens without information loss and relieving sequence length pressure on the language model.

HyperCLOVAX Vision V2 and GTE

HyperCLOVAX Vision V2 is a vision-language model developed by NAVER, using the HyperClovaX language model backbone paired with a Qwen2.5-VL vision encoder. It supports text, image, and video inputs, and enables chain-of-thought reasoning through built-in thinking tokens (<think>...</think>).

GTE comes from the paper "mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval." It is a BERT-style bidirectional encoder that replaces absolute position embeddings with RoPE, uses gated MLP, and applies layer normalization after each residual connection. The same architecture underpins Alibaba's gte-*-v1.5, gte-multilingual-* series, and Snowflake's snowflake-arctic-embed-m-v2.0 checkpoints — making it highly relevant for teams working on text retrieval and reranking.

RoPE (Rotary Position Embedding) is a relative position encoding scheme that applies position-dependent rotation transforms to query and key vectors during attention computation. This allows the model to naturally perceive the relative distance between tokens while retaining some ability to extrapolate to absolute positions. Compared to traditional absolute position embeddings, RoPE generalizes better in long-text scenarios and is increasingly adopted by encoder models. Gated MLP (also known as SwiGLU or GeGLU) introduces a gating branch into the standard feed-forward layer, where the output is the element-wise product of two parallel linear transformations. In practice, this typically improves expressiveness for the same parameter count. GTE brings both of these techniques to a BERT-style bidirectional encoder, significantly outperforming the original BERT architecture in long-text retrieval and multilingual settings.

Breaking Changes: Review Before Upgrading

Version 5.18.0 introduces six breaking changes marked with "🚨". Pay close attention to these when upgrading:

• gpt-oss on ROCm: Routes FA3 to aiter-flash-attn and generates ROCm fixtures;
• DETR image processing acceleration, which may affect pipelines that depend on the previous preprocessing behavior;
• Remapping of layer_type for indexers;
• Fix for token counting with video inputs in the Transformers backend (targeting vLLM);
• Modernization of DINO-family models to align with current standards;
• Kernels version bump.

These changes mean that some existing code may behave differently after a direct upgrade. Users relying on ROCm inference, DETR object detection, or the vLLM backend should thoroughly test before upgrading.

Large-Scale Fixes and Parallel Inference Improvements

Beyond the new models, this release includes extensive bug fixes and improvements across several key areas.

Distributed and Parallel Inference

The release unifies device_mesh initialization to support pipeline parallel (PP) + tensor parallel (TP) inference, and combines FSDP2 with Expert Parallelism (EP) via a two-dimensional (fsdp, tp) device mesh. Alongside PEFT × DTensor TP integration, these changes significantly improve training and inference flexibility for large models in multi-GPU environments.

Modern large model training and inference typically requires combining multiple parallelism strategies. Tensor parallelism (TP) shards a single layer's weight matrices across multiple devices for synchronous computation. Pipeline parallelism (PP) partitions the model by layers across devices, with different devices handling different depths of the network in a pipeline fashion. Fully Sharded Data Parallel (FSDP) distributes model parameters, gradients, and optimizer states across multiple ranks, gathering them on demand. PyTorch's device_mesh is a logical device grid abstraction that lets developers describe TP × PP × DP combinations using a two-dimensional or multi-dimensional grid, unifying communication primitive scheduling. Expert Parallelism (EP) is a parallelism dimension unique to Mixture-of-Experts (MoE) models, assigning different expert sub-networks to different devices. When combined with FSDP, this requires additional handling for cross-device communication of routed tokens — making this release's FSDP2 + EP support a significant engineering milestone.

Caching and Generation Logic

The continuous batching (CB) cache has been upgraded to support different attention types, with a new pause mechanism added. Generation-related logic has also been simplified, including making all methods and logits processors agnostic to lm_head output size, fixing assisted decoding after dropping the attention mask, and properly constraining the maximum length and EOS token in assisted decoding.

Quantization and Hardware Support

GGUF integration now supports MoE models and updates tokenizer and kernel paths. The compressed-tensors FP8 kernel now runs on MPS (torch ≥ 2.15). The release also introduces an OpenVINO HF exporter, providing a new path for deploying models on Intel hardware. Flash Attention support has also been extended to XPU and MUSA platforms.

What This Means for Developers

Overall, 5.18.0 continues Transformers' established rhythm: absorbing new models from the community and vendors while refining core capabilities around distributed computing, quantization, and generation. The four new models cover streaming speech, multimodal reasoning, and text retrieval — mapping almost directly onto the major tracks of current AI applications.

For production teams, the key upgrade decision comes down to weighing the benefits of new features against the migration cost of breaking changes. If you need speaker diarization or joint multimodal reasoning, this version is well worth trying. If you rely solely on stable existing pipelines, focus your evaluation on whether those six 🚨 changes affect your current code. Either way, thorough regression testing before promoting to production remains essential.