NVIDIA Nemotron 3 Speaker Diarization Tested: Accurately Identifying 8 Speakers in a Mixed Audio

The Last Mile of Speech Transcription: Who's Speaking?

Speech recognition models are already quite mature — Whisper, Parakeet, Canary. If all you need is a transcript, that problem is largely solved. But the moment you want to feed audio data into an LLM or build an agentic application, raw text is nowhere near enough. You need timestamps to know when each sentence was spoken, and more critically, you need to know who said it.

Imagine a four-person debate podcast where you need to track who holds which position, or meeting notes where you need to know who agreed to which action item. When agents process this content, extract implied intent, and attribute it to specific individuals, a wall of undifferentiated text with no speaker labels is nearly useless. This is the core problem that Speaker Diarization is designed to solve.

Simply put: speech recognition tells you what was said, speaker diarization tells you who said it — combine them and you get a speaker-labeled transcript.

Nemotron 3 Speaker Diarization: Lightweight and Open

NVIDIA's newly released Nemotron 3 speaker diarization model is an open-weight model purpose-built for this problem, released under a commercially permissive license. Its most compelling feature is that it has only around 100 million parameters and requires just 4GB of VRAM, meaning many smaller GPUs — including laptop RTX cards — can run it quite smoothly.

In terms of capability, it supports up to 8 speakers, whereas most previous models typically topped out at 3 to 4. More importantly, it handles not just clear, turn-by-turn speech among 8 people, but remains stable even when speakers interrupt each other or talk simultaneously.

The model is designed to be decoupled from any specific ASR model — you can pair it with your preferred speech recognition model (such as Parakeet) without being locked into a single provider.

An Overlooked Family of Speech Models

NVIDIA released this as part of the Nemotron speech model family. Many people only know Nemotron for its LLMs, but the family also includes a rich set of speech models:

• Recognition: Parakeet, Canary, and Nemotron ASR, collectively supporting streaming and batch transcription across 43 languages;
• Speech synthesis: A TTS model called Magpie, with multilingual support and voice customization;
• Translation and speech-to-speech: Such as PersonaPlex, which supports full-duplex processing.

On the diarization side, the previous option was called SortFormer, which supported up to 4 speakers and was quite popular — the older SortFormer alone received over 300,000 downloads in August, indicating significant production use. Nemotron 3 can be seen as a direct replacement with meaningfully better performance. The pace of these releases suggests NVIDIA is systematically completing a full-stack speech model suite.

Evaluation Criteria: DER and the Streaming vs. Offline Distinction

One of the biggest challenges in speaker diarization is overlapping speech. When someone asks a question in a meeting, two or three people often respond simultaneously. Older systems tended to break down in these scenarios. The author acknowledges that no model handles this perfectly — it's a hard problem even for humans.

The industry uses Diarization Error Rate (DER) for evaluation, which sums three types of errors: missed speech, non-speech falsely detected as speech, and speech attributed to the wrong speaker. Lower is better. Based on data shown in the video demo, NVIDIA's latest model outperforms both competitors and its own previous generation by a significant margin.

Another key concept is the streaming vs. offline distinction:

• Offline mode: The model can view the entire recording before making decisions — ideal for pre-recorded podcasts and meeting archives;
• Streaming mode: Decisions must be made in real time as audio arrives — suited for live captions or voice agents, at a slight cost to accuracy.

Nemotron 3 supports both streaming and offline modes and is language-agnostic.

DER is worth understanding more precisely. It's calculated by summing three types of error duration on the audio timeline, divided by total actual speech duration: Miss errors occur when real speech is labeled as silence or noise; False Alarm errors occur when background noise is misclassified as speech; Speaker Error occurs when a speech segment is attributed to the wrong speaker. Speaker Error is typically the largest contributor and the hardest to optimize. Notably, DER does not directly penalize incorrect speaker count predictions — if the model identifies 8 speakers as 6, DER can still be low as long as each audio segment is correctly attributed within its time boundaries. This is why evaluations typically pair DER with optimal speaker mapping (Hungarian algorithm) for label alignment.

Local Deployment Test: Pushing the Limits with 8 Synthetic Voices

The author set up a complete local pipeline: a Next.js frontend running on a Mac, calling a backend via Tailscale; the backend runs on a DGX Spark using a Docker container with NVIDIA NeMo, which integrates the speaker diarization model, Parakeet ASR, and Nemotron 3.5 multilingual ASR. The service layer is wrapped with FastAPI (for large-scale deployment, Dynamo or Triton would be preferred).

The most impressive test involved 8 different open-source TTS voices running simultaneously — including NVIDIA's Nemotron voice Chris, QTI's Pocket, Qwen3's voice Serena, Cocoro's Adam, Resemble AI's Chatterbox, Nari Labs' John Dyer, Maya Research's Maya, and others. Even as these eight voices interrupted each other and spoke simultaneously, the model accurately attributed each audio segment to the correct speaker.

Processing speed was equally impressive: a 90-second audio clip was transcribed and diarized in just three to four seconds. The raw JSON returned by the model labels speakers as speaker_1, speaker_2, and so on — the application layer then remaps them to real names (the model itself outputs numbers, not names, but a short snippet of code handles the substitution).

NVIDIA NeMo is key to understanding this deployment setup. NeMo (Neural Modules) is NVIDIA's open-source framework for large-scale speech and language model training and inference, providing a complete toolchain from data processing and model training to inference deployment, covering ASR, speaker diarization, TTS, NLP, and more. Its diarization component encapsulates the full pipeline including speaker embedding extraction and clustering algorithms (such as the MSDD multi-scale diarization decoder), allowing users to call pretrained models directly without needing to understand the underlying implementation. DGX Spark is NVIDIA's compact AI workstation aimed at individual developers and small teams, powered by the Grace Blackwell chip and positioned as a "desktop supercomputer" — the hardware the author used to run the local backend. For users without a DGX Spark, the 4GB VRAM requirement means mainstream consumer GPUs (such as the RTX 3060/4060) are equally capable.

Long-Form Audio and Analytics

The author also fed in an entire episode of the All-In Podcast (the Jensen Huang interview episode, over an hour long). The model completed all processing in approximately 148 seconds (about two and a half minutes), outputting a per-speaker transcript along with statistics — each person's total speaking time, number of turns, and so on. Results showed Jensen Huang spoke the most as the guest, which matches expectations.

Transcripts can be exported as plain text or SRT subtitle files. For multilingual use cases, you can switch to the multilingual ASR model, which is labeled as "adapter-ready" and can be fine-tuned for better performance in other languages.

Why This Matters

As demand for voice capabilities continues to rise — whether for single-user interactions with agents or agents scraping multimodal content like YouTube videos to extract context — precise attribution of "who said what" is becoming foundational infrastructure for reliable voice applications.

The value of Nemotron 3 speaker diarization lies in three compounding factors: open weights with commercial licensing, an extremely accessible 4GB VRAM threshold, and stable performance with 8-speaker mixed audio — plus the ability to pair with any ASR model in a decoupled fashion, with the entire pipeline running fully on-premises. For teams that prioritize data privacy or need to batch-process meetings and podcasts, this is a solution worth serious evaluation.