The benchmark dataset
Evaluating diarization fairly requires data that reflects real-world messiness. We built a custom benchmark from true stereo calls sourced from real-world production telephony domains, then merged each stereo call into a single mono waveform for model inference. Ground-truth speaker activity was derived by running Silero VAD independently on each original stereo channel, giving clean speaker-attributed reference labels without any manual annotation effort.
Call lengths range from 6 seconds to over 42 minutes, reflecting realistic telephony distributions. With average speaker overlap at just 2.8% and silence accounting for 44% of audio, this is a precision-demanding benchmark — models are penalized for false alarms in silence as much as for missed speech.
The models
Four architecturally distinct approaches were evaluated — from end-to-end neural models to separation-first pipelines. Each card links directly to the official repository or model page.
WavLM-based end-to-end diarizer that directly produces speaker activity segments. Configured for two speakers with post-filtering to retain the top two by speech duration.
NVIDIA's streaming-capable diarization model run in high-latency offline mode for maximum accuracy. Deterministic sorted speaker outputs simplify downstream pipelines.
Multi-scale diarization decoder tuned specifically for telephony. A clustering-based pipeline run with a hard two-speaker constraint and multi-scale temporal resolution.
Results
| Model | Calls | DER% | Missed% | FA% | Confus% | Avg lat (s) |
|---|---|---|---|---|---|---|
| DiariZen best DER | 200 | 5.45 | 3.42 | 1.69 | 0.34 | 8.73 |
| SortFormer fastest | 200 | 8.64 | 2.39 | 4.63 | 1.62 | 1.89 |
| MSDD Telephonic | 200 | 12.48 | 2.92 | 7.73 | 1.83 | 8.49 |
| MossFormer | 200 | 21.76 | 4.51 | 5.86 | 11.38 | 28.13 |
Key takeaways
Discussion
Looking at DER alone can be misleading — each model fails differently. DiariZen's errors are almost entirely boundary-level (missed speech 3.42%, false alarm 1.69%), with speaker confusion at a negligible 0.34%. This makes it ideal for downstream tasks sensitive to speaker identity, like per-speaker ASR or voice analytics.
SortFormer's higher false alarm rate (4.63%) suggests it hallucinates short speech segments during silence. This is worth accounting for in downstream ASR pipelines but is unlikely to matter for turn-taking analysis. At 1.89 seconds average latency, it is by far the most practical model for near-real-time applications.
MSDD Telephonic's large false alarm contribution (7.73%) dominates its error budget. In telephony with long silence regions, this is a significant weakness. Post-filtering with a tighter speech-activity threshold would likely close much of the gap with SortFormer.
MossFormer's outsized confusion (11.38%) is the defining characteristic of this benchmark run. The separation-first approach was designed for overlap-heavy meeting scenarios — applied to two-speaker telephony calls at 2.8% overlap, it introduces permutation errors that compound across the call, while the pipeline overhead drives latency to 28 seconds.
Methodology
All models were run on an NVIDIA A100 80 GB GPU (NV0d0a) with CUDA 12.1. Latency figures represent wall-clock time per call including model forward pass, pre/post-processing, and disk I/O. Each call was processed sequentially; no batching was applied so numbers reflect single-call real-world latency.
Shared pre-processing
All stereo source calls were downmixed to mono by averaging channels. Audio was resampled to 16 kHz before being passed to any model. No noise suppression or channel equalisation was applied, preserving real-world telephony conditions including background noise, codec artefacts, and level imbalance between speakers.
DiariZen
Run via the pyannote-compatible pipeline at 16 kHz. Minimum and maximum speakers fixed at 2. Post-processing: speakers ranked by total speech duration; only the top-2 labels retained and renamed to speaker_0 / speaker_1. No additional smoothing applied.
SortFormer
Loaded via NeMo (nvidia/diar_streaming_sortformer_4spk-v2) and configured with high-latency offline streaming parameters to maximise accuracy. Output speaker tokens are sorted deterministically by first-appearance time, removing the need for post-hoc permutation alignment. Speaker count fixed at 2.
MSDD Telephonic
NeMo MSDD pipeline with num_speakers=2, max_speakers=2. Multi-scale windows of 1.5s, 1.0s, 0.5s, and 0.25s were used for the temporal encoder. Speaker embeddings extracted using TitaNet-Large. Clustering performed with NME-SC (Normalised Maximum Eigengap Spectral Clustering).
MossFormer2
Audio first passed through Silero VAD (threshold 0.5, min speech 250 ms, min silence 100 ms) to detect speech chunks. Each chunk downsampled to 8 kHz before separation. MossFormer2 run on each chunk independently, producing two source-separated streams. TitaNet-Large embeddings extracted per chunk per stream; cosine similarity used to resolve inter-chunk speaker permutation. Final streams stitched with a same-speaker merge gap of 0.3s; overlap bleed-through suppressed using a channel energy ratio threshold of 0.6.
Evaluation protocol
DER computed using the pyannote.metrics library with a collar of 0.25s (standard telephony tolerance). Overlap regions are included in the DER calculation. The three DER components — Missed Speech, False Alarm, and Speaker Confusion — are reported separately to diagnose failure modes independently.
Reproducibility
All models were evaluated on the same 200-call set from a fixed, versioned internal dataset. Mono conversion and channel-wise Silero VAD annotation followed identical protocols across all runs. Sample rates and model checkpoints were kept constant, and DER, Missed, FA, Confusion, and per-call latency were reported for every model across the full evaluation set.
References & model links
All checkpoints used in this benchmark are publicly available via the links below.