Benchmarking Practices, Datasets, and Metric Gaps for General and Indic Conversational Text-to-Speech Systems

Across modern AI products, Text-to-Speech has moved from a convenience layer to core infrastructure. It now powers screen readers, dubbing pipelines, customer support bots, IVR flows, voice assistants, and increasingly real-time conversational agents. Published TTS systems have advanced simultaneously across phoneme-aware modeling, non-autoregressive streaming, prosody control, multi-speaker adaptation, and instruction-following synthesis. Because these models optimize different tradeoffs, benchmarking is not simply about ranking systems; it is about understanding which model is best for which deployment setting, especially when real systems must handle numbers, names, abbreviations, code-switching, long-form dialogue, and strict latency constraints.

Benchmarking TTS is substantially harder than benchmarking text-only models. Human perception shifts with context, task framing, and even the wording of evaluation instructions. A model that excels on short read speech may fail badly in continuous dialogue, on financial amounts, or in multilingual transactional speech. Modern TTS evaluation has gradually shifted from a single MOS score toward multi-dimensional, task-aware, and sometimes automated evaluation pipelines. Even so, MOS still dominates in practice, while richer protocols remain uncommon.

Quality

In most papers, quality receives the most attention because users notice it first. Yet the field has learned that "quality" is not one thing. It conflates naturalness, intelligibility, pronunciation, prosody, speaker realism, and absence of artifacts, which is one reason evaluation results so often become difficult to interpret across papers.

1. Mean Opinion Score

MOS asks listeners to rate perceived naturalness or quality on a fixed scale and averages ratings across N listeners. Although simple and widely used, MOS is unstable across time, listener pools, instructions, and comparison sets. Near-human systems create ceiling effects where several models receive similarly high scores even when their real weaknesses differ. Empirical work has shown that changing the evaluation instruction from "rate quality" to "rate naturalness" alone can invert system rankings. Recent work repeatedly argues that MOS alone should not be treated as a complete benchmark.^{[1, 2]}

2. CMOS and SMOS

Comparative MOS asks listeners which sample sounds better and by how much, producing a signed preference score averaged across raters. It can detect smaller differences than plain MOS in some settings but remains sensitive to prompt selection, task framing, and experimental design. Similarity MOS rates how closely synthesized speech matches a target speaker identity, shifting the question from "How natural?" to "How similar?" This is especially important in voice cloning, multi-speaker TTS, and personalized assistants.

3. MUSHRA

MUSHRA compares multiple systems in one sitting using references and anchors, rating each sample on a continuous 0 to 100 scale. Because all systems are heard together, MUSHRA discriminates better than MOS when systems are close in quality. Recent Indic-focused work, specifically with Hindi and Tamil listeners, shows that classic MUSHRA can be biased by human references, motivating refined variants that allow scores above the human reference and reduce rater judgment ambiguity.^[3]

4. Signal-Based Objective Metrics

Mel Cepstral Distortion (MCD) measures spectral distance between synthesized and reference speech. Lower values indicate a closer spectral match, though MCD captures only one part of perceived quality and often correlates imperfectly with human judgments. PESQ estimates perceptual speech quality relative to a clean reference, modeling distortions relevant to telephony. STOI estimates intelligibility from short-time spectral envelope agreement, scoring 0 to 1. SNR compares signal energy to noise energy; better SNR means fewer audible artifacts but does not alone guarantee human-like speech. F0 RMSE measures pitch prediction error against a reference contour, revealing monotone or unstable intonation. Duration RMSE measures timing error between synthesized and reference alignments; rhythm and pacing contribute strongly to naturalness and conversational smoothness.

5. Human-Likeness and Deception Tests

The Audio Turing Test^[4] asks listeners whether a sample is human or synthesized, shifting evaluation philosophy from "How good is it?" to "Can it pass as human?" This change in framing reveals weaknesses hidden by MOS. Human Fooling Rate^[5] measures how often listeners mistake synthesized speech for human speech, expressed as the fraction of synthetic samples judged human. These metrics are especially useful when several systems already sit near the MOS ceiling and conventional naturalness scores can no longer discriminate.

6. Comprehension and Decomposed Scales

SP-MCQA^[6] benchmarks whether listeners answer questions correctly after hearing spoken passages, measuring key-information comprehension accuracy as the fraction of correct answers over total questions. This matters because a TTS model can achieve low WER yet still fail to communicate critical information accurately, an important mismatch for applications in finance, support, and navigation. Decomposed scales, which rate clarity, fluency, pronunciation, understandability, and human-likeness separately, break "quality" into interpretable dimensions and are becoming essential for identifying exactly where a system fails rather than collapsing everything into a single score.

7. TTSDS2

TTSDS2 combines multiple feature distributions, including prosody, speaker characteristics, and intelligibility proxies, by comparing distances between real speech, synthetic speech, and noise-like baselines. Validated against over 11,000 MOS, CMOS, and SMOS ratings across four domains including clean and noisy audiobooks, in-the-wild YouTube, and children's dialogue, TTSDS2 is reported as the only metric among sixteen tested to maintain Spearman correlation above 0.5 across twelve domain-score pairs. It is one of the strongest objective correlates of human ratings currently available and points toward what post-MOS benchmarking can look like.^[7]

Latency

For interactive TTS, quality alone is not enough. In voice assistants, IVR, and streaming conversational systems, users care not only about how the output sounds but also about how fast it starts and how steadily it continues. Latency is an increasingly visible benchmark dimension, even though it remains underreported in many model papers and almost entirely absent from Indic TTS evaluations.

1. Real-Time Factor

RTF divides generation time by output audio duration. When RTF falls below 1, the model generates faster than playback time and can support real-time use. This remains the most commonly reported latency metric, appearing in the Bangla TTS benchmark and the Indic vocoder evaluation, though its presence elsewhere is sporadic.

2. Time to First Audio and End-to-End Latency

TTFA measures the delay before the first audible chunk becomes available by subtracting the request time from the time of the first audio output. For streaming systems, TTFA often matters more to user experience than total synthesis time, because it controls how quickly the agent starts speaking and how responsive the interaction feels. End-to-end latency covers the full pipeline, including preprocessing, normalization, acoustic modeling, vocoding, and output transfer, and is the most practical deployment metric because it reflects what users actually experience.

3. Tail Latency and Streaming Continuity

Tail latency measures worst-case or high-percentile delays, typically p95 or p99, over repeated synthesis trials. Production systems fail on outliers, not averages; a fast median response with unstable p99 behavior still feels unreliable in live use. Streaming continuity measures whether audio arrives smoothly without stalls or buffer gaps during incremental synthesis, directly affecting conversational naturalness in dialogue agents. Dinh et al.'s responsiveness benchmark,^[8] combining latency distribution, tail latency, and intelligibility for 13 open-source TTS systems, represents one of the first open and reproducible frameworks where latency is a primary evaluation axis.

Consistency

Once a model sounds good on one prompt, the next question is whether it stays good across many. In deployment, consistency means stable speaker identity, predictable prosody, reliable pronunciation, and reduced randomness under changing sentence structure or domain content. The literature treats this as an emerging but under-standardized category.

1. Speaker Similarity and Voice Consistency

Speaker similarity measures whether synthesized speech matches the target speaker's identity, commonly computed as cosine similarity between speaker embeddings extracted from reference and synthesized speech. Higher similarity generally means better speaker preservation, though human judgments remain important. Voice consistency extends this to check whether the same speaker sounds stable across prompts, covering timbre stability, speaking style stability, and absence of drift. A model may match a speaker well on one sentence yet slowly shift accent, energy, or vocal character across longer sessions.

2. Prosody Consistency and Turn-Taking Cues

Prosody consistency checks whether rhythm, stress, and intonation remain coherent across utterances, which is strongly relevant in multi-turn dialogue and long-form reading where sentence-level MOS routinely misses cross-utterance drift. Turn-taking cue quality tests whether speech signals continuation, completion, holding, or yielding correctly, affecting interruption timing, barge-in handling, and conversational comfort. Voice Activity Projection-based automatic evaluation^[9] makes this one of the few functionally conversational TTS metrics, with commercial systems producing reasonable but often ambiguous cues.

3. Error Severity and Time-Aligned Annotation

Severity-aware evaluation tracks not only errors but how harmful each error type is: a wrong date or amount should count more than mildly awkward prosody. Time-aligned annotation marks exactly where an audio sample fails and why, enabling targeted debugging far more effectively than a single global score. Pine et al.'s framework for use-case-specific severity ratings^[10] and VoxEvaluator's automated multi-level error detection^[11] represent the closest the field has come to standardizing this approach. IndicMOS^[12] provides learned MOS prediction for seven Indian languages using challenge evaluation data, offering scalable automated quality estimation as evaluation infrastructure for Indic TTS.

Deployment and Optimization

In research papers, a model can look excellent while being impossible to deploy within a real product budget. Industry teams care about memory, throughput, fine-tuning cost, streaming support, reproducibility, and whether the model can be adapted without full retraining. Deployment-oriented benchmarking remains less standardized than quality benchmarking, and latency dimensions are only sporadically reported even in model papers.

1. Model Size, Memory, and Throughput

Model size measures total parameter count and storage requirements, directly affecting device compatibility, cold-start time, and scaling economics in production. Memory footprint covers runtime RAM or VRAM including model weights, activations, caching, and vocoder overhead. For edge TTS and concurrent serving, memory often becomes a stricter bottleneck than parameter count alone. Throughput measures how many utterances or audio seconds a system serves per unit time, which is critical in call-center and assistant platforms where many syntheses run simultaneously under load.

2. Fine-Tuning Efficiency and Reproducibility

Fine-tuning efficiency measures how much data and compute adaptation requires. A model that adapts with little data is far easier to customize for new voices or domains, though public TTS benchmarks rarely standardize this dimension. Reproducibility of evaluation, meaning whether a study can be rerun with clearly documented protocols covering rater recruitment, instructions, filtering, scales, and test prompts, remains a serious weakness in TTS evaluation papers. Surveys of Interspeech papers^[22] show that most omit listener demographics, recruitment source, and payment details that empirically change results.

Task-Specific and Emerging Metrics

Some of the most useful modern TTS metrics do not fit cleanly inside classic quality or speed categories. They ask whether the model succeeds on the real task, survives unusual input text, and can be trusted in application contexts. This is where benchmarking is expanding most quickly, and where the gap between research practice and industrial need is most visible.

1. WER, CER, and PER

Word Error Rate measures transcription mistakes after synthesized audio is decoded by an ASR system, calculated as substitutions plus deletions plus insertions divided by the reference word count. Lower WER suggests clearer speech but still misses comprehension failures and some normalization errors. Character Error Rate operates at character level, useful for morphologically rich or script-sensitive settings where word segmentation is unreliable. Phone Error Rate measures pronunciation mismatch at the phoneme level and is especially useful when diagnosing phoneme-aware TTS pipelines or evaluating how a system handles specific pronunciation classes.

2. LLM and LALM as Judge

Model-as-a-judge metrics use large audio language models to score speech dimensions automatically, evaluating prosody, instruction-following, pronunciation accuracy, and emotional realization at scale without requiring human listeners. EmergentTTS-Eval uses a LALM-based judge; InstructTTSEval uses Gemini. VoxEvaluator detects erroneous segments automatically and supports corrective synthesis. These approaches are promising for scaling experiments but still require careful validation against human evaluation and should not entirely replace it.

3. Robustness to Challenging Text

Robustness measures whether a model handles difficult input text without collapsing or hallucinating. Critical examples include URLs, mathematical formulas, foreign words, numbers, named entities, and code-switched text. This is one of the clearest gaps in current benchmarks, especially for Indic and transactional TTS where production failures are rarely "speech sounds slightly robotic"; they are "amount spoken wrong," "name mispronounced," or "question intonation inappropriate." These failures are invisible to standard MOS evaluations.

4. Scenario-Based Suitability and Interactive Appropriateness

Suitability ratings ask whether speech fits a particular role, task, or interaction setting rather than evaluating generic quality. Contextual framing experiments show that adding situational context, such as describing the intended task or embedding TTS in actual dialogue, shifts ratings and system preferences and reveals shortcomings not visible in isolated-sentence MOS. For sales and industrial TTS, scenario-based evaluation with explicit framing around outbound calls, payment reminders, and support dialogues is essential. Interactive appropriateness measures how well speech functions during real dialogue exchange, covering turn timing, style fit, and conversational usefulness.

Existing Benchmark Datasets

TTS benchmarks fall into four historical layers: classic shared challenges, corpus-based defaults, newer evaluation-specific suites, and context-aware emerging benchmarks. These layers are not fully aligned. Some datasets are famous but not carefully designed evaluation suites; some newer suites are methodologically richer but narrow in coverage. Together they provide ingredients for rigorous evaluation, but not an integrated solution.

1. Blizzard Challenge ^[13]

Historically, Blizzard is the best-known large-scale public benchmark tradition in TTS. It established shared corpora, listening tests, and system comparisons focused on naturalness, intelligibility, and speaker similarity, shaping what the field informally meant by "benchmarking" for nearly two decades. Its structure also normalized a narrow evaluation pattern centered on clean, read, monolingual speech rather than realistic conversational or transactional content. Almost no explicit coverage of numbers, currencies, abbreviations, or code-switching appears in the Blizzard corpus design.

2. LibriTTS ^[14]

LibriTTS is a widely used corpus rather than a full benchmark protocol. Because it is public, large, and easy to compare on, many papers treat it as a benchmark anyway. Its strengths are scale, multi-speaker coverage, and reproducibility. Its weakness is that audiobook text does not systematically stress text normalization, conversational timing, or task-specific correctness, the exact challenges that matter most in production deployments.

3. TTSDS and TTSDS2 ^[7]

Among newer evaluation resources, TTSDS2 is significant because it moves beyond one-dimensional scoring. It provides multi-domain evaluation data including clean and noisy audiobooks, in-the-wild YouTube content, and children's dialogue, and introduces composite objective scoring that correlates more strongly with human judgments than classical signal metrics. Validated against 11,000-plus ratings across 20 voice cloning models and four domains, it is one of the few multi-language, multi-domain evaluation resources available, though its domains remain content genres rather than task-oriented transactional speech.

4. Audio Turing Test ^[4]

ATT shifts the evaluation question from rating quality to judging humanness through binary human-versus-machine decisions. Its corpus includes diverse styles, contexts, and trap utterances, making it stronger on human-likeness evaluation than ordinary MOS-only setups. The Auto-ATT variant fine-tunes a large audio language model to mimic ATT human judgments automatically. ATT is currently monolingual Chinese and does not target transactional or sales domains.

5. EmergentTTS-Eval, SP-MCQA-Eval, and PodEval ^{[15, 6, 16]}

EmergentTTS-Eval deliberately targets difficult prompt phenomena such as foreign words, URLs, formulas, syntactic complexity, and questions, building 1,645 test cases programmatically and using a LALM-based judge for prosody and pronunciation. SP-MCQA-Eval creates an 8.76-hour news-style dataset showing that low WER does not guarantee high key-information comprehension, exposing gaps in text normalization and phonetic accuracy in current SOTA models. PodEval brings long-form and podcast-style generation into evaluation with multimodal scoring, reminding the field that long-form synthesis is a separate evaluation regime, not simply a longer version of sentence-level TTS.

6. InstructTTSEval, VoiceAssistant-Eval, and VocalBench ^{[17, 18, 19]}

InstructTTSEval benchmarks instruction-following TTS across acoustic parameter control, descriptive style directives, and role-play in English and Chinese, using Gemini as an automatic judge. VoiceAssistant-Eval includes speaking and listening tasks, role-play, and multi-turn dialogue across 10,497 examples and 13 task categories, one of the clearest signs that benchmarking is moving toward integrated voice system evaluation. VocalBench targets spoken conversational ability across semantic, acoustic, conversational, and robustness dimensions across roughly 24,000 instances. Neither fully isolates numeric correctness, named entities, or Indic-style mixed-script complexity.

7. Bangla TTS Benchmark and Indic Vocoder Evaluation ^{[20, 21, 12]}

The Bangla benchmark is one of the clearer published attempts to compare multiple TTS models under a standardized Indic setup, reporting MOS for quality and intelligibility alongside MCD, Mel-MSE, PER, WER, SNR, and RTF across Tacotron2, FastSpeech2, VITS, and Grad-TTS. The Indic vocoder evaluation benchmarks eight non-Indic vocoders on six Indic languages and four dialects using NISQA-MOS, PESQ, MCD, MSD, STOI, and WER/CER. Both show that multi-metric evaluation is feasible for Indic speech, but neither tests domain-specific text phenomena or conversational robustness. IndicMOS provides learned MOS prediction for seven Indian languages, offering scalable automated quality estimation as evaluation infrastructure rather than a self-contained public benchmark.

How to Read a TTS Benchmark Paper

When evaluating a new TTS paper, five questions separate a research demo from evidence of deployability. Never ask only "Which model has the best MOS?" Ask instead which benchmark was used, which dimensions were measured, which failure modes were stressed, and which deployment setting the benchmark actually represents.

1. What kind of speech does the benchmark use?

Clean read speech, dialogue, long-form passages, and transactional prompts reveal very different weaknesses. A system evaluated only on audiobooks tells you almost nothing about its behavior on financial amounts or customer names.

2. What does the paper actually measure?

A paper with only MOS and MCD tells you far less than one with MOS, intelligibility, latency, and contextual evaluation. The metric selection reveals the team's implicit model of how systems can fail.

3. How carefully was the subjective evaluation run?

Listener count, instructions, filtering, recruitment source, and scale labels all materially affect results. Empirical work shows that changing the instruction from "rate quality" to "rate naturalness" can invert system rankings,^{[2, 22]} and this effect is rarely reported.

4. Were hard text cases included?

Numbers, currencies, names, formulas, URLs, and code-switching should never be treated as optional edge cases. They are the exact inputs most likely to produce real failures in production systems, and they are systematically absent from most published evaluations.

5. Does the benchmark match the intended product?

A model that wins on audiobook MOS may still lose badly in IVR, customer support, or multilingual conversational use. The benchmark must represent the deployment setting to be predictive of real performance, and most do not.

The State of the Field

Over time, TTS benchmarking has evolved from early intelligibility and total-quality testing into a much richer, though still fragmented, ecosystem of subjective tests, objective distances, learned evaluators, conversational metrics, and emerging task-based benchmarks. Blizzard normalized shared evaluation culture. MOS became the field's default language. More recent work exposed MOS's instability, added multi-dimensional scales, introduced latency and turn-taking metrics, and began testing comprehension, human-likeness, and instruction-following.

Even with that progress, the biggest gap remains practical relevance. Current public benchmarks are still dominated by clean read speech and generic quality scoring. Far fewer benchmarks test the things that matter most in production: difficult normalization, mixed-language inputs, code-switching, names, amounts, dates, latency outliers, and severity-weighted errors. That gap becomes sharper still for Indic TTS and for conversational industrial use cases where the consequences of a wrong number or mispronounced name extend well beyond a lower naturalness rating.

The tools exist: TTSDS2-style distribution scores,^[7] IndicMOS prediction,^[12] Pine et al.'s error-severity frameworks,^[10] contextual framing methods,^[8] LLM-as-judge evaluation, and turn-taking metrics.^[9] What is missing is a benchmark that combines them, designed from the ground up for Indic languages, task-oriented conversational scenarios, systematic coverage of transactional text phenomena, and task-weighted error criticality. Designing that benchmark would occupy a genuine, currently unfilled niche in TTS evaluation research and practice.

Summary

A consolidated reference across all five evaluation dimensions. Type indicates whether the metric requires human listeners (Subjective), computes from signal or model outputs (Objective), or uses a learned model as an evaluator (Automatic).