Synthesizing natural speech for Indian-language voice agents is not primarily a fidelity problem. Modern neural systems already produce clean, human-sounding audio on well-formed sentences. The failures that matter in production are narrower and harsher: a ten-digit account number read as a compound quantity, an abbreviation like NEFT pronounced as a word, a brand name mangled, or a model that loses the speaker's identity the moment an English clause appears inside a Hindi sentence.
These behaviours are invisible to a benchmark built from clean, natural sentences, because every competent system scores near zero error on such text. Ranking therefore requires deliberately adversarial input alongside the natural case. The evaluation below is constructed on that principle.
Before the tables, the audio. Four clips in the Sherin voice — banking and property-services copy in English, Hindi, and Hindi–English code-mix.
Select a sample to listen to generated voice output:
“Your home loan application has been approved, and the sanction letter will reach your registered email address by this evening.”
“आपकी जमा राशि अगले सोमवार को परिपक्व हो रही है, ब्याज सहित पूरी रकम आपके बचत खाते में अपने आप आ जाएगी।”
“आपका home loan pre-approved है, property documents verify होने के बाद disbursement उसी हफ़्ते शुरू हो जाएगा।”
“Sure, I can arrange a site visit for that two bedroom apartment on Saturday morning, and the builder will share the floor plan before you arrive.”
Systems compared
Five systems, forty utterances per language, two hundred in total. The text set is held identical across every system, so differences are attributable to the synthesizer rather than to what it was asked to say.
The benchmark data is taken from Sarvam AI's tts-general-benchmark, with sampled data from AI4Bharat's IndicVoices-R.
| Model | Engine | Voice | Languages |
|---|---|---|---|
| Ragini | — | Target speaker | 5 |
| ElevenLabs | eleven_v3 | Matched clone | 5 |
| Cartesia | sonic-3 | Native per language | 5 |
| Google Chirp | Chirp 3: HD | Single name, all locales | 5 |
| Sarvam | bulbul:v3 | Native, male | 5 |
Intelligibility and speech round-trip accuracy
Each system's output is transcribed by a single fixed ASR model with the language forced per utterance, and the transcript compared against the reference text. WER and CER are corpus-level within each language — total edit distance divided by total reference length, rather than a mean of per-utterance rates, so one short bad utterance cannot dominate. PER measures phone-level distance after phonemisation, which partially credits near-miss pronunciations instead of scoring a whole word wrong. The headline figures below are the unweighted mean of the five per-language rates, so no single language dominates the ranking purely by having longer references; the per-language breakdowns follow.
| Model | WER ↓ | CER ↓ | PER ↓ |
|---|
Per-language results
Word error rate for every system across the five target languages. English is included as a control: if all systems cluster near zero there, the English half of the text set is saturated and is not ranking anything.
| Model | English | Hindi | Tamil | Telugu | Kannada |
|---|
English is indeed saturated: every system lands between 0.042 and 0.090, and three of the five tie exactly, so the control ranks nothing. Hindi separates the systems only mildly and the three Dravidian languages carry the entire ordering — Telugu is the hardest for all five, with no system below 0.44.
The same breakdown at character and phone level. These are the informative views for Indic scripts: a single wrong vowel sign fails a whole word under WER while barely moving CER, so a large WER gap that does not reproduce here is an inflection error rather than a system mishearing the sentence.
| Model | English | Hindi | Tamil | Telugu | Kannada |
|---|
| Model | English | Hindi | Tamil | Telugu | Kannada |
|---|
They largely agree. Ragini and Sarvam tie at the top on averaged PER (0.068), with Sarvam ahead on CER. Ragini is the strongest system on Telugu under all three metrics, and by the widest margin at phone level (0.122 against 0.158 for the next system) — the language where every system is weakest is the one where it leads most clearly.
What these numbers cannot tell you
The ASR is a measuring instrument, not ground truth
Every error rate here is mediated by a single ASR model. Where it mishears correct audio, that scores against the synthesizer; where it silently repairs a mispronunciation into the intended word, the error vanishes. Transcripts are released alongside the scores so individual failures can be inspected rather than assumed.
Naive text normalisation destroys Indic scripts
The standard Whisper BasicTextNormalizer replaces every Unicode Mark character with a space. In Devanagari, Tamil, Telugu and Kannada the vowel signs are marks, so it silently rewrites कॉल as क ल and inflates error rates for every system simultaneously. This evaluation strips only punctuation and symbols and preserves marks.
Speaker identity is not held constant
Only Ragini and ElevenLabs speak with a matched voice. Cartesia and Google have no clone of the target speaker, and Sarvam's is a different gender. This does not affect WER, CER or PER, which measure intelligibility — but any speaker-similarity sub-score is not comparable across these systems and must not be read as one.
Sample size
40 utterances per language per system is adequate for error rates, which aggregate over hundreds of words. It is thin for any distributional metric, which compares whole populations of speech features. Treat small gaps in such scores as noise and read the standard deviations before ranking on them.
Reproducing
The full 200-row benchmark text set, per-utterance scores and ASR transcripts are released as CSV. Because every system speaks identical text, adding a new system requires only generating those 200 utterances and rerunning the scoring step.