IndicTelephony-Bench: An Open ASR Benchmark for Indian Telephone Speech

Indian Phone Calls Are Hard on Several Fronts at Once

Most public benchmarks for Indian-language speech recognition are read or extempore speech, recorded through a phone app or a microphone. Most Indian voice deployments are phone calls, and a phone call is hard in four ways at once. Callers speak in short bursts (“haan”, “wait karo”, “balance batao”); they switch between English and their own language mid-sentence; what they say carries business terms, such as amounts, plan names and places, that a voice agent has to get exactly right; and all of it reaches the model through a narrowband telephone channel.

Each of these trips models up in its own way, and the benchmark measures them separately: code-mixed against monolingual turns, short turns against long ones, keyword misses against overall word error. The channel is the least visible of the four, so it is worth being precise about why it matters, because the reasons are physical rather than a matter of recording quality.

Telephone speech does exist in prior work. IndicVoices records its two-person conversations over 8 kHz telephony, about a sixth of the collection. Gram Vaani released spontaneous telephone Hindi. Voice of India evaluates fifteen languages on unscripted phone conversations, but keeps its test set closed. What was missing is an open, telephony-only benchmark, split into single turns across many languages, with the business vocabulary a voice agent acts on tagged in every reference.

IndicTelephony-Bench is built for all four conditions together. Every file was recorded over a live SIP telephone line and is released exactly as the line delivered it: 8 kHz, mono, 16-bit, never resampled.

What Is in the Benchmark

25,393 single-speaker utterances (30.18 hours) across Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Tamil and Telugu. Every language has at least three hours. Utterances are short, with a median of 3.1 seconds, because a telephony system decodes one conversational turn at a time.

Every utterance carries a business domain. Automotive and real estate lead, followed by banking, telecom, insurance, e-commerce, education and healthcare, and 66.0% of utterances carry curator-tagged keywords: the policy numbers, loan amounts, plan names and car variants a support call turns on.

Code-mixing is the norm, not a special case. Outside English, most utterances carry English words, and a system has to keep them in Latin script while writing the rest in the native script, inside one breath:

Listen to the Benchmark

One turn from each language, played from the release files themselves: 8 kHz, straight off the phone line. English words are shown in blue and native-script words in violet; outlined words are the turn's tagged keywords, the terms KWER checks.

How We Built It

The goal was real telephone acoustics carrying real business language. The text comes from real customer-service and sales calls across many domains (banking, insurance, real estate, automotive, telecom, e-commerce, education, healthcare). Curators transcribed each source call by ear and anonymized it, replacing every name, number and address with a placeholder; the source audio is never released. The transcripts were then cut into turn-sized chunks and shuffled, and two speakers read them aloud to each other over a live phone line. The wording, fillers and code-mixing are those of real calls, and the channel and codec are genuine; the delivery is read, and the turn order is shuffled.

Every transcript follows one written convention, and the scoring code applies the same convention to every system:

Scoring Code-Mixed Speech Fairly

Reference and hypothesis pass through one published normalizer before any metric is computed, so no system is charged for differences of writing convention rather than recognition. It never transliterates between scripts, because that would forgive real errors. Everything below is measured after that step.

We define five metrics and specify all of them, and report two companions alongside: macro WER and keyword precision. The five are lower-is-better and computed at corpus level (total errors over total reference units, not an average of per-utterance rates).

Baselines: Five Systems, Every Utterance

We ran five contemporary speech-to-text systems, one per vendor: Akshara (ConvoZen AI), Saaras v4 (Sarvam), Scribe v2 (ElevenLabs), Pulse (Smallest.ai) and Nova-3 (Deepgram). Each received the released 8 kHz audio (resampled to the model's 16 kHz for Akshara) and the utterance's base language wherever its API accepts one; Akshara received audio only, with no language hint. None of the benchmark audio, transcripts or speakers, nor the source calls, was used to train, tune or select Akshara. All systems were queried once per utterance in August–September 2026. Nova-3 does not support Malayalam, so it is scored on the other eight languages. A turn a system returned nothing for is scored as fully deleted. Lower is better.

All values in %. Bold marks the best value per column; on narrow screens only WER, KWER and semantic WER are shown. Macro: per-language WER averaged with every supported language weighted equally. KwP and KwF1 (keyword precision and F1) are higher-is-better; every other column is lower-is-better. Intervals: 1,000 bootstrap resamples over calls, because turns within a call share a line and speakers. KWER is scored on the 35,881 tagged keywords that occur in their reference transcripts, and on 31,600 for a system that does not cover every language.

What Semantic WER Changes

Semantic WER forgives a difference only when a check confirms the two spellings are the same word. It is not a kinder WER applied evenly: recovery tracks how much lexical error a system had to begin with, so Nova-3 gains 14.9 points and Akshara only 4.2 points, and the leader's margin over second place narrows from 6.9 points to 4.0 points.

One consequence: Pulse and Scribe v2 become indistinguishable, 0.3 points apart with a paired interval that includes zero. Every other gap survives. Because an LLM proposes the candidate pairs, this is the one number here that is not perfectly repeatable: two runs of the judge moved a system by at most 0.48 points. WER, CER, MER and KWER involve no model at all.

Because Nova-3 skips Malayalam, the like-for-like ranking uses the eight languages every system supports: Akshara 8.9%, Saaras v4 15.8%, Scribe v2 22.5%, Pulse 24.7%, Nova-3 31.5%. Every step in that ranking holds up statistically: scoring all systems on the same resampled calls, the 95% interval of each gap excludes zero.

Language-Specific Breakdown

Word error rate per language. Compare systems within a row, not languages with each other: each language was recorded on its own small set of lines, so a harder row may reflect its speakers as much as its language. Lower is better.

Where Systems Break: Code-Mixing and Short Turns

Two properties of phone traffic pull the systems apart in ways the corpus average hides. On code-mixed turns a model must switch scripts mid-utterance, and on turns under two seconds it has almost no acoustic or linguistic context to lean on. 30.0% of the benchmark sounds like this:

The two slices are not the same weakness. Code-mixing costs Nova-3 16.5 points, from 18.1% on monolingual turns to 34.5% on mixed ones, while Akshara moves only 0.6 points. Short turns behave differently again: going from turns of ten seconds or more down to turns under two seconds costs Scribe v2 14.4 points, while Saaras v4 is 2.0 points better on the short turns. A system can be robust to one and not the other, which is why a single corpus number is a poor guide to how a model will behave on your traffic.

Keywords sharpen the picture further. Measured against its own word error rate, Nova-3 misses tagged keywords at 1.69 times that rate (53.1% KWER against 31.5% WER). A transcript can read plausibly and still drop the amounts, product names and places the rest of the pipeline acts on. The reverse failure is rare: keyword precision, the share of domain keywords a system writes that were actually said, is between 94.7% and 98.4% for every system. Systems differ in the keywords they miss, not in keywords they invent.

Results by Domain

Word error rate and keyword miss rate per business domain. Domains overlap with languages (each language's calls lean towards a few scenarios), so read this as where each system's errors land, not as how hard a domain is in itself. Lower is better.

What This Benchmark Is Not

A benchmark whose limits are discovered by its users is worse than one that states them.

1. 01

   Read, not spontaneous. The wording comes from real business calls, anonymized, but each turn was read aloud in shuffled order. The line, channel and codec are real; spontaneous disfluencies, overlaps and turn-taking are not.

2. 02

   Mostly clean telephony. Most turns are clear, a few carry background noise or line artefacts. Sustained street or office noise is not covered.

3. 03

   Language and line move together. Each language was recorded on essentially one of nine phone lines by few speakers, so speaker and line vary along with it. Compare systems within a language.

What Is Released

@misc{indictelephonybench2026,
  title  = {IndicTelephony-Bench: An Open ASR Benchmark for Indian Telephone Speech},
  author = {Priyadarshi, Prasoon and Abdul Azeez, Zaheer and {ConvoZen AI}},
  year   = {2026},
  note   = {25,393 utterances, 30.18 hours, nine languages, 8 kHz telephony}
}