Identifying 42 Indian Languages: How Encoder and Objective Choices Shape Indic Spoken Language ID, Leveraging the Vaani Dataset

Spoken language identification (LID) is the front door of any multilingual speech pipeline: it routes incoming audio to the right language-specific ASR or translation model. For India it is also genuinely hard, because there are many languages, heavy phonetic overlap between closely related varieties, and little labeled data for the long tail. Researchers at ARTPARK and IISc ran a controlled comparison to answer two practical questions: which pre-trained encoder is best for large-scale Indic LID, and which training objective generalizes across corpora. They evaluated two encoders (Whisper and FastConformer) and three objectives (cross-entropy, contrastive plus cross-entropy, and hierarchical softmax) over 42 languages across four linguistic families, training on Vaani and testing cross-corpus on Vaani-Test, FLEURS, and Kathbath.

The standout result: a frozen FastConformer encoder reached over 90% macro accuracy on both FLEURS (94.2%) and Kathbath (90.9%) with no task-specific adaptation, substantially beating even a fine-tuned Whisper out-of-domain. Hierarchical softmax (HSM) was the best objective for both encoders across every benchmark, with the largest gains out-of-domain, while the contrastive objective actually hurt FastConformer's cross-corpus generalization. A per-family breakdown pinpoints the remaining frontier: Central Indo-Aryan varieties are the hardest to tell apart (58.7%), with Hindi-Urdu and the Sadri-Chhattisgarhi-Surgujia cluster driving most errors.

This work is available as a preprint (arXiv:2606.09317). The FastConformer encoder evaluated here is itself a Vaani-trained model (ARTPARK-IISc / Vaani-FastConformer-Multilingual).

India has over 1,600 mother tongues and 22 constitutionally recognized languages spanning the Indo-Aryan, Dravidian, and Sino-Tibetan families. Building a reliable LID front-end across this diversity runs into several obstacles:

• Severe phonetic overlap: Closely related varieties such as Hindi, Bhojpuri, Chhattisgarhi, and Sadri share strong phonological and lexical structure, making fine-grained discrimination difficult.
• Low-resource long tail: Many target languages have very limited labeled training data, so methods that lean on abundant supervision do not apply evenly.
• Cross-corpus domain gap: Public benchmarks differ in recording conditions, speaking style, and speaker demographics, so a model that scores well in-domain can fail on out-of-domain audio.
• Unsettled design choices: It was unclear which encoder architecture suits Indic LID, and how much the training objective (standard cross-entropy, contrastive metric learning, or hierarchical softmax) affects accuracy and generalization.

Prior Indic LID studies mostly evaluated in-domain and on a handful of languages, leaving these encoder and objective questions open at scale.

The team built a deliberately simple and controlled setup: a pre-trained speech encoder produces frame-level features, self-attention pooling forms a fixed-length utterance embedding, and a single linear layer predicts one of 42 languages. By holding this architecture constant, differences in results can be attributed to the encoder and the training objective rather than to bespoke modeling tricks.

Data Strategy with Vaani

Vaani's scale and diversity made a balanced large-language-set experiment possible. The researchers curated 10 hours of speech for each of the 42 languages, maximizing district and speaker diversity, and split the data 8:1:1 into train, validation, and test with no speaker overlap. They then adopted a cross-corpus evaluation protocol, training only on Vaani and testing on three benchmarks: Vaani-Test (in-domain held-out), 13 languages from FLEURS, and 11 from Kathbath.

Encoders and Training Objectives

Two encoders were compared: Whisper (the openai/whisper-medium encoder, 350M parameters, 1024-dimensional, pretrained on 90+ languages) and FastConformer (the ARTPARK-IISc Vaani-FastConformer-Multilingual encoder, 430M parameters, 1024-dimensional, pretrained on 60+ Indic languages). Each was tested frozen (only the final two layers trained, i.e. linear probing) and fine-tuned end-to-end. Three objectives were compared on both backbones: cross-entropy (CE); CE plus supervised contrastive loss (CE+SupCon), which also shapes the embedding space (weighting 0.5, temperature 0.07); and hierarchical softmax (HSM), which factors prediction along a four-level Root, Family, Sub-family, Language tree. Models used AdamW on a single NVIDIA L40 GPU, and performance was reported as macro-averaged accuracy so every language counts equally.

The two encoders behaved very differently. Fine-tuning helped Whisper substantially on every benchmark, but for FastConformer it barely moved in-domain accuracy and clearly hurt out-of-domain performance, so the frozen FastConformer was the stronger generalizer.

Strikingly, the frozen FastConformer (FLEURS 94.2%, Kathbath 90.9%) outperformed the fine-tuned Whisper (72.7% and 68.3%) out-of-domain, evidence that its Indic pre-training produces more domain-robust representations. Across training objectives, HSM was best for both encoders on all three benchmarks, with its largest improvement on Kathbath.

External baselines are not directly comparable: FB MMS supports 30 of the 42 languages and SpeechBrain ECAPA supports only 13, which inflates their narrower-set scores and explains MMS's low Vaani-Test result.

For Whisper, HSM lifted Kathbath by 7.5 points over CE, and for FastConformer the CE+SupCon objective degraded cross-corpus accuracy (FLEURS fell from 89.9% with CE to 82.7%), indicating the contrastive loss over-specializes the embedding space to Vaani's acoustic conditions. Notably, FastConformer with HSM matched SpeechBrain ECAPA on FLEURS (91.4% vs. 91.3%) and beat it on Kathbath (90.0% vs. 87.9%) while covering over three times as many languages.

A per-family breakdown (HSM) shows where the difficulty concentrates. Sino-Tibetan, English, and Dravidian languages are well-identified, while Central Indo-Aryan is by far the hardest sub-family.

The error analysis traces the Central Indo-Aryan deficit to two confusion patterns shared by both encoders: Hindi and Urdu are heavily mutually confused (expected, given their near-identical spoken form), and the Sadri-Chhattisgarhi-Surgujia cluster blurs together due to heavy phonological overlap. By contrast, acoustically distinct groups like Sino-Tibetan (97.1%) and Dravidian (85.9%) are recognized well even when their training data is limited, underlining that inter-language acoustic distinctiveness, not just data volume, drives LID accuracy.

• The right frozen encoder beats a fine-tuned general one: A domain-robust, Indic-pretrained encoder used as a frozen feature extractor (FastConformer) generalized better out-of-domain than a fine-tuned multilingual model (Whisper), a useful default for practitioners.
• Objective choice is not neutral: A linguistically structured objective (hierarchical softmax) consistently won across encoders and benchmarks, while a contrastive objective could over-specialize representations and hurt cross-corpus transfer.
• In-domain and out-of-domain pull in different directions: Fine-tuning Whisper helped in-domain but not across corpora; teams must choose based on whether deployment audio resembles training audio.
• Linguistic proximity is the bottleneck: Errors concentrate in Central Indo-Aryan, especially Hindi-Urdu and the Sadri-Chhattisgarhi-Surgujia cluster, giving a precise target for future work.

• Proof of value: Vaani made a balanced 42-language study feasible, supplying 10 hours per language across four families with wide district and speaker diversity, a scale of diverse Indic LID not attempted before.
• A blueprint for honest evaluation: Training on Vaani and testing on FLEURS and Kathbath provides a realistic cross-corpus yardstick for generalization, and the Vaani-trained FastConformer encoder is itself a reusable building block for Indic speech systems.
• A catalyst for focused effort: By isolating Central Indo-Aryan as the frontier, the study directs data collection and family-aware training toward the varieties that matter most for inclusive, nationwide LID.

The study offers a clear, deployable recipe: for large-scale Indic LID, pair a domain-robust Indic-pretrained encoder with a linguistically structured objective, and keep that encoder frozen when out-of-domain robustness matters. Fine-tuning a broad multilingual model can still win in-domain, but it trades away cross-corpus generalization. The remaining hard problem is the set of closely related Central Indo-Aryan varieties.

This capability directly enables:

• A reliable routing front-end that sends multilingual audio to the correct language-specific model.
• Realistic cross-corpus evaluation that reports generalization rather than only in-domain scores.
• Targeted improvement of the confusable varieties that currently limit nationwide accuracy.

By leveraging the Vaani dataset's breadth and the Vaani-trained FastConformer encoder, ARTPARK and IISc turned an open question about architectures and objectives into concrete, reproducible guidance for building Indic language identification at scale.