Mapping Geographical Variation in Indic Speech: Joint Language–District Representation Learning Leveraging the Vaani Dataset

Self-supervised speech encoders are routinely fine-tuned with language-only supervision, an approach that does not include the rich geographical variation within each language. Researchers at ARTPARK and IISc set out to test whether finer-grained supervision could recover that lost structure. Using a subset of the Vaani corpus spanning 60 Indic languages across 165 districts (386 language–district classes), they fine-tuned Whisper-base and Wav2Vec 2.0-base under two objectives, joint language–district classification versus language-only classification, and probed the resulting embedding geometry.

The result: joint supervision lifted within-language district probing F1 to 91.29 for Whisper-base (vs. 80.89 for language-only) while holding marginal language accuracy at ~84.79%, essentially matching the language-only model on language identity while additionally encoding district-level structure that language-only models cannot represent. Crucially, this hierarchical geometry, in which districts cluster locally within globally separated languages, emerged implicitly, with no hierarchical constraint imposed during training. The study establishes that Vaani's district-level metadata, available at scale across dozens of languages, is what makes this class of analysis possible at all.

This work is available as a preprint (arXiv:2606.19940).

Modern speech systems depend on embedding spaces that capture structured acoustic and linguistic information. Prior work has shown these spaces encode typological relationships and fine-grained pronunciation variation. Yet most Indic language-identification research forces an uncomfortable choice, and several failure modes follow:

• Geography treated as noise: When models are supervised only at the language level, systematic within-language regional variation collapses, even though district-level and dialect-level differences are structured signal, not random error.
• The coverage and granularity trade-off: Existing studies either scale to large language inventories with supervision only at the language level, or restrict to a handful of languages to study dialect in detail. Capturing systematic within-language geographical variation at scale remained underexplored.
• Opaque embedding geometry: How the granularity of supervision actually reshapes the internal structure of multilingual embeddings, and whether finer labels help or hurt language separability, was not well understood.
• A scarcity of fine-grained geo-metadata: Generic public speech corpora simply lack district-level labels spanning many languages, making the question impossible to study without a purpose-built resource.

For a country whose languages shift every few kilometres, this gap matters. It caps how well speech models can serve regional and dialectal diversity in downstream tasks like ASR and text-to-speech.

The team hypothesized that joint language–district supervision could organize geographical structure in a linearly accessible way without sacrificing language separability. They built a controlled experiment to isolate the effect of supervision granularity from data scale.

Data Strategy with Vaani

Vaani's fine-grained geographical metadata was the enabling ingredient. From the corpus, the researchers selected 60 languages across 165 districts, yielding 386 balanced language–district classes at roughly 3 hours each (1,158 hours total), split at the speaker level (about 2.4 h train, 0.2 h validation, and 0.4 h test per class) to prevent speaker leakage. To separate granularity from scale, they also built two language-only regimes: a balanced set (3 h per language, 180 h, uniformly sampled across districts) and an unbalanced set (the same 1,158 h with labels collapsed to language).

Models and Supervision Settings

Two complementary encoders were fine-tuned end-to-end. Whisper-base uses log-Mel input, embedding dimension D=512, and multilingual pretraining including Indic data, while Wav2Vec 2.0-base uses raw waveform input, D=768, and English-only pretraining. Frame-level outputs were converted to utterance embeddings via attention-based temporal pooling. Three supervision settings were compared: L-60 (language-only, 60 classes), LD-386 (joint language–district, 386 classes), and L-60-FD (language-only over the full LD-386 data, districts collapsed).

Measuring Structure

Beyond classification accuracy, the team used language-conditioned logistic-regression probing to test whether district identity is linearly recoverable within each language's subspace, and used Normalized Conditional Mutual Information (NCMI), evaluated across neighborhood scales k, to characterize how district and language structure are organized locally versus globally. All training used AdamW on a single NVIDIA L4 GPU.

Joint supervision preserved language identity while unlocking district structure that language-only training cannot represent. On marginal language accuracy, Whisper-base LD-386 (84.79%) essentially matched the language-only L-60-FD model (84.77%) while additionally predicting district at 47.09% accuracy.

The clearest signal came from language-conditioned district probing, which measures how cleanly district structure separates within each language. Joint supervision produced the strongest district geometry for both encoders:

A revealing contrast: for Wav2Vec 2.0-base, simply scaling language-only data (L-60-FD) collapsed district structure to 58.53 F1, proof that more data is not a substitute for finer supervision. Across 114,596 test samples, language-conditioned NCMI analysis showed LD-386 consistently achieving higher district-level NCMI at small neighborhood sizes, confirming that districts cohere locally while languages remain separated globally, a hierarchy that arose without any explicit hierarchical training objective.

• Geography becomes linearly recoverable: Joint supervision organizes district variation so it is linearly separable within each language's embedding subspace, turning regional identity from discarded noise into usable structure.
• Granularity beats raw scale: The collapse of district structure under scaled language-only training (Wav2Vec 2.0-base, 58.53 F1) demonstrates that how data is labelled shapes embedding geometry more than how much data there is.
• Pretraining alignment matters: Whisper-base, pretrained on multilingual (including Indic) speech, encodes geography more stably than the English-pretrained Wav2Vec 2.0-base, offering a practical guide for model selection in Indic settings.
• Hierarchy for free: A clean district-within-language hierarchy emerges implicitly from joint supervision, suggesting a path to representation sharing across related regional varieties.

• Proof of value: This analysis is only possible because Vaani supplies district-level metadata at scale, covering 60 languages across 165 districts with balanced per-class data. No generic corpus offers this combination of breadth and geographical granularity.
• A blueprint for low-resource India: The structured within-language subclusters point toward sharing representations across language–district pairs, a promising direction for the long tail of under-resourced regional varieties.
• A catalyst for regionally aware Indic AI: Better geographical encoding feeds directly into the speech tasks that matter for Bharat, such as dialect-sensitive ASR and natural, regionally appropriate text-to-speech.

The study lands on a simple principle: finer supervision yields richer embedding geometry with no loss in language separability. By supervising jointly on language and district, the researchers showed that multilingual speech embeddings can carry both global language identity and local geographical structure at once, and that the structure organizes itself hierarchically when provided with district and language labels.

This capability directly enables:

• Region-aware representations that distinguish districts within a language rather than averaging them away.
• Smarter model selection for Indic speech, favouring multilingually pretrained encoders for geographical fidelity.
• A foundation for downstream gains in low-resource language–district modelling, ASR, and TTS across India's linguistic landscape.

By leveraging the Vaani dataset's unique geographical depth, ARTPARK and IISc have turned a long-standing blind spot in multilingual speech modelling, within-language regional variation, into a measurable, recoverable, and useful signal.