NVIDIA and AI4Bharat Launch Indic Pretraining Corpus
NVIDIA and IIT Madras' AI4Bharat lab have unveiled a major new resource for Indic language models, featuring approximately 2.8 trillion tokens and an open, GPU-accelerated generation pipeline.
Radwan AbdulhadiEditor and technology writer
Expanding Indic Language Model Resources
NVIDIA and IIT Madras' premier engineering institute, through its AI4Bharat lab, have introduced Nemotron-pretraining-Indic-v1. Detailed on the [Hugging Face Blog](https://huggingface.co/blog/nvidia/scaling-indic-pretraining), the new release brings approximately 2.8 trillion tokens across 22 Indian languages. These 22 languages correspond to those listed in India's Constitution, representing the first languages for 97% of Indian citizens, or nearly 1.2 billion people based on the 2011 Census.
AI4Bharat, headed by Mitesh Khapra, has long worked to build open-source datasets and models for Indian languages. Khapra has previously emphasized that Indian language technology trails English due to a fundamental lack of sufficient data for Indian languages. This joint initiative directly targets that gap by providing substantial new training material.
Corpus Composition and Script Configurations
The newly released corpus accounts for diverse writing systems, spanning 34 language-script configurations. It features native-script data for all 22 languages and romanized text for 12 languages. The dataset includes a broad mix of curated English material covering mathematics, code, and question-answer content, translated into Indian languages to assist models in explaining technical concepts.
The corpus encompasses 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted using the Gemma-4 tokenizer. The languages featuring both native-script and romanized data include Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu. Languages featuring native-script data exclusively include Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.
The Open Data Generation Pipeline
Alongside the corpus, the collaboration is releasing an open, GPU-accelerated data generation pipeline. The recipe captures the full generation workflow, including prompts, large language model workflows, and quality checks, allowing developers to adapt the pipeline to their own source material. Developers can follow updates and developer discussions through contributors such as [Will Jennings WillJenningsDC Follow](https://huggingface.co/WillJenningsDC).
The workflow utilizes NeMo Data Designer, an open-source framework by [nvidia](https://huggingface.co/nvidia) designed for synthetic data generation. The pipeline handles document-level translation, quality assessment, and targeted repair while preserving content like equations, tables, and code.
Translation, Quality Assessment, and Provenance
To construct the corpus, the team began with source documents from 15 Nemotron-CC-v2 subsets, covering web text, math, code, and specialized content. The developers selected Gemma-4-31B-it after comparing models on complete documents to check translation quality and technical content preservation. Prompts were specifically structured to distinguish prose requiring translation from code, equations, URLs, and formatting that needed to remain untouched.
Following translation, the system evaluates each output against its English source using a rubric inspired by Multidimensional Quality Metrics. Only outputs scoring at least 75 out of 100 are retained. Lower-scoring outputs undergo targeted repairs guided by assessment feedback and are subsequently reassessed against the same threshold. Every individual record traces back to a curated source document to maintain clear provenance for auditing.
Integration Into Nemotron Models
The newly generated data has already begun finding practical application within NVIDIA's ecosystem. Continued pretraining for Nemotron 3.5 Super VL successfully integrated 10 billion Indic tokens drawn from a 12-language subset of this corpus alongside natural Indic text. The technical report on the model covers the full deployment story, and the creators plan to expand upon this corpus in future Nemotron model iterations.
Sources
- Hugging Face BlogScaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Continue chronologically
Related entity coverage
Related stories
NVIDIA Fine-Tunes Nemotron for Gold-Level IMO and IOI Results
NVIDIA demonstrates how fine-tuning the Nemotron model family using SFT, RL, and inference loops achieved gold-medal levels at IMO 2026 and IOI 2026.
NVIDIA Expands Nemotron 3.5 with New Retrieval Models
NVIDIA expands its Nemotron collection with open models for embedding, reranking, and document parsing to enhance enterprise agent workflows.
NVIDIA Launches Nemotron 3.5 Transcribe and Granary v2
NVIDIA introduces Nemotron 3.5 Transcribe and Granary v2, delivering advanced multilingual transcription, speaker attribution, and promptable audio tools.
Nvidia Developer Builds 'MonsterHouse' Halloween Voice Agent
An Nvidia developer creates MonsterHouse, an interactive Halloween voice agent running locally on a single workstation using advanced AI models.
