Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Authors: AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan. NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya.
We're excited to introduce Nemotron-pretraining-Indic-v1 approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it. These are the 22 languages listed in India's Constitution: the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census. Developed by NVIDIA with IIT Madras, India's premier engineering institute, and its AI4Bharat lab, this release gives model builders a substantial new resource for Indic pretraining. The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their own source material and generate more data for their needs.
AI4Bharat, IIT Madras' Indian language research lab, builds open-source datasets and models for Indian languages. Its resources, such as Sangraha, an open pretraining corpus for 22 Indian languages, have helped lay the foundation for open Indic language-model research. Our joint open-source release represents a substantial expansion of the open synthetic Indic training data that will be available to the community. It brings a broad mix of curated English material, including mathematics, code, and question-answer content, into Indian languages, giving teams more training material for building models that can explain technical concepts, work through problems, and answer questions in those languages. Every record traces back to a curated source document, and each output is checked against its input, so the synthetic text stays grounded in real information.
Mitesh Khapra, who heads AI4Bharat and made the 2025 TIME100 AI list, has named the core problem: Indian language technology trails English because "we do not have enough data for Indian languages." We're answering that call with more data, plus an open recipe for generating more.
The corpus also accounts for how people write. It includes native-script data for all 22 languages and romanized text for twelve, spanning 34 language-script configurations. Our workflows combine document-level translation, quality assessment, and targeted repair, with checks to preserve content such as equations, tables, and code. This data complements naturally authored Indic text, which remains essential for local knowledge and cultural context.
We're sharing the open generation recipe so the community can not just reproduce the data but take this work further. Teams can use the corpus directly, adapt the prompts and quality checks, or bring their own source documents to the workflow. By opening up both the data and the process, we aim to make large-scale Indic data generation more accessible and enable more developers to contribute to stronger models across Indian languages.
Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens, drawn from a twelve-language subset of this corpus and natural Indic text. Our technical report on the model covers the full story. We plan to build on this corpus in future Nemotron models.
What is in the corpus?
Figure 1. The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Counts use the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they are not counts of unique English documents.
The corpus has two groups of language configurations:
- Native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.
- Native-script data: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.
Here, native-script describes the writing system: the text is still translated from English. Romanized means Indic-language text written in Latin script, not an English translation. For each of the twelve dual-script languages, romanization covers a subset of the accepted native-script translations.
This distinction matters when constructing a training mixture. Native and romanized records offer different script exposure, but can express the same underlying content. The token counts describe the retained corpus, including accepted rewrites, rather than a training run's token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately.
From English source documents to Indic training data
To build this corpus, we start with documents from 15 Nemotron-CC-v2 subsets, covering web text, question-answer material, mathematics, code, and other specialized content. We use NeMo Data Designer, NVIDIA's open-source framework for generating synthetic data, to run translation, quality checks, and targeted repair. Throughout the workflow, we retain each source identifier so developers can trace an output back to its original document. Clear provenance like this is part of what makes open data practical to audit and build on. Figure 2 shows the workflow for twelve languages.
Figure 2. Four-stage Indic synthetic-data pipeline for twelve languages, implemented using NeMo Data Designer and Gemma-4-31B-it. We translate English documents, assess translation quality, repair low-scoring outputs, and convert a subset of accepted translations into Latin script. The other ten languages use the separate Indic Translate workflow described below.
The twelve-language workflow
For these twelve languages, we translate documents in full so the model can use context across paragraphs, tables, and equations. We selected Gemma-4-31B-it by comparing models on complete documents, checking translation quality and preservation of technical content. Our prompts distinguish prose to translate from content to preserve, including code, equations, URLs, and formatting.
After translation, we use the same model to assess each output against its English source. A rubric inspired by Multidimensional Quality Metrics (MQM), an open framework for classifying translation errors by type and severity, covers meaning, fluency, terminology, style, and formatting. We retain only outputs scoring at least 75 out of 100 (a rubric score, not an accuracy percentage). For lower-scoring outputs, we use the assessment feedback to guide repairs, then reassess them against the same threshold. This process recovered 438.4 billion tokens across native-script and romanized data, already included in the corpus total. Sampled human reviews and tests with deliberately introduced errors helped identify omissions and residual English, informing improvements to our prompts and checks.
Accepted native-script translations also provide the starting point for romanized data. We convert a subset into Latin script and check each result against its native-script input. This separates translation from script conversion. In our initial human comparison, this two-step approach produced fewer hallucinated additions and less residual English than generating romanized text directly from English.
The ten-language workflow
For the other ten languages, we adapt the workflow to address long-document quality and script-stability issues observed in our comparisons. We translate with Indic Translate, a document-level translation model for English and 22 Indian languages from Bodhan AI, a nonprofit center for AI in education incubated at IIT Madras. Gemma-4-31B-it then assesses each translation against its English source. For eight languages, we keep the whole-document approach. For longer Santali documents, we translate overlapping windows and reconstruct the document; for Manipuri, we translate into Bengali script before converting to Meitei Mayek, its native script. These workflows retain outputs scoring at least 75 out of 100 and produce native-script data only, without feedback-based rewriting.
What changed after Indic continued pretraining?
The Nemotron 3.5 Super VL technical report compares the checkpoint immediately before Indic continued pretraining (CPT) with the checkpoint after that stage. The configured budget was 100 billion training tokens, with 10% (10 billion tokens) allocated to a mixture of synthetic and natural Indic text. The remaining 90% maintained the preceding training mixture's relative proportions.
Scores increased on all five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K.
Figure 3. Nemotron 3.5 Super VL before and after the Indic CPT stage. Gains range from 4.98 to 9.82 percentage points. The comparison reports the change across a training stage that combines synthetic and natural Indic data, rather than isolating the synthetic corpus's contribution. Language coverage differs by benchmark; these are stage-level checkpoints, not a comparison of final model releases.
Source: Nemotron 3.5 Super VL technical report, Table 2. GSM8K excludes Odia because evaluation exceeded the serving context limit.
These gains support the combined synthetic-and-natural Indic CPT recipe, while leaving the choice of mixture for other models and languages to further study.
The companion Indic report examines source mixtures, synthetic-data pool size, and script coverage using first-pass accepted data for the twelve-language group. Evaluating the full corpus, including recovered rewrites and the other ten languages, remains a next step for those controlled studies.
Broader Indic capabilities in Nemotron 3.5 Super VL
Beyond the CPT-stage results, how does the released model compare with the previous generation? Figure 4 compares Nemotron 3.5 Super VL with Nemotron 3 Super across Indic task performance and language quality. These results reflect the released model's full training pipeline, including Indic CPT and post-training.
Nemotron 3.5 Super VL scores higher on five of the six task benchmarks shown, spanning knowledge, reasoning, mathematics, code, and translation. Instruction following on IFEval 1.5 dipped 2.04 points. It also improves across all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is Sarvam AI's open benchmark of assistant-style prompts in Indian languages. Together, these evaluations examine both what the model can do and how well it communicates in Indian languages.
Figure 4. Indic task performance and language quality across Nemotron Super generations. Left: six task benchmarks. Right: preliminary IndiVibe conditional preference scores, not direct head-to-head win rates. Labels show changes in percentage points, except WMT24++, which uses chrF points. See the companion Indic technical report for benchmark and evaluation details.