# Sangraha attribution and changes

Corpus source: AI4Bharat, Sangraha (IndicLLMSuite).
Source URL: https://huggingface.co/datasets/ai4bharat/sangraha
Pinned revision: 8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8
Included source files: verified/eng/data-0.parquet and verified/eng/data-1.parquet.

The source repository declares Creative Commons Attribution 4.0 International:
https://creativecommons.org/licenses/by/4.0/
The full license is preserved in LICENSE. Original content creators retain their rights.
This release makes no claim that sraivante wrote the third-party source texts.

Changes by sraivante / this release:
- Selected two existing English shards for the project; raw bytes are unchanged.
- Encoded the corpus with the project's custom 16,000-token byte-level BPE tokenizer,
  appended EOS after documents and split the token stream at 90% of total tokens.
- Packaged the saved arrays, source identity, integrity hashes and original release documentation.
No additional filtering, rewriting, redaction or deduplication was applied during publication.
The tokenizer was fitted before the split; one document straddles the split.

Please retain this attribution and cite the source authors' work:

```bibtex
@article{khan2024indicllmsuite,
  title = {IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages},
  author = {Mohammed Safi Ur Rahman Khan and Priyam Mehta and Ananth Sankar and Umashankar Kumaravelan and Sumanth Doddapaneni and Suriyaprasaad G and Varun Balan G and Sparsh Jain and Anoop Kunchukuttan and Pratyush Kumar and Raj Dabre and Mitesh M. Khapra},
  year = {2024},
  journal = {arXiv preprint arXiv: 2403.06350}
}
```

Copyright (c) 2026 sraivante, for original code, release documentation and original
packaging contributions only, where applicable, under Apache License 2.0.
See LICENSE-APACHE-2.0. This grant does not replace the corpus's CC BY 4.0 terms.