---
pretty_name: Custom GPT 40M Sangraha Subset
license: cc-by-4.0
language:
- en
task_categories:
- text-generation
tags:
- language-modeling
- sangraha
- tokenized
- provenance
size_categories:
- 100K<n<1M
configs:
- config_name: corpus
  default: true
  data_files:
  - split: corpus
    path: raw/*.parquet
dataset_info:
  config_name: corpus
  features:
  - name: doc_id
    dtype: string
  - name: text
    dtype: string
  - name: type
    dtype: string
  splits:
  - name: corpus
    num_examples: 699050
---

# Custom GPT 40M — Sangraha training snapshot

This repository preserves the raw English corpus and saved token streams associated with **[sraivante/Custom-GPT-40M-Base](https://huggingface.co/sraivante/Custom-GPT-40M-Base)**, a 39.85M-parameter model trained from scratch in the CUSTOM_LLM3 / CUSTOM_LLM_OLLAMA_IMPORT project.

The corpus is an unchanged two-shard subset of **AI4Bharat Sangraha**, not original writing by sraivante. Original article authors, publishers, speakers and other contributors retain their rights. The corpus and its tokenized representation retain **CC BY 4.0** terms and upstream attribution.

## Contents

| Artifact | Records / tokens | Storage |
|---|---:|---|
| `raw/data-0.parquet` | 349,525 documents | Original Parquet |
| `raw/data-1.parquet` | 349,525 documents | Original Parquet |
| `tokens/train_ids.npy` | 411,919,454 tokens | One-dimensional uint16 NumPy array |
| `tokens/val_ids.npy` | 45,768,829 tokens | One-dimensional uint16 NumPy array |
| `bpe_tokenizer.json` | 16,000 vocabulary entries | Original tokenizer |
| `provenance/data_audit.json` | Hashes, source identity, split details and verification | JSON |

The raw corpus contains **699,050 documents**: 689,417 tagged `web`, 9,578 `pdf` and 55 `speech`. Its exact original fields are:

| Field | Type | Meaning |
|---|---|---|
| `doc_id` | string | Original upstream document identifier |
| `text` | string | Extracted text used for tokenization |
| `type` | string | Upstream content category |

No null values were found in these three columns. The combined encoded corpus contains **457,688,283 tokens**, including 699,050 document-ending EOS markers.

## Source and attribution

- Dataset: [AI4Bharat Sangraha](https://huggingface.co/datasets/ai4bharat/sangraha)
- Pinned revision: `8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8`
- Source files: [verified/eng/data-0.parquet](https://huggingface.co/datasets/ai4bharat/sangraha/blob/8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8/verified/eng/data-0.parquet) and [verified/eng/data-1.parquet](https://huggingface.co/datasets/ai4bharat/sangraha/blob/8b813c3f62d37b2fa174d68c31e8b35ae2fe85e8/verified/eng/data-1.parquet)
- Upstream license: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/)
- Research paper: [IndicLLMSuite](https://arxiv.org/abs/2403.06350)
- Preserved upstream description and citation: [provenance/upstream_README.md](provenance/upstream_README.md)
- Credit and modification statement: [ATTRIBUTION.md](ATTRIBUTION.md)

Both raw files are byte-for-byte identical to that upstream revision, verified using the upstream LFS SHA-256 identifiers. Raw data was not rewritten, shuffled or filtered for this release.

The tokenized files are the project's saved arrays. They encode the original text in shard order with one `<eos>` token after each document. This tokenization and the train/validation split are transformations of the upstream corpus.

## Splitting and historical identity

The notebook concatenates `data-0.parquet` then `data-1.parquet`, encodes texts using its custom BPE tokenizer, appends EOS, and cuts at `floor(0.9 * total_tokens)`. The training array is the prefix and validation is the suffix.

**The split is not document-disjoint.** Document index 629,027, counting from zero, contributes 1,620 tokens to training and its remaining 294 tokens to validation. The tokenizer is fitted before this split in the notebook and can therefore learn vocabulary from validation text.

Raw documents are exposed as a single `corpus` split. This avoids assigning entire documents to token-level train/validation splits they do not exactly match. Use the saved arrays for the exact training/validation boundaries.

The packaged raw hashes match the upstream snapshot. All 261 deterministic sampled documents, including shard endpoints and the split-boundary document, reproduced their saved token spans exactly. This was a sample-based token reconstruction check, not a complete re-tokenization. The arrays have complete SHA-256 hashes and are copied unchanged from the local training artifacts. Their counts, matching samples, notebook and checkpoint metadata support the association with the run; the historical run has no external file-access log.

There is no separate test split or instruction fine-tuning dataset for this model. The reported validation loss uses random token windows from `val_ids.npy`.

## Load the corpus

```python
from datasets import load_dataset

corpus = load_dataset(
    "sraivante/Custom-GPT-40M-Sangraha-Subset",
    "corpus",
    split="corpus",
    streaming=True,
)
row = next(iter(corpus))
print(row.keys())  # doc_id, text, type
```

## Load the exact token streams

```python
import numpy as np
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo = "sraivante/Custom-GPT-40M-Sangraha-Subset"
train_path = hf_hub_download(repo, "tokens/train_ids.npy", repo_type="dataset")
val_path = hf_hub_download(repo, "tokens/val_ids.npy", repo_type="dataset")
tok_path = hf_hub_download(repo, "bpe_tokenizer.json", repo_type="dataset")

train = np.load(train_path, mmap_mode="r", allow_pickle=False)
val = np.load(val_path, mmap_mode="r", allow_pickle=False)
tokenizer = Tokenizer.from_file(tok_path)

assert train.shape == (411919454,)
assert val.shape == (45768829,)
# Next-token training window; promote uint16 IDs to int64 for PyTorch.
x = train[:256].astype("int64")
y = train[1:257].astype("int64")
```

Special tokens: `<unk>=0`, `<pad>=1`, `<bos>=2`, `<eos>=3`. The tokenizer uses byte-level BPE with `add_prefix_space=True`; it does not automatically insert BOS or EOS. The corpus tokenizer was trained on text with line breaks normalized for tokenizer training, while the saved token arrays encode original text. Not every possible character is necessarily covered by its learned vocabulary.

The raw files and arrays require approximately 2.11 GB together, before any Arrow cache or model files. Pin a repository commit in `revision=...` for later reproducibility; file hashes are also included in `SHA256SUMS`.

## Appropriate uses and limitations

This snapshot supports reproducibility, English causal language modeling and small-model experiments. It represents only two English shards of Sangraha. It is not a representative sample of all English or all of Sangraha, and it contains no instruction/response annotations.

Web articles, PDF extractions and transcriptions may contain mistakes, duplicated text, extraction noise, biases, offensive material, and names or other personal information that was present in the public sources. No additional de-identification, redaction, deduplication or content moderation was performed for this publication. The data should not be treated as verified ground truth or an independent downstream benchmark.

## Copyright and licensing

The **corpus and token arrays are CC BY 4.0**; preserve AI4Bharat / Sangraha attribution, acknowledge original content creators, link the license and state your modifications. See [LICENSE](LICENSE) and [ATTRIBUTION.md](ATTRIBUTION.md).

**Copyright (c) 2026 sraivante** applies only to original project code, original release documentation and original packaging contributions where copyright exists. Those contributions are licensed under [Apache License 2.0](LICENSE-APACHE-2.0). This does not transfer ownership of, or relicense, the third-party text. The model weights have their own Apache-2.0 release.

Snapshot version: **2026-09-25**. Model, dataset, tokenizer, source revision and hashes are linked for this authorized release.