Add mid-training data recipe to README
Browse files
README.md
CHANGED
|
@@ -75,3 +75,60 @@ tokenizer = AutoTokenizer.from_pretrained(
|
|
| 75 |
year={2024}
|
| 76 |
}
|
| 77 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
year={2024}
|
| 76 |
}
|
| 77 |
```
|
| 78 |
+
|
| 79 |
+
## Mid-Training Data Recipe (201305 cutoff)
|
| 80 |
+
|
| 81 |
+
Three pre-cutoff text sources are concatenated (no upsampling), packed to a 2,048-token
|
| 82 |
+
context, and trained for one epoch.
|
| 83 |
+
|
| 84 |
+
| Source | Time filter | Documents | Est. tokens |
|
| 85 |
+
|---|---|---|---|
|
| 86 |
+
| peS2o (academic abstracts/full text) | published before May 2013 | 1,859,534 | ~1.0 B |
|
| 87 |
+
| Wikipedia (English) | first-revision date before May 2013 | 3,966,112 | ~3.5 B |
|
| 88 |
+
| DCLM (Common Crawl, filtered) | none (assumed pre-cutoff web text) | 3,218,997 | ~4.5 B |
|
| 89 |
+
| **Total** | | **~9.0 M docs** | **~9.0 B** |
|
| 90 |
+
|
| 91 |
+
Token estimates use a chars-per-token ratio of ~4 (verified ratios are ~0.21–0.23 tokens/char
|
| 92 |
+
with the OpenLM tokenizer; the table reports the 4-char approximation). See the project repo
|
| 93 |
+
for the per-cutoff data prep code (`prepare_midtrain_data.py`) and the slice statistics
|
| 94 |
+
(`stats.json`).
|
| 95 |
+
|
| 96 |
+
### LLaMA-Factory dataset wiring
|
| 97 |
+
|
| 98 |
+
```yaml
|
| 99 |
+
dataset: midtrain_pes2o_pre201305,midtrain_wiki_pre201305,midtrain_dclm
|
| 100 |
+
template: empty
|
| 101 |
+
cutoff_len: 2048
|
| 102 |
+
mix_strategy: concat
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
Per-source files (relative to the dataset root):
|
| 106 |
+
|
| 107 |
+
- `midtrain/pes2o_slices/pes2o_pre201305_1b.jsonl`
|
| 108 |
+
- `midtrain/wiki_slices/wiki_pre201305.jsonl`
|
| 109 |
+
- `midtrain/dclm_4_5b.jsonl`
|
| 110 |
+
|
| 111 |
+
All three are jsonl with a single `text` column.
|
| 112 |
+
|
| 113 |
+
### Training hyperparameters
|
| 114 |
+
|
| 115 |
+
| Hyperparameter | Value |
|
| 116 |
+
|---|---|
|
| 117 |
+
| Framework | LLaMA-Factory `stage: pt`, `finetuning_type: full` |
|
| 118 |
+
| Optimizer | DeepSpeed ZeRO-2 |
|
| 119 |
+
| Precision | bf16 |
|
| 120 |
+
| GPUs | 4 × H200 |
|
| 121 |
+
| Per-device batch | 64 |
|
| 122 |
+
| Gradient accumulation | 1 |
|
| 123 |
+
| Effective batch (tokens) | 4 × 64 × 2048 ≈ 524,288 / step |
|
| 124 |
+
| Learning rate | 5.0e-5, cosine schedule, 3% warmup |
|
| 125 |
+
| Epochs | 1.0 |
|
| 126 |
+
| Total optimizer steps | 16,034 |
|
| 127 |
+
| Tokens consumed | ~8.4 B (≈ 1 pass over the corpus) |
|
| 128 |
+
|
| 129 |
+
### Why mid-train?
|
| 130 |
+
|
| 131 |
+
The mid-training stage **re-exposes the model to pre-cutoff facts** drawn from peS2o,
|
| 132 |
+
Wikipedia, and DCLM to *consolidate* (rather than extend) the model's knowledge. No
|
| 133 |
+
post-cutoff text is included, so the knowledge cutoff date is preserved while the
|
| 134 |
+
representation of pre-cutoff knowledge is strengthened.
|