---
license: apache-2.0
base_model: openeurollm/datamix-9b-80-20
language:
- en
- cs
- de
- es
- fi
- fr
- it
- sv
library_name: transformers
tags:
- llama
- sft
- multilingual
- european-languages
- dolci-translated
- continued-sft
- cross-architecture
datasets:
- allenai/Dolci-Instruct-SFT
- openeurollm/Dolci-Instruct-SFT-translated
---

# datamix-9b Dolci-Translated A-75EN

Cross-architecture validation of the *Translate, Replay, Mix* recipe applied to
`openeurollm/datamix-9b-80-20`, a Llama-architecture, multilingually-pretrained
$9$B base with a Gemma tokenizer. Same $75/25$ English replay + Dolci-Translated
EU mixture as `openeurollm/OLMo-3-7B-Dolci-Translated-A-75EN`, but trained with
a different framework and at a different context length because of the base's
architecture and pretraining limits.

## Recipe

| | |
|---|---|
| **Base checkpoint** | `openeurollm/datamix-9b-80-20` ($9$B Llama, Gemma tokenizer, vocab 262k, $2048$ context) |
| **English half (Dolci replay)** | `allenai/Dolci-Instruct-SFT`, 75% of the mixture |
| **EU half (Dolci-Translated)** | `openeurollm/Dolci-Instruct-SFT-translated`, 25%, 7 EU languages translated with `gemma-3-27b-it` |
| **EU languages** | cs, de, es, fi, fr, it, sv |
| **Total samples** | 2.87M (same mixture as OLMo A-75EN) |
| **Final step** | 41000 |
| **Chat template** | `simple_chat` (no built-in template on the base; use `--chat_template_name simple_chat` or apply manually) |

## Training configuration

Departs from the OLMo runs in five places (see Section 3.3 of the paper):

- Llama-architecture $9$B base (vs OLMo)
- Maximum context length $2048$ (vs $32$k)
- Micro-batches accumulated to effective batch of $128$ (vs ${\sim}1$M tokens)
- Plain DDP (vs DeepSpeed ZeRO 2)
- `open-instruct/finetune.py` entry point (vs OLMo-core)

Peak LR $8\times10^{-5}$ matches the OLMo runs.

## Evaluation

Per-language Bradley-Terry Elo at matched $1024/1024$ input/output truncation,
500 battles/language, against `openeurollm/OLMo-3-7B-Dolci-Translated-A-75EN`
re-evaluated under the same truncation (paper Table 4, Figure 6):

| Model                                             | en        | cs        | de        | es        | fi        | fr        | it        | sv        |
|---------------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| **This repo** (`datamix-9b-Dolci-Translated-A-75EN`) | $879 \pm 16$         | $\mathbf{833 \pm 15}$ | $\mathbf{792 \pm 19}$ | $764 \pm 20$         | $\mathbf{815 \pm 35}$ | $790 \pm 18$         | $780 \pm 18$         | $\mathbf{820 \pm 32}$ |
| OLMo-3-7B A-75EN                                  | $\mathbf{970 \pm 14}$ | $733 \pm 18$         | $742 \pm 20$         | $\mathbf{820 \pm 17}$ | $752 \pm 38$         | $\mathbf{804 \pm 16}$ | $\mathbf{821 \pm 15}$ | $786 \pm 32$ |

This checkpoint leads on cs ($+100$), de ($+50$), fi ($+63$), sv ($+34$);
OLMo-3-7B A-75EN leads on en ($+91$), es ($+56$), it ($+41$), fr ($+14$). The
crossover is consistent with a multilingually-pretrained base partially
substituting for the multilingual SFT pool on lower-resource EU languages.

## How to load

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("openeurollm/datamix-9b-Dolci-Translated-A-75EN")
model = AutoModelForCausalLM.from_pretrained("openeurollm/datamix-9b-Dolci-Translated-A-75EN", torch_dtype="bfloat16")
# No built-in chat template; use 'simple_chat' from open-instruct or apply your own.
```

## Citation

Please cite the paper if you use this checkpoint.