README.md

---
license: apache-2.0
language:
- ar
base_model: oddadmix/Emhotob-25M
pipeline_tag: text-generation
library_name: transformers
tags:
- translation
- darija
- moroccan-arabic
- msa
- fusha
- arabic
- small-language-model
- slm
- tiny-lm
- chatml
- scaling-study
metrics:
- bleu
- chrf
---

# Emhotob-25M-Darija-MSA-v1 — Bidirectional Moroccan Darija ↔ MSA (~25.3M params)

A **25.3M-parameter** model that translates **both ways** between **Moroccan Darija (الدارجة المغربية)** and **Modern Standard Arabic (الفصحى)**.
A single set of weights serves both directions; a direction-specific system prompt selects
which way to translate.

Finetuned from [`oddadmix/Emhotob-25M`](https://huggingface.co/oddadmix/Emhotob-25M), a tiny Llama-architecture
base (hidden 384, 8 layers, 6 heads, vocab 32000, tied embeddings).

> **Scaling study.** This is one rung of a from-scratch Arabic scaling study that runs an
> identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where
> translation *emerges*. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M,
> becomes real-but-rough at 5M, and usable at 10M+. See the sibling
> [`oddadmix/50M-Darija-MSA-v1`](https://huggingface.co/oddadmix/50M-Darija-MSA-v1) for the fluent reference.

## Evaluation

Deterministic held-out set of **2,961 pairs** (`seed=42`),
decoded **greedily** (`do_sample=False`, no repetition penalty), scored with sacreBLEU:

| Direction | sacreBLEU | chrF |
|---|---:|---:|
| **Darija → MSA** | **24.31** | 43.95 |
| **MSA → Darija** | **32.07** | 43.36 |

Saved weights are the best checkpoint by validation loss (`eval_loss = 1.615`). 20 samples
per direction with references are in [`eval_bidirectional.json`](./eval_bidirectional.json).

### Example translations

Real greedy-decoded outputs from the held-out set:

**Darija → MSA**

| Source | Model output | Reference |
|---|---|---|
| لا، عندنا تذاكر يا حبيبة | لا، لدينا تذاكر يا حبيبة. | لا، لدينا تذاكر يا حبيبتي. |
| غطّا وجهو و بكا | أجده وشاحه. | لقد غطى وجهه وبكى. |

**MSA → Darija**

| Source | Model output | Reference |
|---|---|---|
| لا، لدينا تذاكر يا حبيبتي. | لا، عندنا تذاكر يا حبيبتي | لا، عندنا تذاكر يا حبيبة |
| لقد غطى وجهه وبكى. | راه جبتو وجهو و كاتكبر | غطّا وجهو و بكا |

## Usage

ChatML format. **Pick the system prompt for the direction you want:**

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-25M-Darija-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYSTEM = "أنت مترجم محترف. ترجم النص من الدارجة المغربية إلى اللغة العربية الفصحى."

def translate(text, system=SYSTEM):
    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()
```

## Training

- **Base model:** `oddadmix/Emhotob-25M` (Llama arch, hidden 384, 8 layers, 6 heads, vocab 32000, tied embeddings;
  **25,271,424 params** after resizing for 2 ChatML tokens)
- **Dataset:** `oddadmix/darija_english_msa_parallel_dataset`
- **Method:** HuggingFace `Trainer`, ChatML, **prompt-masked cross-entropy** (loss only on the
  assistant turn). Each row is exploded into **two** training examples (one per direction).
- **Hyperparameters:** 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
  bf16 · max length 1024 · `load_best_model_at_end` on `eval_loss`.
- **Eval split:** 2,961 deterministic held-out pairs (`seed=42`), scored both directions.

## Limitations

A ~25.3M model: reliable on short/common sentences, but drift, repetition, and errors appear
on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the
50M sibling.

## License

Apache-2.0, inherited from the base model.