README.md

---
license: apache-2.0
language:
- ar
- en
base_model: oddadmix/Emhotob-25M
pipeline_tag: text-generation
library_name: transformers
tags:
- translation
- msa
- fusha
- english
- arabic
- small-language-model
- slm
- tiny-lm
- chatml
- scaling-study
metrics:
- bleu
- chrf
---

# Emhotob-25M-English-MSA-v1 — Bidirectional English ↔ MSA (~25.3M params)

A **25.3M-parameter** model that translates **both ways** between **English** and **Modern Standard Arabic (الفصحى)**.
A single set of weights serves both directions; a direction-specific system prompt selects
which way to translate.

Finetuned from [`oddadmix/Emhotob-25M`](https://huggingface.co/oddadmix/Emhotob-25M), a tiny Llama-architecture
base (hidden 384, 8 layers, 6 heads, vocab 32000, tied embeddings).

> **Scaling study.** This is one rung of a from-scratch Arabic scaling study that runs an
> identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where
> translation *emerges*. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M,
> becomes real-but-rough at 5M, and usable at 10M+. See the sibling
> [`oddadmix/50M-English-MSA-v1`](https://huggingface.co/oddadmix/50M-English-MSA-v1) for the fluent reference.

## Evaluation

Deterministic held-out set of **3,000 pairs** (`seed=42`),
decoded **greedily** (`do_sample=False`, no repetition penalty), scored with sacreBLEU:

| Direction | sacreBLEU | chrF |
|---|---:|---:|
| **English → MSA** | **40.65** | 62.42 |
| **MSA → English** | **44.70** | 60.68 |

Saved weights are the best checkpoint by validation loss (`eval_loss = 0.703`). 20 samples
per direction with references are in [`eval_bidirectional.json`](./eval_bidirectional.json).

### Example translations

Real greedy-decoded outputs from the held-out set:

**English → MSA**

| Source | Model output | Reference |
|---|---|---|
| I'm just trying to reassure myself. What's the worst that could happen? It's just mentally exhausting, but they're stand | أنا فقط أحاول طمأنة نفسي. ما الأسوأ أن يحدث ذلك؟ إنه مرهق عقليًا، لكنهم يقفون بقوة. حقًا، إنهم يقفون بقوة. | أنا فقط أحاول أن أطمئن نفسي. ما هو الأسوأ الذي يمكن أن يحدث؟ إنه مرهق عقليًا فقط، لكنهم يقفون بحزم. حقًا، إنهم يقفون بحز |
| Thank you very much, you are so kind. | شكرًا جزيلًا لك، أنت لطيف جدًا. | شكرًا جزيلًا لك، أنت لطيف للغاية. |

**MSA → English**

| Source | Model output | Reference |
|---|---|---|
| أنا فقط أحاول أن أطمئن نفسي. ما هو الأسوأ الذي يمكن أن يحدث؟ إنه مرهق عقليًا فقط، لكنهم يقفون بحزم. حقًا، إنهم يقفون بحز | I'm just trying to reassure myself. What's the worst can happen? It's just mentally exhausted, but they're standing down | I'm just trying to reassure myself. What's the worst that could happen? It's just mentally exhausting, but they're stand |
| شكرًا جزيلًا لك، أنت لطيف للغاية. | Thank you so much, you're so sweet. | Thank you very much, you are so kind. |

## Usage

ChatML format. **Pick the system prompt for the direction you want:**

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-25M-English-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYSTEM = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللغة العربية الفصحى."

def translate(text, system=SYSTEM):
    prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
              f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()
```

## Training

- **Base model:** `oddadmix/Emhotob-25M` (Llama arch, hidden 384, 8 layers, 6 heads, vocab 32000, tied embeddings;
  **25,271,424 params** after resizing for 2 ChatML tokens)
- **Dataset:** `oddadmix/egyptian-msa-2.9-openai-bytedance-translations`
- **Method:** HuggingFace `Trainer`, ChatML, **prompt-masked cross-entropy** (loss only on the
  assistant turn). Each row is exploded into **two** training examples (one per direction).
- **Hyperparameters:** 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
  bf16 · max length 1024 · `load_best_model_at_end` on `eval_loss`.
- **Eval split:** 3,000 deterministic held-out pairs (`seed=42`), scored both directions.

## Limitations

A ~25.3M model: reliable on short/common sentences, but drift, repetition, and errors appear
on long or rare inputs. Gender is disambiguated only from context. For fluent translation use the
50M sibling.

## License

Apache-2.0, inherited from the base model.