Emhotob-10M-MSA-Egyptian-v1 — Bidirectional MSA ↔ Egyptian Arabic (~11M params)

An 10.9M-parameter model that translates both ways between Modern Standard Arabic (الفصحى) and Egyptian colloquial Arabic (المصرية العامية). A single set of weights serves both directions; a direction-specific Arabic system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-10M, a tiny Llama-architecture base (hidden size 256, 4 layers, 8 heads, tied embeddings).

Scaling study. This runs the exact recipe of oddadmix/50M-MSA-Egyptian-v1 on a base ~5× smaller. At 10M the model produces genuinely usable short-sentence translations — roughly half the 50M BLEU and a large step up from the 5M rung. Still expect drift on long or rare inputs.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
MSA → Egyptian	15.64	43.34
Egyptian → MSA	15.47	40.87

The saved weights are the best checkpoint by validation loss (eval_loss = 2.190, epoch 3 of 3).

Scaling curve (same data, recipe, and eval; only the base model changes)

Base	Params	eval_loss	BLEU (both dir)	Behavior
Emhotob-500K	0.52M	8.42	~0.01	degenerate (repeats punctuation)
Emhotob-1M	1.07M	7.39	~0.00	degenerate (repeats function words)
Emhotob-5M	5.08M	3.55	~4.3	real, rough translation
Emhotob-10M (this)	10.9M	2.19	~15.5	usable short-sentence translation
50M-2048-Emhotob	51.8M	~1.25	~24–26	fluent

Example translations

Real greedy-decoded outputs from the held-out set:

MSA → Egyptian

MSA input	Model output (Egyptian)	Reference
شكرًا جزيلًا لك، أنت لطيف للغاية.	شكراً جداً، إنتي حلوة قوي.	شكراً جداً، إنت طيب قوي.
ليحفظ الله الجميع الذين لديهم أصدقاء مخلصون.	ربنا يبارك في كل الناس اللي عندهم صح. لازم يقدروا…	ربنا يبارك لكل واحد عنده صاحب وافي…

Egyptian → MSA

Egyptian input	Model output (MSA)	Reference
شكراً جداً، إنت طيب قوي.	شكرًا جزيلًا لك، أنت لطيف جدًا.	شكرًا جزيلًا لك، أنت لطيف للغاية.
انا بس بحاول اطمن نفسي. ايه أسوأ حاجة ممكن تحصل؟	أنا فقط أحاول أن أتناول نفسي. ما هو أسوأ شيء قد يحدث	أنا فقط أحاول أن أطمئن نفسي…

Short, common sentences are handled well and register-switching is reliable; longer inputs still drift. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-10M-MSA-Egyptian-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_EGY = "أنت مترجم محترف. ترجم النص من اللغة العربية الفصحى إلى اللهجة المصرية العامية."
SYS_TO_MSA = "أنت مترجم محترف. ترجم النص من اللهجة المصرية العامية إلى اللغة العربية الفصحى."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("شكرًا جزيلًا لك، أنت لطيف للغاية.", SYS_TO_EGY))

Training

• Base model: oddadmix/Emhotob-10M (Llama arch, hidden 256, 4 layers, 8 heads, vocab 32000, tied embeddings; 10,947,328 params after resizing for 2 ChatML tokens)
• Dataset: oddadmix/egyptian-msa-2.9-openai-bytedance-translations (132K rows, egyptian/msa columns)
• Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
• Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
• Split: 129,009 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

• An 11M model: reliable on short/common sentences, but drift, repetition, and errors appear on long or rare inputs.
• Gender is disambiguated only from context; ambiguous inputs may default one way.
• For fluent translation use oddadmix/50M-MSA-Egyptian-v1.

License

Apache-2.0, inherited from the base model.