File size: 2,339 Bytes
2e48b3f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
language:
- ar
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- arabic
- llama
- pretraining
- from-scratch
- emhotob
- small-language-model
datasets:
- kaust-generative-ai/fineweb-edu-ar
---
# Emhotob-25M
**Emhotob** is a family of small Arabic language models pretrained **from scratch** on
Arabic web text. This is the **25M** rung of the ladder (~25.27M parameters),
part of a scaling series ranging from 500K to 25M parameters that all share the same
tokenizer, context length, and training recipe.
> ⚠️ These are tiny, research-scale models trained on a limited token budget. They are
> intended for scaling-law experiments, education, and Arabic NLP research — **not** for
> production use.
## Model details
| Property | Value |
|---|---|
| Architecture | Llama (decoder-only, RoPE, GQA) |
| Parameters | 25,270,656 (~25.27M) |
| Hidden size | 384 |
| Layers | 8 |
| Attention heads | 6 (KV heads: 3) |
| Intermediate size | 1024 |
| Context length | 2048 |
| Vocabulary | 32,000 (custom Byte-Level BPE) |
| Tied embeddings | Yes |
| RoPE theta | 10,000 |
| Precision | bf16 |
## Training
| Property | Value |
|---|---|
| Data | [`kaust-generative-ai/fineweb-edu-ar`](https://huggingface.co/datasets/kaust-generative-ai/fineweb-edu-ar) (Arabic) |
| Tokens seen | ~2.5B (1 epoch) |
| Optimizer | AdamW (fused), β=(0.9, 0.95), wd=0.1 |
| LR schedule | 6e-4, cosine, 2% warmup |
| Effective batch | 128 sequences × 2048 tokens |
| Grad clipping | 1.0 |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "oddadmix/Emhotob-25M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
prompt = "الذكاء الاصطناعي هو"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
```
## Limitations
Given its size and limited pretraining budget, Emhotob-25M has a narrow capability
range and will produce factually unreliable and sometimes incoherent text. It has not been
instruction-tuned or aligned, and no safety filtering has been applied. Use accordingly.
---
*© SupraLabs 2026 — PROJECT EMHOTOB.*
|