---
license: other
language: [sn, sw, ha, yo, ig, zu, xh, rw, tn, af, nr, pcm, en]
tags: [african-languages, from-scratch, instruct, research-preview, isheanesu-misi, vambo-ai]
inference: false
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/fineweb-2
  - codeparrot/codeparrot-clean
  - codeparrot/github-code-clean
  - open-web-math/open-web-math
  - EleutherAI/proof-pile-2
  - castorini/wura
  - allenai/MADLAD-400
  - castorini/afriberta-corpus
  - vamboai/fikira
  - ise-uiuc/Magicoder-OSS-Instruct-75K
  - m-a-p/CodeFeedback-Filtered-Instruction
  - thisisisheanesu/morena-sft-corpus
base_model: thisisisheanesu/morena-1.5b-base
---

# MORENA 1.5B instruct

MORENA, Sesotho and Setswana for a king, a lord or a chief, is a 1.5B-parameter decoder trained
from scratch for twelve Latin-script African languages plus English, French and code. This is the
instruction-tuned model described in the whitepaper *MORENA: Built, not adapted*. **Private research
preview. Not for user-facing deployment.**

## Headline numbers

| | MORENA 1.5B instruct | reference |
|---|---|---|
| African bits per byte, mean of 12 (lower is better) | **1.441** | Lugha-Llama-8B 1.423, gemma-3-12b-it 2.159, Llama-3.2-1B 2.498 |
| Translation, FLORES+ chrF++, English into 5 African languages, 3-shot | **45.8** | NLLB-600M 45.6, NLLB-1.3B 47.2, Lugha-Llama-8B 36.8, 1B-class general models 9 to 14 |
| Translation, 5 African languages into English | **48.7** | NLLB-600M 55.1 |
| Tool calling, correct tool / valid JSON (constrained, prefilled) | **98.1% / 100%** | |
| Retrieval QA, open-book accuracy / grounding, African mean | 0.325 / 0.734 | 0.25 chance |
| Safety, good of scorable, 3,341 prompts, 13 languages, 11 categories | **90.2%** | |
| Degenerate output | 1.3% | |
| Benign requests answered well | 58.9% | |

Safety by category (share of all attempts handled well, 24 prompts per category per language):
self-harm 92.5, child safety 91.2, drugs 86.2, violence 88.9, election 89.2, privacy 93.8, fraud 94.3,
weapons 93.3, hate 95.4, medication 95.4. Weakest languages: Igbo 75%, Yoruba 82%, Setswana 83%,
isiXhosa 87%. Every number is a model judging a model; no native speaker has yet rated an answer.

## What it is not good at

Retrieval-augmented QA is at chance in African languages even though the model demonstrably reads
the passage (grounding 0.73). Grounded generation is fully faithful to given facts in about 31% of
attempts. The model over-refuses: 41% of ordinary benign requests still get a refusal. Tool calling
is measured with the tool marker prefilled; the model does not reliably decide on its own that a tool
is needed. Reading comprehension (belebele) 0.302 against 0.250 chance.

## Chat format

Single reserved tokens mark turns: `<reserved_0>` opens a user turn and `<reserved_1>` an assistant
turn (token ids 3 and 4). `load_example.py` in this repo shows a full prompt. Do not use
`<|user|>`-style strings; they are not in the vocabulary and produce degenerate output.

## Files

`model.safetensors` (bf16), `config.json`, `tokenizer.json`, `modeling_morena.py` (reference
implementation, plain PyTorch, no transformers dependency), `load_example.py`, `SHA256SUMS`.
A GGUF build for llama.cpp is in `thisisisheanesu/morena-1.5b-instruct-gguf`.

## Training

251.7B tokens of pretraining (45% English, 23% code, 10% native African text, 22% machine-translated
African text, 7.5% French), 63B tokens of mid-training, 4,500 steps of supervised fine-tuning with
loss masking and a 500-step safety anneal. Release lineage 12,834 A100 GPU-hours; the research
programme that produced it about 22,450. Architecture: 28 layers x 2048, GQA 16/4, SwiGLU 6144,
RoPE theta 500,000, 4,096 context, tied embeddings. Optimiser: Muon for non-embedding weights,
AdamW for the rest, WSD schedule.

## The MORENA family

| model | params | African bpb (all 12, lower is better) | role |
|---|---|---|---|
| MORENA 1.5B base | 1.485B | 1.408 | pretrained and mid-trained; fine-tuning starting point |
| MORENA 1.5B instruct | 1.485B | 1.441 | chat, translation, tool calling; the model described in the whitepaper |
| MORENA 0.5B mini | 503M | 1.520 | pruned and distilled from the 1.5B base |
| MORENA 0.5B mini instruct | 503M | 1.540 | chat fine-tune of the mini |
| MORENA 0.2B nano | 209M | 1.583 | cheap trunk for ASR rescoring, keyboards, normalisation |

Every outside model we measured (25 in total, from 125M to 12B, including the 8B African specialist
Lugha-Llama-8B at 1.423 and gemma-3-12b-it at 2.159) sits behind all five on African bits per byte.
Twelve languages: Shona, Swahili, Hausa, Yoruba, Igbo, isiZulu, isiXhosa, Kinyarwanda, Setswana,
Afrikaans, isiNdebele, Naija Pidgin; plus English, French and code. Tokenizer: 65,536 entries trained
on the target mix, fertility 0.239 tokens per byte on African text against 0.246 on English.

## Author and citation

Isheanesu Misi, Vambo AI. Trained on CINECA Leonardo (EuroHPC allocation aih4a_vamboai) with support
from the AI Hub for Sustainable Development.

```
@techreport{misi2026morena,
  title  = {MORENA: Built, not adapted. A 1.5B-parameter language model trained from scratch for twelve African languages},
  author = {Misi, Isheanesu},
  institution = {Vambo AI},
  year   = {2026},
  month  = {December}
}
```

## Licence and status

Private research preview. Weights are released under a custom licence pending resolution of one
data-licensing question: 22% of the pretraining corpus derives from an NLLB model under a
NonCommercial licence. Until that is settled the weights are not for commercial use or public
redistribution. **Not for user-facing deployment**: see the evaluation notes above.