---
license: apache-2.0
language: [sn, sw, ha, yo, ig, zu, xh, rw, tn, af, nr, pcm, en]
tags: [african-languages, from-scratch, instruct, isheanesu-misi, vambo-ai]
inference: false
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/fineweb-2
  - codeparrot/codeparrot-clean
  - codeparrot/github-code-clean
  - open-web-math/open-web-math
  - EleutherAI/proof-pile-2
  - castorini/wura
  - allenai/MADLAD-400
  - castorini/afriberta-corpus
  - vamboai/fikira
  - ise-uiuc/Magicoder-OSS-Instruct-75K
  - m-a-p/CodeFeedback-Filtered-Instruction
  - vamboai/morena-sft-corpus
base_model: vamboai/morena-1.5b-base
---

<div align="center">
<img src="poster.png" alt="MORENA, an African foundation model" width="600">
</div>

# MORENA 1.5B instruct

MORENA, Sesotho and Setswana for a king, a lord or a chief, is a 1.5B-parameter decoder trained
from scratch for twelve Latin-script African languages plus English, French and code. This is the
instruction-tuned model described in the paper *MORENA: An African Foundation Model*.

## Headline numbers

| | MORENA 1.5B instruct | reference |
|---|---|---|
| African bits per byte, mean of 12 (lower is better) | **1.441** | Lugha-Llama-8B 1.423, gemma-3-12b-it 2.159, gemma-3-1b-pt 2.335 |
| Translation, FLORES+ chrF++, English into 5 African languages, 3-shot | **45.8** (45.1 to 46.5) | MADLAD-400-3B 37.8, Lugha-Llama-8B 36.8 |
| Translation, 5 African languages into English | **48.7** (47.5 to 49.7) | MADLAD-400-3B 53.6, Lugha-Llama-8B 52.4 |
| Belebele reading comprehension, released checkpoint | 0.309 | mean over 10 African languages, 0.24 Yoruba to 0.43 Afrikaans, 0.25 chance |
| Retrieval QA, open-book accuracy / grounding, African mean | 0.325 (0.297 to 0.355) / 0.73 | 0.25 chance |
| Grounded generation, fully faithful to given facts | 23% (22 of 96, 15.6 to 32.3) | previous version 31% (30 of 96); un-instructed base 6% (6 of 96) |
| Tool calling, correct tool / valid JSON (marker prefilled) | **98.1% / 100%** | |
| Safety, share of all 3,341 attempts handled well, 13 languages, 11 categories | **89.0%** (87.8 to 90.0) | 90.2% of scorable attempts |
| Degenerate output | 1.3% | |
| Refused harmful requests, share of all attempts | 91.5% | |
| Benign requests answered well | 58.9% (53.3 to 64.2) | previous version 47.9% |

MORENA 1.5B instruct's English-into-African chrF++ of 45.8 is eight points above MADLAD-400-3B (37.8).

Safety by category (share of all attempts handled well, 24 prompts per category per language):
self-harm 92.5, child safety 91.2, drugs 86.2, violence 88.9, election 89.2, privacy 93.8, fraud 94.3,
weapons 93.3, hate 95.4, medication 95.4. Weakest languages: Igbo 75%, Yoruba 82%, Setswana 83%,
isiXhosa 87%. Every number above is a model (google/gemma-3-12b-it) judging a model; no native
speaker has yet rated an answer.

## What it is not good at

Retrieval-augmented QA is at chance in African languages even though the model demonstrably reads
the passage (grounding 0.73). Grounded generation is fully faithful to given facts in 23% of
attempts, down from 31% in the previous version. The model fails to answer 41% of ordinary benign
requests well, most of them by refusing; an earlier version once refused to recommend a dry
cleaner, citing "illegal substances or services". That specific failure is fixed, but the broader
over-refusal problem is not. Tool calling is measured with the tool marker prefilled; left to decide
for itself the model almost never calls one. Multiple-choice comprehension in African languages is
at chance for this model and for every model under 12B measured.

The checkpoint was chosen against a rule fixed before the final experiments: every harm category
within 3.5 points of the previous version, refusals on at least 91% of all attempts, degeneracy at
most 1%, no capability lost. It misses that rule on privacy by 0.1 points, on drugs by 1.0 and on violence by 2.6, and on
degeneracy, which is 1.3%.

## Chat format

Single reserved tokens mark turns: `<reserved_0>` opens a user turn and `<reserved_1>` an assistant
turn (token ids 3 and 4). `load_example.py` in this repo shows a full prompt. Do not use
`<|user|>`-style strings; they are not in the vocabulary and produce degenerate output.

## Files

`model.safetensors` (bf16), `config.json`, `tokenizer.json`, `modeling_morena.py` (reference
implementation, plain PyTorch, no transformers dependency), `load_example.py`, `SHA256SUMS`.
A GGUF build for llama.cpp is in `vamboai/morena-1.5b-instruct-gguf`.

## Training

251.7B tokens of pretraining, 63B tokens of mid-training (315B tokens seen in total; see MORENA
1.5B base), then 4,500 steps of supervised fine-tuning with loss masking and a 500-step safety
anneal. Training mixture moved in three regimes as machine-translated languages landed: African
text was 24.8% of tokens seen (14.8% machine-translated) for steps 1 to 25,304, 39.1% (31.0%
machine-translated) for steps 25,305 to 60,000, and 50.2% (41.6% machine-translated) during
mid-training. Nine languages were machine-translated from English documents: this is 24% of
pretraining tokens seen and 28% including mid-training (57.1B tokens on disk, 21% of the 271B
on disk).

Release lineage: 12,834 A100 GPU-hours, an estimated twenty-five to forty thousand dollars at
$2 to $3 per A100-hour; the research programme that produced it, about 22,450 GPU-hours (about
22% of a 100,000 GPU-hour allocation). Architecture: 28 layers x 2048, GQA 16/4, SwiGLU 6144, RoPE theta
500,000, 4,096 context, tied embeddings. Optimiser: Muon for non-embedding weights, AdamW for
the rest, warmup-stable-decay schedule.

## The MORENA family

| model | params | African bpb (all 12, lower is better) | role |
|---|---|---|---|
| MORENA 1.5B base | 1.485B | 1.408 | pretrained and mid-trained; fine-tuning starting point |
| MORENA 1.5B instruct | 1.485B | 1.441 | chat, translation, tool calling; the model described in the paper |
| MORENA 0.5B mini | 503M | 1.520 | pruned and distilled from the 1.5B base |
| MORENA 0.5B mini instruct | 503M | 1.540 | chat fine-tune of the mini |
| MORENA 0.2B nano | 209M | 1.583 | cheap trunk for ASR rescoring, keyboards, normalisation |

26 models in total were measured on African bits per byte, 21 of them outside models from 125M
to 12B parameters. MORENA 1.5B base is lowest of all 26 at 1.408; the 8B African specialist
Lugha-Llama-8B is second at 1.423, ahead of the other four MORENA sizes on the twelve-language
mean. Every general-purpose model measured, from 1B to 12B parameters, trails well behind: the
best of them, gemma-3-12b-it, reaches only 2.159. Twelve languages: Shona, Swahili,
Hausa, Yoruba, Igbo, isiZulu, isiXhosa, Kinyarwanda, Setswana, Afrikaans, isiNdebele and
Nigerian Pidgin, plus English, French and code. isiNdebele (ISO code nbl) has no FLORES+ or
MasakhaNEWS coverage and is evaluated on NCHLT transcripts only.

Tokenizer: 65,536-entry byte-fallback BPE trained on the target mix. African text costs 0.249
tokens per byte against 0.234 for English, about 6% more per byte than English in MORENA's
vocabulary, but that same African text needs 1.39x fewer tokens than under Gemma 3's
vocabulary and 1.53x fewer than under Llama 3.2's.

Translation is measured for seventeen models in total, thirteen of them outside models.

## Author and citation

Isheanesu Misi, Vambo AI. Trained on CINECA Leonardo, with support from the AI Hub for
Sustainable Development.

```
@techreport{misi2026morena,
  title  = {MORENA: An African Foundation Model},
  author = {Misi, Isheanesu},
  institution = {Vambo AI},
  year   = {2026},
  month  = {September},
  note   = {Released 18 September 2026}
}
```

## Licence

Apache 2.0. Commercial use is permitted, including fine-tuning and redistributing derivatives,
under the terms of the licence.

The training corpus is not released with these weights.