Naqta - Arabic Punctuation Restoration

Restores punctuation in Modern Standard Arabic. Two components in one repo:

component	path	task
point-mark model	repo root	8 classes (XLM-RoBERTa-large)
span tagger	span-tagger/	paired interjection dashes (BIO, XLM-RoBERTa-base)

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("MostafaMaroof/Naqta")
model = AutoModelForTokenClassification.from_pretrained("MostafaMaroof/Naqta")

span_tok = AutoTokenizer.from_pretrained("MostafaMaroof/Naqta", subfolder="span-tagger")
span_model = AutoModelForTokenClassification.from_pretrained(
    "MostafaMaroof/Naqta", subfolder="span-tagger")

Use sliding-window inference for long text. The model was fine-tuned on short windows; a single truncated pass leaves everything past roughly 230 words unpunctuated. See app.py in the Space for a reference implementation.

Evaluation

Two evaluations on deliberately different data. The gap between them is the main practical finding: Arabic punctuation is convention-dependent, so in-domain scores overstate performance on text punctuated to an external editorial standard.

1. In-domain test set

Held-out split of the training corpus (books, Wikipedia, diacritised text) — same sources and editorial conventions as training.

metric	score
Macro F1	0.8960
Token accuracy	0.9714

2. External guideline benchmark

91 Modern Standard Arabic passages (about 27k tokens) annotated by an independent Arabic language centre in Saudi Arabia, following its own published punctuation guideline. No overlap with the training data.

metric	previous release	current
Macro F1	0.5464	0.6160
Weighted F1	0.8990	0.9154
Token accuracy	0.9096	0.9133

Per-class F1 (current):

mark	O	:	؟	.	،	؛	-	!
F1	0.967	0.821	0.812	0.700	0.647	0.454	0.273	0.255

The drop from 0.896 to 0.616 is concentrated in marks whose placement depends on editorial convention rather than syntax. Structurally determined marks (question mark, colon) transfer well; the semicolon and exclamation, whose rules differ between style guides, do not.

Limitations

• Exclamation and semicolon are over-predicted (precision 0.204 and 0.362): the model finds these marks but does not withhold them reliably.
• Brackets and quotation marks are not produced. The span tagger covers them in training but their precision is too low to enable in production.
• Expect lower accuracy on text following a punctuation standard different from the training sources.

Training

XLM-RoBERTa-large fine-tuned on about 1.3M paragraphs (classical and modern books, Wikipedia, news, QA corpora) with weighted focal loss, oversampling of rare marks, and a final adaptation stage on guideline-conforming data.