README.md

---
language:
- ar
- mey
tags:
- punctuation-restoration
- token-classification
- marbert
- mauritania
- dialect
license: apache-2.0
datasets:
- hassaniya-punctuation-dataset
---

# Hassaniya Punctuation Restoration Model (الحسانية)

This is the first open-source model dedicated to restoring punctuation for the **Hassaniya Arabic dialect** (spoken in Mauritania, Southern Morocco, etc.). 

It restores the following punctuation marks:
- Period (.)
- Comma (،)
- Question Mark (؟)
- Exclamation Mark (!)
- Colon (:)

## Model Details
- **Base Model:** `UBC-NLP/MARBERTv2` (State-of-the-art for Arabic Dialects).
- **Training Data:** ~10,000 sentences of Hassaniya dialogue and narrative text.
- **Task:** Token Classification (The model tags each word with the punctuation that should follow it).

## How to use

You can use this model directly with the Hugging Face `pipeline`:

```python
from transformers import pipeline

# Load model
model_name = "Emin009/hassaniya-punctuation-restoration"
punct_pipe = pipeline("token-classification", model=model_name, aggregation_strategy="simple")

# Define punctuation map
punct_map = {"PERIOD": ".", "COMMA": "،", "QUESTION": "؟", "EXCLAM": "!", "COLON": ":"}

def restore_punct(text):
    results = punct_pipe(text)
    output_text = ""
    last_idx = 0
    for res in results:
        output_text += text[last_idx:res['end']]
        if res['entity_group'] in punct_map:
            output_text += punct_map[res['entity_group']]
        last_idx = res['end']
    output_text += text[last_idx:]
    return output_text

# Test
text = "منهو هذا الراجل جا منين"
print(restore_punct(text))
# Output: منهو هذا الراجل؟ جا منين؟