Emin009 commited on
Commit
260aa48
ยท
verified ยท
1 Parent(s): afe183a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +59 -2
README.md CHANGED
@@ -1,4 +1,61 @@
1
  ---
2
- library_name: transformers
 
 
 
 
 
 
 
 
3
  license: apache-2.0
4
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - ar
4
+ - mey
5
+ tags:
6
+ - punctuation-restoration
7
+ - token-classification
8
+ - marbert
9
+ - mauritania
10
+ - dialect
11
  license: apache-2.0
12
+ datasets:
13
+ - hassaniya-punctuation-dataset
14
+ ---
15
+
16
+ # Hassaniya Punctuation Restoration Model (ุงู„ุญุณุงู†ูŠุฉ)
17
+
18
+ This is the first open-source model dedicated to restoring punctuation for the **Hassaniya Arabic dialect** (spoken in Mauritania, Southern Morocco, etc.).
19
+
20
+ It restores the following punctuation marks:
21
+ - Period (.)
22
+ - Comma (ุŒ)
23
+ - Question Mark (ุŸ)
24
+ - Exclamation Mark (!)
25
+ - Colon (:)
26
+
27
+ ## Model Details
28
+ - **Base Model:** `UBC-NLP/MARBERTv2` (State-of-the-art for Arabic Dialects).
29
+ - **Training Data:** ~10,000 sentences of Hassaniya dialogue and narrative text.
30
+ - **Task:** Token Classification (The model tags each word with the punctuation that should follow it).
31
+
32
+ ## How to use
33
+
34
+ You can use this model directly with the Hugging Face `pipeline`:
35
+
36
+ ```python
37
+ from transformers import pipeline
38
+
39
+ # Load model
40
+ model_name = "Emin009/hassaniya-punctuation-restoration"
41
+ punct_pipe = pipeline("token-classification", model=model_name, aggregation_strategy="simple")
42
+
43
+ # Define punctuation map
44
+ punct_map = {"PERIOD": ".", "COMMA": "ุŒ", "QUESTION": "ุŸ", "EXCLAM": "!", "COLON": ":"}
45
+
46
+ def restore_punct(text):
47
+ results = punct_pipe(text)
48
+ output_text = ""
49
+ last_idx = 0
50
+ for res in results:
51
+ output_text += text[last_idx:res['end']]
52
+ if res['entity_group'] in punct_map:
53
+ output_text += punct_map[res['entity_group']]
54
+ last_idx = res['end']
55
+ output_text += text[last_idx:]
56
+ return output_text
57
+
58
+ # Test
59
+ text = "ู…ู†ู‡ูˆ ู‡ุฐุง ุงู„ุฑุงุฌู„ ุฌุง ู…ู†ูŠู†"
60
+ print(restore_punct(text))
61
+ # Output: ู…ู†ู‡ูˆ ู‡ุฐุง ุงู„ุฑุงุฌู„ุŸ ุฌุง ู…ู†ูŠู†ุŸ