Update README.md
Browse files
README.md
CHANGED
|
@@ -16,23 +16,17 @@ pipeline_tag: text2text-generation
|
|
| 16 |
|
| 17 |
# byt5-mitra-bo-tagger
|
| 18 |
|
| 19 |
-
A byte-level seq2seq tagger for
|
| 20 |
-
Tibetan in Buddhist studies who read Tibetan translations against their Sanskrit originals, and at
|
| 21 |
-
dictionary building. Given a block of raw Wylie it returns, in one terse string:
|
| 22 |
|
| 23 |
1. **sentence segmentation** (one sentence per line),
|
| 24 |
-
2. **dictionary-lexeme word segmentation**
|
| 25 |
-
`sems_can`, `byang_chub_sems_dpa'`, `tshe_dang_ldan_pa` stay whole; case particles, clause particles,
|
| 26 |
-
clitics and attached affixes are separate),
|
| 27 |
3. a **coarse part of speech** for every word (noun / verb / verbal noun / adjective / particle …), and
|
| 28 |
4. **Sanskrit-unit brackets**: runs of Tibetan words that render one Sanskrit word or compound
|
| 29 |
(`[chos/N kyi/C rgyal_po/N]` = dharmarāja).
|
| 30 |
|
| 31 |
It is a fine-tune of [buddhist-nlp/byt5-mitra-bo](https://huggingface.co/buddhist-nlp/byt5-mitra-bo)
|
| 32 |
(ByT5 continued-pretrained on Tibetan Wylie) on ~5k sentences and short paragraphs from the
|
| 33 |
-
Sanskrit–Tibetan parallel corpus of the [Dharmamitra](https://dharmamitra.org) project.
|
| 34 |
-
annotations were produced by Gemini from the Tibetan, the Sanskrit original with a morphological
|
| 35 |
-
analysis, and automatic Sanskrit–Tibetan word alignments, then validated to reproduce the input exactly.
|
| 36 |
|
| 37 |
## Output scheme
|
| 38 |
|
|
@@ -50,9 +44,6 @@ POS letters: **N** noun · **P** proper noun · **V** verb (finite / predicate s
|
|
| 50 |
(demonstrative, plural, indefinite) · **X** negation · **C** case particle · **K** clause particle
|
| 51 |
(after verbs) · **L** clitic / final particle · **I** interjection · **p** punctuation.
|
| 52 |
|
| 53 |
-
The surface text is recoverable (`_` → space, drop `+`, `|` → `/`), so outputs can be validated
|
| 54 |
-
against the input; ~10% of outputs fail that check (a dropped or altered syllable) and should be
|
| 55 |
-
rejected or retried.
|
| 56 |
|
| 57 |
```
|
| 58 |
input : de nas tshe dang ldan pa kun dga' bos bcom ldan 'das la 'di skad ces gsol to/ /
|
|
@@ -84,15 +75,13 @@ Inputs: Wylie (EWTS) as found in ACIP / DharmaNexus e-texts, up to ~700 characte
|
|
| 84 |
|
| 85 |
## Evaluation
|
| 86 |
|
| 87 |
-
Held-out 100 items
|
| 88 |
|
| 89 |
| model | valid output | word boundary F1 | POS acc | verb/noun acc | sentence F1 | Sanskrit-unit F1 |
|
| 90 |
|---|---|---|---|---|---|---|
|
| 91 |
| byt5-mitra-bo-tagger (this model, 10 epochs) | .90 | .959 | .914 | .914 | .914 | .663 |
|
| 92 |
| Qwen3.5-9B mitra base, same data | 1.00 | .962 | .942 | .909 | .891 | .687 |
|
| 93 |
|
| 94 |
-
Two runs of the teacher itself agree at word F1 .974 / POS .947, so the tagger is close to the
|
| 95 |
-
teacher's own consistency on segmentation.
|
| 96 |
|
| 97 |
## Limitations
|
| 98 |
|
|
|
|
| 16 |
|
| 17 |
# byt5-mitra-bo-tagger
|
| 18 |
|
| 19 |
+
A byte-level seq2seq tagger for Classical Tibetan in Wylie transliteration. Given a block of raw Wylie it returns, in one terse string:
|
|
|
|
|
|
|
| 20 |
|
| 21 |
1. **sentence segmentation** (one sentence per line),
|
| 22 |
+
2. **dictionary-lexeme word segmentation**
|
|
|
|
|
|
|
| 23 |
3. a **coarse part of speech** for every word (noun / verb / verbal noun / adjective / particle …), and
|
| 24 |
4. **Sanskrit-unit brackets**: runs of Tibetan words that render one Sanskrit word or compound
|
| 25 |
(`[chos/N kyi/C rgyal_po/N]` = dharmarāja).
|
| 26 |
|
| 27 |
It is a fine-tune of [buddhist-nlp/byt5-mitra-bo](https://huggingface.co/buddhist-nlp/byt5-mitra-bo)
|
| 28 |
(ByT5 continued-pretrained on Tibetan Wylie) on ~5k sentences and short paragraphs from the
|
| 29 |
+
Sanskrit–Tibetan parallel corpus of the [Dharmamitra](https://dharmamitra.org) project.
|
|
|
|
|
|
|
| 30 |
|
| 31 |
## Output scheme
|
| 32 |
|
|
|
|
| 44 |
(demonstrative, plural, indefinite) · **X** negation · **C** case particle · **K** clause particle
|
| 45 |
(after verbs) · **L** clitic / final particle · **I** interjection · **p** punctuation.
|
| 46 |
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
```
|
| 49 |
input : de nas tshe dang ldan pa kun dga' bos bcom ldan 'das la 'di skad ces gsol to/ /
|
|
|
|
| 75 |
|
| 76 |
## Evaluation
|
| 77 |
|
| 78 |
+
Held-out 100 items:
|
| 79 |
|
| 80 |
| model | valid output | word boundary F1 | POS acc | verb/noun acc | sentence F1 | Sanskrit-unit F1 |
|
| 81 |
|---|---|---|---|---|---|---|
|
| 82 |
| byt5-mitra-bo-tagger (this model, 10 epochs) | .90 | .959 | .914 | .914 | .914 | .663 |
|
| 83 |
| Qwen3.5-9B mitra base, same data | 1.00 | .962 | .942 | .909 | .891 | .687 |
|
| 84 |
|
|
|
|
|
|
|
| 85 |
|
| 86 |
## Limitations
|
| 87 |
|