snehrdich commited on
Commit
dfcaaa8
·
verified ·
1 Parent(s): c0ea6e7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -15
README.md CHANGED
@@ -16,23 +16,17 @@ pipeline_tag: text2text-generation
16
 
17
  # byt5-mitra-bo-tagger
18
 
19
- A byte-level seq2seq tagger for **Classical Tibetan in Wylie transliteration**, aimed at learners of
20
- Tibetan in Buddhist studies who read Tibetan translations against their Sanskrit originals, and at
21
- dictionary building. Given a block of raw Wylie it returns, in one terse string:
22
 
23
  1. **sentence segmentation** (one sentence per line),
24
- 2. **dictionary-lexeme word segmentation** (units a learner looks up in Hopkins / Rangjung Yeshe / Negi:
25
- `sems_can`, `byang_chub_sems_dpa'`, `tshe_dang_ldan_pa` stay whole; case particles, clause particles,
26
- clitics and attached affixes are separate),
27
  3. a **coarse part of speech** for every word (noun / verb / verbal noun / adjective / particle …), and
28
  4. **Sanskrit-unit brackets**: runs of Tibetan words that render one Sanskrit word or compound
29
  (`[chos/N kyi/C rgyal_po/N]` = dharmarāja).
30
 
31
  It is a fine-tune of [buddhist-nlp/byt5-mitra-bo](https://huggingface.co/buddhist-nlp/byt5-mitra-bo)
32
  (ByT5 continued-pretrained on Tibetan Wylie) on ~5k sentences and short paragraphs from the
33
- Sanskrit–Tibetan parallel corpus of the [Dharmamitra](https://dharmamitra.org) project. The training
34
- annotations were produced by Gemini from the Tibetan, the Sanskrit original with a morphological
35
- analysis, and automatic Sanskrit–Tibetan word alignments, then validated to reproduce the input exactly.
36
 
37
  ## Output scheme
38
 
@@ -50,9 +44,6 @@ POS letters: **N** noun · **P** proper noun · **V** verb (finite / predicate s
50
  (demonstrative, plural, indefinite) · **X** negation · **C** case particle · **K** clause particle
51
  (after verbs) · **L** clitic / final particle · **I** interjection · **p** punctuation.
52
 
53
- The surface text is recoverable (`_` → space, drop `+`, `|` → `/`), so outputs can be validated
54
- against the input; ~10% of outputs fail that check (a dropped or altered syllable) and should be
55
- rejected or retried.
56
 
57
  ```
58
  input : de nas tshe dang ldan pa kun dga' bos bcom ldan 'das la 'di skad ces gsol to/ /
@@ -84,15 +75,13 @@ Inputs: Wylie (EWTS) as found in ACIP / DharmaNexus e-texts, up to ~700 characte
84
 
85
  ## Evaluation
86
 
87
- Held-out 100 items (reference = the Gemini teacher annotation):
88
 
89
  | model | valid output | word boundary F1 | POS acc | verb/noun acc | sentence F1 | Sanskrit-unit F1 |
90
  |---|---|---|---|---|---|---|
91
  | byt5-mitra-bo-tagger (this model, 10 epochs) | .90 | .959 | .914 | .914 | .914 | .663 |
92
  | Qwen3.5-9B mitra base, same data | 1.00 | .962 | .942 | .909 | .891 | .687 |
93
 
94
- Two runs of the teacher itself agree at word F1 .974 / POS .947, so the tagger is close to the
95
- teacher's own consistency on segmentation.
96
 
97
  ## Limitations
98
 
 
16
 
17
  # byt5-mitra-bo-tagger
18
 
19
+ A byte-level seq2seq tagger for Classical Tibetan in Wylie transliteration. Given a block of raw Wylie it returns, in one terse string:
 
 
20
 
21
  1. **sentence segmentation** (one sentence per line),
22
+ 2. **dictionary-lexeme word segmentation**
 
 
23
  3. a **coarse part of speech** for every word (noun / verb / verbal noun / adjective / particle …), and
24
  4. **Sanskrit-unit brackets**: runs of Tibetan words that render one Sanskrit word or compound
25
  (`[chos/N kyi/C rgyal_po/N]` = dharmarāja).
26
 
27
  It is a fine-tune of [buddhist-nlp/byt5-mitra-bo](https://huggingface.co/buddhist-nlp/byt5-mitra-bo)
28
  (ByT5 continued-pretrained on Tibetan Wylie) on ~5k sentences and short paragraphs from the
29
+ Sanskrit–Tibetan parallel corpus of the [Dharmamitra](https://dharmamitra.org) project.
 
 
30
 
31
  ## Output scheme
32
 
 
44
  (demonstrative, plural, indefinite) · **X** negation · **C** case particle · **K** clause particle
45
  (after verbs) · **L** clitic / final particle · **I** interjection · **p** punctuation.
46
 
 
 
 
47
 
48
  ```
49
  input : de nas tshe dang ldan pa kun dga' bos bcom ldan 'das la 'di skad ces gsol to/ /
 
75
 
76
  ## Evaluation
77
 
78
+ Held-out 100 items:
79
 
80
  | model | valid output | word boundary F1 | POS acc | verb/noun acc | sentence F1 | Sanskrit-unit F1 |
81
  |---|---|---|---|---|---|---|
82
  | byt5-mitra-bo-tagger (this model, 10 epochs) | .90 | .959 | .914 | .914 | .914 | .663 |
83
  | Qwen3.5-9B mitra base, same data | 1.00 | .962 | .942 | .909 | .891 | .687 |
84
 
 
 
85
 
86
  ## Limitations
87