README.md

---
library_name: transformers
tags:
- translation
- Tibetan
- Buddhism
- dharma
license: mit
language:
- bo
- en
metrics:
- bleu
- ter
- chrf
base_model:
- google-t5/t5-base
pipeline_tag: translation
---

# Model Card for mlotsawa-ground-base

This model is a transformers machine translation model for translating Tibetan Buddhist texts to English, produced as part of the larger [MLotsawa project](https://github.com/billingsmoore/MLotsawa) in collaboration with the Tibetan and Himalayan Library at the University of Virginia.

## Model Details

### Model Description

This model is a finetuned T5 model (base size) with 223 million parameters. It is intended for translation of Tibetan Buddhist texts into English. It expects input in Uchen script. 
This model uses the **[getok](https://huggingface.co/billingsmoore/getok-v0)** tokenizer.
Details on the training data and procedure can be found below.

This model is a *ground* model in that, while its performance is reasonably good, it is intended to be used as a base for further finetuning on either a larger corpus or a tradition-specific (i.e. Dzogchen) corpus for improved translation quality.

- **Developed by:** billingsmoore
- **Model type:** translation
- **Languages:** Tibetan, English
- **License:** MIT
- **Finetuned from model:** google-t5/t5-base

### Model Sources

- **Repository:** [MLotsawa on GitHub](https://github.com/billingsmoore/MLotsawa)
- **Paper code:** [optimizing-t5-tibetan-english-mt on GitHub](https://github.com/billingsmoore/optimizing-t5-tibetan-english-mt)
- **Paper:** Moore & Lauren (2025), *Optimizing T5 for Lightweight Tibetan-English Translation*, Research Square. [https://doi.org/10.21203/rs.3.rs-7409829/v1](https://doi.org/10.21203/rs.3.rs-7409829/v1)

## Uses

This model may be used directly for translation, or further finetuned for improved performance.

### Direct Use

This model can be used directly for translation using a transformers pipeline as in the code block below.

```python
from transformers import pipeline

pipe = pipeline('translation', 'billingsmoore/mlotsawa-ground-base', device='cpu') # select a device of your choice (i.e. 'cuda:0')

input = ["ཁྱེད་ལ་བསྟོད་ཅིང་གསོལ་བ་བཏབ་པའི་མཐུས༔",
"བདག་གི་ཚེ་བསོད་དཔལ་འབྱོར་རྒྱས་པ་དང་༔",
"འཇིགས་པ་བཅུ་དྲུག་རྐྱེན་ངན་བར་ཆད་སོལ༔"]

output = pipe(input)

translation = [elt['translation_text'] for elt in output]

print(translation)
```

The code above will produce the following output.

>['Through the power of praising and praying to you', 'Increase my lifespan merit and prosperity', 'Remove the sixteen fears and obstacles of adversity.']

Alternatively, this model is used as the free local-CPU translation backend in
[SimpleTranslationUI](https://huggingface.co/spaces/billingsmoore/SimpleTranslationUI), a Hugging
Face Space providing a graphical editor for translating `.txt`, `.docx`, and `.pdf` documents
(source: [GitHub](https://github.com/billingsmoore/SimpleTranslationUI)).

### Downstream Use

The performance of this model can be improved with additional finetuning. You might finetune using a larger dataset for better general performance or finetune on a specific set of material for improved performance on that subset (i.e. Dzogchen texts).

The model can be finetuned following the recipe below.

```python
# Load Your Data
from datasets import load_dataset

dataset = load_dataset(<your dataset>)

# Load the Model and Tokenizer
from transformers import AutoTokenizer, DataCollatorForSeq2Seq, AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("billingsmoore/mlotsawa-ground-base", device_map="cuda:0") # this line assumes you want to use a single CUDA enabled gpu
tokenizer = AutoTokenizer.from_pretrained('billingsmoore/mlotsawa-ground-base')
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

# Preprocess the Data
def translation_preprocess_function(examples):

    # Prepare translation inputs and targets
    translation_inputs = ['Translate Tibetan to English: ' + example for example in examples['bo']]
    translation_targets = [example for example in examples['en']]
    
    # Tokenize translation inputs and targets
    translation_model_inputs = tokenizer(translation_inputs, text_target=translation_targets, 
                                         max_length=256, truncation=True, padding="max_length")
    
    
    return translation_model_inputs

tokenized_dataset = dataset.map(translation_preprocess_function, batched=True)

# Define Evaluation Metrics
import numpy as np
import evaluate

# Load BLEU and CHRF metrics
bleu_metric = evaluate.load("sacrebleu")
chrf_metric = evaluate.load("chrf")
ter_metric = evaluate.load("ter")

def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [[label.strip()] for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    
    # Decode predictions and labels
    preds = np.where(preds != -100, preds, tokenizer.pad_token_id)
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Postprocess text
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    # Compute BLEU score
    bleu_result = bleu_metric.compute(predictions=decoded_preds, references=decoded_labels)
    bleu_score = bleu_result["score"]

    # Compute CHRF score
    chrf_result = chrf_metric.compute(predictions=decoded_preds, references=decoded_labels)
    chrf_score = chrf_result["score"]

    # Compute TER score
    ter_result = ter_metric.compute(predictions=decoded_preds, references=decoded_labels)
    ter_score = ter_result["score"]

    # Return rounded results
    metrics = {
        "bleu": round(bleu_score, 4),
        "chrf": round(chrf_score, 4),
        "ter": round(ter_score, 4)
    }

    #print("Computed Metrics:", metrics)

    return metrics

# Set Up Training Arguments and Optimizer
from transformers import Seq2SeqTrainingArguments, Seq2SeqTrainer, Adafactor, EarlyStoppingCallback
from accelerate import Accelerator

accelerator = Accelerator()

optimizer = Adafactor(
    model.parameters(), 
    scale_parameter=True, 
    relative_step=False, 
    warmup_init=False, 
    lr=3e-4
)

model, optimizer = accelerator.prepare(model, optimizer)

training_args = Seq2SeqTrainingArguments(
    output_dir=f"output-dir", # select an output directory of your choice
    auto_find_batch_size=True,
    predict_with_generate=True,
    fp16=False,
    push_to_hub=False,
    eval_strategy='epoch',
    save_strategy='epoch',
    num_train_epochs=100, # select your preferred number of training epochs
    load_best_model_at_end=True,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['dev'],
    processing_class=tokenizer,
    optimizers=(optimizer, None),
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    callbacks=[EarlyStoppingCallback()]
)

trainer.train()
```

## Bias, Risks, and Limitations

This model is intended for the translation of Buddhist texts. Because of the complexity and importance of this material, all translations should be treated as preliminary and should never be used without the input of an experienced human translator.

Additionally, this model was trained exclusively on Tibetan Buddhist material and should not be expected to perform well on other material (i.e. vernacular Tibetan).

## Training Details

### Training Data

The training data for this model was 861,417 translation pairs from Buddhist texts. This data was collected from publically available material as well as material generously provided by Monlam AI and the Tibetan and Himalayan Library.

### Training Procedure

The model underwent continued pretraining as well as finetuning as described below.

#### Pretraining

The model was pretrained on the training data for one epoch with a learning rate of 3e-4.The pretraining objective remained the original span corruption denoising task, in which random spans of input tokens are masked and the model is trained to reconstruct the missing content. This pretraining allowed the model to adapt to the new tokenizer and to learn the linguistic and structural characteristics of the Tibetan Buddhist materials. 

#### Finetuning

The model was finetuned on the translation pairs for 50 epochs using the Adafactor optimizer and an initial learning rate of 3e-4.

## Evaluation

The model was evaluated on test data with BLEU, chrF, and TER. The results are shown below.

BLEU|chrF|TER
----|----|---
4.08|20.85|86.74

These scores are exceptionally low, however actual translation results are relatively good. Sample translations are shown below.

From *Advice on Bending Mind Toward the Good* by Khenchen Ngawang Palzang | Translated by Joseph McClellan with editorial assistance from Ninjyed N.T., 2024.

| **Original** | **Human Translation** | **Machine Translation** |
|:-------------|:----------------------|:------------------------|
| གྲུབ་བརྒྱའི་སྤྱི་མེས་པཎ་ཆེན་བི་མ་ལ། །<br>བསམ་བཞིན་སྤྲུལ་པའི་ཟློས་གར་ཉེར་བཟུང་བ། །<br>རྒྱལ་བའི་དབང་པོ་ཀློང་ཆེན་རབ་འབྱམས་པ། །<br>འདི་ཙམ་མ་ཡིན་ཚེ་རབས་གཏན་གྱི་སྐྱབས། ། | Grandsire of a hundred siddhas—great scholar, Vimalamitra,<br>And you who fully embraced the spectacle of intentional emanation,<br>Lord of conquerors, Longchen Rabjam—<br>You are my unfailing refuge; not just now, but in the concatenation of my lives. | Great paṇḍita Vimalamitra, forefather of hundreds of siddhas,<br>Manifesting in the form of a play,<br>Lord of the victorious ones, Longchen Rabjam,<br>Not just this but the constant refuge throughout all my lives, |

From *Protection from All Fears A Prayer to Ārya Tārā from the Reality Ḍākinīs’ Secret Treasury (Chönyi Khandrö Sangdzö)* by Sera Khandro | Translated by Adam Pearcey, 2025.

| **Original** | **Human Translation** | **Machine Translation** |
|:-------------|:----------------------|:------------------------|
|ཀ་དག་སྤྲོས་བྲལ་འོད་གསལ་རིག་པའི་དབྱིངས༔<br>ལྷུན་གྲུབ་སྣང་ཆ་མ་འགགས་སྒྱུ་འཕྲུལ་གར༔<br>ཐུགས་རྗེ་རྒྱལ་བ་ཀུན་གྱི་ཡུམ་གཅིག་མ༔<br>རྗེ་བཙུན་ཨཱརྱ་ཏཱ་རེ་ཚེ་སྦྱིན་དཔལ༔<br>གསོལ་བ་འདེབས་སོ་རླུང་སེམས་དབང་བསྡུས་ནས༔<br>ཚེ་དང་བསོད་ནམས་འཕེལ་བར་མཛད་དུ་གསོལ༔ | Out of the primordially pure unelaborate space of luminous awareness,<br>As the magical manifestation of unobstructed spontaneous presence,<br>Arises the compassionate one, the one and only mother of all victorious ones,<br>Noble Lady Ārya Tārā, glorious bestower of longevity,<br>To you I pray! Take control of my vital winds and mind,<br>And increase my lifespan and merit! | Within the space of awareness—primordial purity free of elaboration—<br>Illusory dance of spontaneously present appearances unceasing<br>Only mother of all the buddhas of compassion<br>Noble Ārya Tārā glorious Tārā<br>To you I pray: bringing the vāyu-mind under control<br>And increase our lifespan and merit.|


## Citation

If you use this model, please cite the paper it was introduced in:

```bibtex
@article{moore2025optimizing,
  title   = {Optimizing T5 for Lightweight Tibetan-English Translation},
  author  = {Moore, Jacob and Lauren, Paula},
  year    = {2025},
  journal = {Research Square},
  doi     = {10.21203/rs.3.rs-7409829/v1},
  url     = {https://doi.org/10.21203/rs.3.rs-7409829/v1},
  note    = {Preprint}
}
```

## Model Card Authors

billingsmoore

## Model Card Contact

billingsmoore[at]gmail[dot]com