Automatic Speech Recognition
PEFT
Safetensors
Tibetan
tibetan
low-resource
whisper
lora
8bit
Eval Results (legacy)
Instructions to use billingsmoore/tibetan-asr-nict-tib1-whisper-tiny-lora-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use billingsmoore/tibetan-asr-nict-tib1-whisper-tiny-lora-8bit with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Update repo references after rename to include nict-tib1
Browse files
README.md
CHANGED
|
@@ -17,7 +17,7 @@ metrics:
|
|
| 17 |
- cer
|
| 18 |
- wer
|
| 19 |
model-index:
|
| 20 |
-
- name: tibetan-asr-whisper-tiny-lora-8bit
|
| 21 |
results:
|
| 22 |
- task:
|
| 23 |
type: automatic-speech-recognition
|
|
@@ -41,7 +41,7 @@ Fine-tuned **`openai/whisper-tiny`** for automatic speech recognition (ASR) on *
|
|
| 41 |
|
| 42 |
> J. Moore, S. Li and P. Lauren, "Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics," in *IEEE Access*, vol. 14, pp. 101790-101805, 2026, doi: 10.1109/ACCESS.2026.3709206.
|
| 43 |
|
| 44 |
-
This repo is one of 14 model checkpoints released with the paper, benchmarking 5 ASR architectures (Whisper Tiny/Base/Small, Wav2Vec 2.0 Base, HuBERT Base) under full fine-tuning, LoRA, and QLoRA (8-bit/4-bit) adaptation. See [billingsmoore/tibetan-asr-*](https://huggingface.co/billingsmoore) for the full set.
|
| 45 |
|
| 46 |
## Model description
|
| 47 |
|
|
@@ -82,15 +82,15 @@ Evaluated on the 1,547-utterance NICT-Tib1 test set under five metrics: Characte
|
|
| 82 |
|
| 83 |
| Model | Repo | CER | SER | BoTok-SWER | BERT-SWER | Gem-SWER |
|
| 84 |
|---|---|---|---|---|---|---|
|
| 85 |
-
| HuBERT Base | [tibetan-asr-hubert-base](https://huggingface.co/billingsmoore/tibetan-asr-hubert-base) | 0.1352 | 0.3690 | 0.4477 | 0.161 | 0.9653 |
|
| 86 |
-
| Wav2Vec 2.0 Base | [tibetan-asr-wav2vec2-base](https://huggingface.co/billingsmoore/tibetan-asr-wav2vec2-base) | 0.0745 | 0.2152 | 0.2747 | 0.097 | 0.7447 |
|
| 87 |
-
| Whisper Tiny | [tibetan-asr-whisper-tiny](https://huggingface.co/billingsmoore/tibetan-asr-whisper-tiny) | 0.1560 | 0.2351 | 0.2975 | 0.118 | 0.6759 |
|
| 88 |
-
| Whisper Base | [tibetan-asr-whisper-base](https://huggingface.co/billingsmoore/tibetan-asr-whisper-base) | 0.1417 | 0.2083 | 0.2600 | 0.105 | 0.6314 |
|
| 89 |
-
| **Whisper Small** | [tibetan-asr-whisper-small](https://huggingface.co/billingsmoore/tibetan-asr-whisper-small) | **0.1185** | **0.1692** | **0.2042** | **0.086** | **0.5337** |
|
| 90 |
|
| 91 |
|
| 92 |
> [!WARNING]
|
| 93 |
-
> **This is a LoRA/QLoRA fine-tuned checkpoint. In the paper's benchmark, all LoRA and QLoRA configurations showed catastrophic word-level degradation** relative to full fine-tuning of the same base model, despite retaining partial character-level accuracy. This repo is published for reproducibility of that (negative) result, not as a recommended deployment artifact. If you need a usable Tibetan ASR model, use the [standard fine-tuned checkpoint](https://huggingface.co/billingsmoore/tibetan-asr-whisper-tiny) instead.
|
| 94 |
|
| 95 |
## How to use
|
| 96 |
|
|
@@ -103,7 +103,7 @@ bnb_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_skip_modules=None)
|
|
| 103 |
base_model = WhisperForConditionalGeneration.from_pretrained(
|
| 104 |
"openai/whisper-tiny", quantization_config=bnb_config, device_map="auto"
|
| 105 |
)
|
| 106 |
-
model = PeftModel.from_pretrained(base_model, "billingsmoore/tibetan-asr-whisper-tiny-lora-8bit")
|
| 107 |
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny", language="bo", task="transcribe")
|
| 108 |
|
| 109 |
# generate as usual, e.g.:
|
|
|
|
| 17 |
- cer
|
| 18 |
- wer
|
| 19 |
model-index:
|
| 20 |
+
- name: tibetan-asr-nict-tib1-whisper-tiny-lora-8bit
|
| 21 |
results:
|
| 22 |
- task:
|
| 23 |
type: automatic-speech-recognition
|
|
|
|
| 41 |
|
| 42 |
> J. Moore, S. Li and P. Lauren, "Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics," in *IEEE Access*, vol. 14, pp. 101790-101805, 2026, doi: 10.1109/ACCESS.2026.3709206.
|
| 43 |
|
| 44 |
+
This repo is one of 14 model checkpoints released with the paper, benchmarking 5 ASR architectures (Whisper Tiny/Base/Small, Wav2Vec 2.0 Base, HuBERT Base) under full fine-tuning, LoRA, and QLoRA (8-bit/4-bit) adaptation. See [billingsmoore/tibetan-asr-nict-tib1-*](https://huggingface.co/billingsmoore) for the full set.
|
| 45 |
|
| 46 |
## Model description
|
| 47 |
|
|
|
|
| 82 |
|
| 83 |
| Model | Repo | CER | SER | BoTok-SWER | BERT-SWER | Gem-SWER |
|
| 84 |
|---|---|---|---|---|---|---|
|
| 85 |
+
| HuBERT Base | [tibetan-asr-nict-tib1-hubert-base](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-hubert-base) | 0.1352 | 0.3690 | 0.4477 | 0.161 | 0.9653 |
|
| 86 |
+
| Wav2Vec 2.0 Base | [tibetan-asr-nict-tib1-wav2vec2-base](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-wav2vec2-base) | 0.0745 | 0.2152 | 0.2747 | 0.097 | 0.7447 |
|
| 87 |
+
| Whisper Tiny | [tibetan-asr-nict-tib1-whisper-tiny](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-tiny) | 0.1560 | 0.2351 | 0.2975 | 0.118 | 0.6759 |
|
| 88 |
+
| Whisper Base | [tibetan-asr-nict-tib1-whisper-base](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-base) | 0.1417 | 0.2083 | 0.2600 | 0.105 | 0.6314 |
|
| 89 |
+
| **Whisper Small** | [tibetan-asr-nict-tib1-whisper-small](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-small) | **0.1185** | **0.1692** | **0.2042** | **0.086** | **0.5337** |
|
| 90 |
|
| 91 |
|
| 92 |
> [!WARNING]
|
| 93 |
+
> **This is a LoRA/QLoRA fine-tuned checkpoint. In the paper's benchmark, all LoRA and QLoRA configurations showed catastrophic word-level degradation** relative to full fine-tuning of the same base model, despite retaining partial character-level accuracy. This repo is published for reproducibility of that (negative) result, not as a recommended deployment artifact. If you need a usable Tibetan ASR model, use the [standard fine-tuned checkpoint](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-tiny) instead.
|
| 94 |
|
| 95 |
## How to use
|
| 96 |
|
|
|
|
| 103 |
base_model = WhisperForConditionalGeneration.from_pretrained(
|
| 104 |
"openai/whisper-tiny", quantization_config=bnb_config, device_map="auto"
|
| 105 |
)
|
| 106 |
+
model = PeftModel.from_pretrained(base_model, "billingsmoore/tibetan-asr-nict-tib1-whisper-tiny-lora-8bit")
|
| 107 |
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny", language="bo", task="transcribe")
|
| 108 |
|
| 109 |
# generate as usual, e.g.:
|