billingsmoore commited on
Commit
0e5a5e0
·
verified ·
1 Parent(s): 6877941

Update repo references after rename to include nict-tib1

Browse files
Files changed (1) hide show
  1. README.md +9 -9
README.md CHANGED
@@ -17,7 +17,7 @@ metrics:
17
  - cer
18
  - wer
19
  model-index:
20
- - name: tibetan-asr-whisper-tiny-lora-8bit
21
  results:
22
  - task:
23
  type: automatic-speech-recognition
@@ -41,7 +41,7 @@ Fine-tuned **`openai/whisper-tiny`** for automatic speech recognition (ASR) on *
41
 
42
  > J. Moore, S. Li and P. Lauren, "Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics," in *IEEE Access*, vol. 14, pp. 101790-101805, 2026, doi: 10.1109/ACCESS.2026.3709206.
43
 
44
- This repo is one of 14 model checkpoints released with the paper, benchmarking 5 ASR architectures (Whisper Tiny/Base/Small, Wav2Vec 2.0 Base, HuBERT Base) under full fine-tuning, LoRA, and QLoRA (8-bit/4-bit) adaptation. See [billingsmoore/tibetan-asr-*](https://huggingface.co/billingsmoore) for the full set.
45
 
46
  ## Model description
47
 
@@ -82,15 +82,15 @@ Evaluated on the 1,547-utterance NICT-Tib1 test set under five metrics: Characte
82
 
83
  | Model | Repo | CER | SER | BoTok-SWER | BERT-SWER | Gem-SWER |
84
  |---|---|---|---|---|---|---|
85
- | HuBERT Base | [tibetan-asr-hubert-base](https://huggingface.co/billingsmoore/tibetan-asr-hubert-base) | 0.1352 | 0.3690 | 0.4477 | 0.161 | 0.9653 |
86
- | Wav2Vec 2.0 Base | [tibetan-asr-wav2vec2-base](https://huggingface.co/billingsmoore/tibetan-asr-wav2vec2-base) | 0.0745 | 0.2152 | 0.2747 | 0.097 | 0.7447 |
87
- | Whisper Tiny | [tibetan-asr-whisper-tiny](https://huggingface.co/billingsmoore/tibetan-asr-whisper-tiny) | 0.1560 | 0.2351 | 0.2975 | 0.118 | 0.6759 |
88
- | Whisper Base | [tibetan-asr-whisper-base](https://huggingface.co/billingsmoore/tibetan-asr-whisper-base) | 0.1417 | 0.2083 | 0.2600 | 0.105 | 0.6314 |
89
- | **Whisper Small** | [tibetan-asr-whisper-small](https://huggingface.co/billingsmoore/tibetan-asr-whisper-small) | **0.1185** | **0.1692** | **0.2042** | **0.086** | **0.5337** |
90
 
91
 
92
  > [!WARNING]
93
- > **This is a LoRA/QLoRA fine-tuned checkpoint. In the paper's benchmark, all LoRA and QLoRA configurations showed catastrophic word-level degradation** relative to full fine-tuning of the same base model, despite retaining partial character-level accuracy. This repo is published for reproducibility of that (negative) result, not as a recommended deployment artifact. If you need a usable Tibetan ASR model, use the [standard fine-tuned checkpoint](https://huggingface.co/billingsmoore/tibetan-asr-whisper-tiny) instead.
94
 
95
  ## How to use
96
 
@@ -103,7 +103,7 @@ bnb_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_skip_modules=None)
103
  base_model = WhisperForConditionalGeneration.from_pretrained(
104
  "openai/whisper-tiny", quantization_config=bnb_config, device_map="auto"
105
  )
106
- model = PeftModel.from_pretrained(base_model, "billingsmoore/tibetan-asr-whisper-tiny-lora-8bit")
107
  processor = WhisperProcessor.from_pretrained("openai/whisper-tiny", language="bo", task="transcribe")
108
 
109
  # generate as usual, e.g.:
 
17
  - cer
18
  - wer
19
  model-index:
20
+ - name: tibetan-asr-nict-tib1-whisper-tiny-lora-8bit
21
  results:
22
  - task:
23
  type: automatic-speech-recognition
 
41
 
42
  > J. Moore, S. Li and P. Lauren, "Evaluating Tibetan ASR With Segmented Word Error Rate: Beyond Character-Level Metrics," in *IEEE Access*, vol. 14, pp. 101790-101805, 2026, doi: 10.1109/ACCESS.2026.3709206.
43
 
44
+ This repo is one of 14 model checkpoints released with the paper, benchmarking 5 ASR architectures (Whisper Tiny/Base/Small, Wav2Vec 2.0 Base, HuBERT Base) under full fine-tuning, LoRA, and QLoRA (8-bit/4-bit) adaptation. See [billingsmoore/tibetan-asr-nict-tib1-*](https://huggingface.co/billingsmoore) for the full set.
45
 
46
  ## Model description
47
 
 
82
 
83
  | Model | Repo | CER | SER | BoTok-SWER | BERT-SWER | Gem-SWER |
84
  |---|---|---|---|---|---|---|
85
+ | HuBERT Base | [tibetan-asr-nict-tib1-hubert-base](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-hubert-base) | 0.1352 | 0.3690 | 0.4477 | 0.161 | 0.9653 |
86
+ | Wav2Vec 2.0 Base | [tibetan-asr-nict-tib1-wav2vec2-base](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-wav2vec2-base) | 0.0745 | 0.2152 | 0.2747 | 0.097 | 0.7447 |
87
+ | Whisper Tiny | [tibetan-asr-nict-tib1-whisper-tiny](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-tiny) | 0.1560 | 0.2351 | 0.2975 | 0.118 | 0.6759 |
88
+ | Whisper Base | [tibetan-asr-nict-tib1-whisper-base](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-base) | 0.1417 | 0.2083 | 0.2600 | 0.105 | 0.6314 |
89
+ | **Whisper Small** | [tibetan-asr-nict-tib1-whisper-small](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-small) | **0.1185** | **0.1692** | **0.2042** | **0.086** | **0.5337** |
90
 
91
 
92
  > [!WARNING]
93
+ > **This is a LoRA/QLoRA fine-tuned checkpoint. In the paper's benchmark, all LoRA and QLoRA configurations showed catastrophic word-level degradation** relative to full fine-tuning of the same base model, despite retaining partial character-level accuracy. This repo is published for reproducibility of that (negative) result, not as a recommended deployment artifact. If you need a usable Tibetan ASR model, use the [standard fine-tuned checkpoint](https://huggingface.co/billingsmoore/tibetan-asr-nict-tib1-whisper-tiny) instead.
94
 
95
  ## How to use
96
 
 
103
  base_model = WhisperForConditionalGeneration.from_pretrained(
104
  "openai/whisper-tiny", quantization_config=bnb_config, device_map="auto"
105
  )
106
+ model = PeftModel.from_pretrained(base_model, "billingsmoore/tibetan-asr-nict-tib1-whisper-tiny-lora-8bit")
107
  processor = WhisperProcessor.from_pretrained("openai/whisper-tiny", language="bo", task="transcribe")
108
 
109
  # generate as usual, e.g.: