Add 🤗 Transformers weights/processor (converted + verified against the .nemo); card: Transformers usage

README.md

@@ -19,6 +19,7 @@ tags:
 - FastConformer
 - RNNT
 - NeMo
+- transformers
 - persian
 - farsi
 model-index:
@@ -85,18 +86,86 @@ Word error rate (WER) and character error rate (CER), in %. Lower is better. All
 
 ## How to use
 
-Install [NeMo](https://github.com/NVIDIA-NeMo/Speech) 3.0 (`pip install "nemo_toolkit[asr]>=3.0"`; Python ≥ 3.11). Then download `nemotron-asr-streaming-farsi.nemo` from this repo:
+The model works with **🤗 Transformers** (≥ 5.18, files in this repo's root) and with **NVIDIA NeMo** (`nemotron-asr-streaming-farsi.nemo`). Both give the same results. The Transformers weights are bit-identical to the `.nemo`, and on FLEURS (852 clips) it scores **8.81% WER at 1.12 s look-ahead vs 8.77%** for NeMo, and 9.02% vs 9.13% at 0.32 s. In streaming, 98 of 100 transcripts were identical.
+
+### 🤗 Transformers: transcribe a file
+
+```python
+from transformers import AutoModelForRNNT, AutoProcessor
+from transformers.audio_utils import load_audio
+
+model_id = "mehdi-hf/nemotron-asr-streaming-farsi"
+processor = AutoProcessor.from_pretrained(model_id)
+model = AutoModelForRNNT.from_pretrained(model_id, device_map="auto")
+
+processor.set_num_lookahead_tokens(13)   # 1.12 s look-ahead (most accurate); 3 = 0.32 s (default), 6, 0
+audio = load_audio("audio.mp3", sampling_rate=processor.feature_extractor.sampling_rate)
+inputs = processor(audio, sampling_rate=processor.feature_extractor.sampling_rate, language="fa-IR")
+inputs = inputs.to(model.device, dtype=model.dtype)
+output = model.generate(**inputs, return_dict_in_generate=True)
+print(processor.decode(output.sequences, skip_special_tokens=True)[0])
+```
+
+`language` accepts only `"fa-IR"`, `"fa"` or `"auto"`, and all three select the Persian prompt the model was trained with. The other languages of the base model aren't supported.
+
+**Fine-tuning:** use NeMo. Transformers 5.18 can't compute this model's training loss. It has no loss entry for `Nemotron3_5AsrForRNNT` and falls back to a causal-LM loss; NVIDIA's original model has the same limitation. The processor's `labels`/`decoder_input_ids` are correct for this model, so this will work once Transformers adds the loss.
+
+### 🤗 Transformers: live streaming
+
+```python
+from threading import Thread
+import numpy as np
+from transformers import AutoModelForRNNT, AutoProcessor, TextIteratorStreamer
+from transformers.audio_utils import load_audio
+
+model_id = "mehdi-hf/nemotron-asr-streaming-farsi"
+processor = AutoProcessor.from_pretrained(model_id)
+model = AutoModelForRNNT.from_pretrained(model_id, device_map="auto")
+processor.set_num_lookahead_tokens(13)
+
+sr = processor.feature_extractor.sampling_rate
+audio = load_audio("audio.mp3", sampling_rate=sr)
+# Chunks are processed whole and the model holds back its look-ahead frames, so append two chunks of
+# silence; otherwise the last ~1 s of speech is never transcribed. (A live app does this when it stops.)
+audio = np.concatenate([audio, np.zeros(2 * processor.num_samples_per_audio_chunk, dtype=np.float32)])
+first = processor(audio[: processor.num_samples_first_audio_chunk], sampling_rate=sr, is_streaming=True,
+                  is_first_audio_chunk=True, language="fa-IR", return_tensors="pt").to(model.device, dtype=model.dtype)
+
+def chunks():   # in a live app, feed microphone audio here as it arrives
+    yield first.input_features[:, : processor.num_mel_frames_first_audio_chunk, :]
+    mel, hop, n_fft = processor.num_mel_frames_first_audio_chunk, processor.feature_extractor.hop_length, processor.feature_extractor.n_fft
+    start = mel * hop - n_fft // 2
+    while (end := start + processor.num_samples_per_audio_chunk) < audio.shape[0]:
+        x = processor(audio[start:end], sampling_rate=sr, is_streaming=True, is_first_audio_chunk=False,
+                      language="fa-IR", return_tensors="pt").to(model.device, dtype=model.dtype)
+        yield x.input_features
+        mel += processor.num_mel_frames_per_audio_chunk
+        start = mel * hop - n_fft // 2
+
+# group_tokens=False is required: the tokenizer's decode merges repeated tokens by default (a CTC rule),
+# which would drop letters from RNN-T output. processor.decode sets it for you; the streamer doesn't.
+streamer = TextIteratorStreamer(processor.tokenizer, skip_special_tokens=True, group_tokens=False)
+Thread(target=model.generate, kwargs={**first, "input_features": chunks(), "streamer": streamer}).start()
+for text in streamer:
+    print(text, end="", flush=True)
+```
+
+### NeMo
+
+Install [NeMo](https://github.com/NVIDIA-NeMo/Speech) 3.0 (`pip install "nemo_toolkit[asr]>=3.0"`; Python ≥ 3.11). Then download the `.nemo` file:
 
 ```python
 from huggingface_hub import hf_hub_download
 model_path = hf_hub_download("mehdi-hf/nemotron-asr-streaming-farsi", "nemotron-asr-streaming-farsi.nemo")
 ```
 
-> **Set the `fa-IR` prompt.** This model was fine-tuned only with the `fa-IR` language prompt.
+> **Set the `fa-IR` prompt in NeMo.** This model was fine-tuned only with the `fa-IR` language prompt.
 > - In streaming, call `model.set_inference_prompt("fa-IR")`.
 > - Don't use `model.transcribe()` or NeMo's `speech_to_text_eval.py` as they are. In NeMo 3.0 their dataloader picks the prompt **at random per utterance** (the untrained `auto` prompt about half the time), which gives much worse, non-reproducible output.
+> - The Transformers version doesn't have this problem: it always uses the Persian prompt.
+
+#### NeMo streaming (Python)
 
-### Streaming (Python)
 
 ```python
 import librosa, soundfile as sf, torch
@@ -133,7 +202,7 @@ with torch.inference_mode():
 
 The loop prints the transcript as it grows, chunk by chunk. On an Apple M1 Pro (`mps`) it runs about 5–10× faster than real time.
 
-### Streaming (command line, NeMo script)
+#### NeMo streaming (command line)
 
 ```bash
 git clone --depth 1 --branch v3.0.0 https://github.com/NVIDIA-NeMo/Speech.git nemo-speech
@@ -188,6 +257,7 @@ The raw data was cut and filtered to 1,181 h:
 
 - **Conversational speech is still hard:** ~26% WER on spontaneous film and YouTube speech with music and noise, against ~9% on read speech.
 - **Spacing variants still cost a little WER:** compound words written with or without a space (`چندتا` / `چند تا`) count as errors.
+- **End of a stream (Transformers):** Transformers 5.18's streaming accepts only full-size chunks, so the end of the audio is padded with silence. A word cut off by the very end of a recording can then be dropped; whole-file mode and NeMo's streaming keep it.
 - **Not evaluated on dialects:** the training data is Iranian media and podcast speech. Performance on regional accents, Dari or Tajik hasn't been measured.
 - **Evaluation coverage:** FLEURS Persian test speakers are all male.
 

config.json

@@ -0,0 +1,52 @@
+{
+  "architectures": [
+    "Nemotron3_5AsrForRNNT"
+  ],
+  "blank_token_id": 1024,
+  "decoder_hidden_size": 640,
+  "default_prompt_id": 38,
+  "dtype": "float32",
+  "encoder_config": {
+    "activation_dropout": 0.1,
+    "attention_bias": false,
+    "attention_dropout": 0.1,
+    "conv_kernel_size": 9,
+    "convolution_bias": false,
+    "default_num_lookahead_tokens": 3,
+    "dropout": 0.1,
+    "dropout_positions": 0.0,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layerdrop": 0.1,
+    "max_position_embeddings": 5000,
+    "model_type": "nemotron_asr_streaming_encoder",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 24,
+    "num_key_value_heads": 8,
+    "num_mel_bins": 128,
+    "scale_input": false,
+    "sliding_window": 57,
+    "subsampling_conv_channels": 256,
+    "subsampling_conv_kernel_size": 3,
+    "subsampling_conv_stride": 2,
+    "subsampling_factor": 8,
+    "supported_num_lookahead_tokens": [
+      3,
+      0,
+      6,
+      13
+    ]
+  },
+  "hidden_act": "relu",
+  "is_encoder_decoder": true,
+  "max_symbols_per_step": 10,
+  "model_type": "nemotron3_5_asr",
+  "num_decoder_layers": 2,
+  "num_prompts": 128,
+  "pad_token_id": 0,
+  "prompt_intermediate_size": 2048,
+  "transformers_version": "5.18.0",
+  "vocab_size": 1025
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 1024,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.18.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2eee08fb901cc2e7b2ba096b79173d4e872278b7169447cb37e24ed66df953f5
+size 2490252108

processor_config.json

@@ -0,0 +1,29 @@
+{
+  "blank_token": "<blank>",
+  "default_num_lookahead_tokens": 3,
+  "feature_extractor": {
+    "feature_extractor_type": "NemotronAsrStreamingFeatureExtractor",
+    "feature_size": 128,
+    "hop_length": 160,
+    "n_fft": 512,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "preemphasis": 0.97,
+    "return_attention_mask": true,
+    "sampling_rate": 16000,
+    "win_length": 400
+  },
+  "num_prompts": 128,
+  "processor_class": "Nemotron3_5AsrProcessor",
+  "prompt_dictionary": {
+    "auto": 38,
+    "fa": 38,
+    "fa-IR": 38
+  },
+  "supported_num_lookahead_tokens": [
+    3,
+    0,
+    6,
+    13
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "processor_class": "Nemotron3_5AsrProcessor",
+  "tokenizer_class": "ParakeetTokenizer",
+  "unk_token": "<unk>"
+}