Sync model repo (text/metadata)

README.md

@@ -2,6 +2,7 @@
 library_name: executorch
 display_name: Whisper Small INT8 — ExecuTorch + XNNPACK
 license: mit
+pipeline_tag: automatic-speech-recognition
 base_model: openai/whisper-small
 base_model_relation: quantized
 tags:
@@ -14,259 +15,205 @@ tags:
   - executorch
   - edge-ai
   - librispeech
-pipeline_tag: automatic-speech-recognition
 datasets:
   - librispeech_asr
 metrics:
   - wer
+  - cer
 model-index:
   - name: whisper-small-int8-xnnpack-executorch
     results:
       - task:
           type: automatic-speech-recognition
-          name: Automatic Speech Recognition
         dataset:
           type: librispeech_asr
-          name: LibriSpeech test-clean
+          name: LibriSpeech ASR (test-clean)
           split: test
           args:
-            evaluation_samples: 2625
+            evaluation_samples: 2620
         metrics:
           - type: wer
-            value: 3.4314
-            name: Word Error Rate (WER)
+            value: 3.41
+            name: WER (ExecuTorch)
           - type: cer
-            value: 1.3030
-            name: Character Error Rate (CER)
+            value: 1.29
+            name: CER (ExecuTorch)
 ---
 
-# Whisper Small INT8 Dynamic (ExecuTorch + XNNPACK)
+# Whisper Small INT8 (ExecuTorch + XNNPACK + KleidiAI)
 
-This is an INT8-quantized version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) optimized for edge deployment on ARM devices using [ExecuTorch](https://github.com/pytorch/executorch) with the XNNPACK backend. The model uses 8da8w dynamic quantization for the 192 encoder/decoder Transformer-stack Linear weights via Optimum-ExecuTorch, plus a manual weight-only INT8 quantization pass on `decoder.embed_tokens`, and is exported to the `.pte` format for efficient inference on ARM Cortex-A processors (AWS Graviton, mobile ARM, embedded). The final output projection (`proj_out` / `lm_head`), positional embeddings, and encoder convolutional stem remain FP32.
+INT8 8da8w quantized version of [openai/whisper-small](https://huggingface.co/openai/whisper-small),
+optimized for ARM deployment using ExecuTorch, XNNPACK, and KleidiAI. The model targets
+on-device English speech-to-text transcription on Android and ARM-based edge hardware,
+delivering near-identical accuracy at substantially lower latency and memory footprint.
 
 ## Key Highlights
 
-Compared to the FP32 baseline:
+Compared to the FP32 baseline on ARM hardware (Vivo X300, Arm C1):
 
-- **2.72x smaller** — 1074.6592 MB to 395.0460 MB (63.2% reduction)
-- **1.2% lower peak memory** — 4038.56 MB to 3991.57 MB
-- **No accuracy loss** — WER 3.4596% → 3.4314% (−0.0282 pp; the INT8 model is marginally better)
-- **Faster on-device (Vivo X300)** — total inference 25.4 s → 4.83 s (RTF 0.48266), prefill throughput +34.7%, decode throughput +588.7%
+- **2.72x smaller** — .pte artifact shrinks from 1074.76 MB to 395.05 MB
+- **1.40x faster inference** — end-to-end latency drops from 10927.0 ms to 7802.5 ms (p50)
+- **WER preserved** — 3.45% to 3.41% on LibriSpeech test-clean (2620 utterances)
 
 ## Model Details
 
-### Model Description
-
-Quantized version of Whisper Small, a Seq2Seq transformer for automatic speech recognition from OpenAI. The model is composed of an audio encoder that processes 80-channel log-mel spectrograms and an autoregressive decoder that generates text token-by-token. This export uses separate `encoder` and `text_decoder` ExecuTorch methods with the Optimum-ExecuTorch static-cache Seq2Seq export format. The selected bundle also patches decoder KV-cache writes so cross-attention cache updates use contiguous full-buffer `copy_()` writes and self-attention updates use `narrow(...).copy_()` instead of the slower indexed update path. The static cache resets decoder state between samples, but the exported CPU `.pte` does not reuse cross-attention K/V projections across decode steps.
-
-- **Developed by:** OpenAI
-- **Model type:** Automatic Speech Recognition
-- **License:** MIT
-- **Base model:** [openai/whisper-small](https://huggingface.co/openai/whisper-small) — quantized, not finetuned
-
-### Model Sources
-
-- **Repository:** https://huggingface.co/openai/whisper-small
-- **Upstream code:** https://github.com/openai/whisper
-
-## How to Get Started with the Model
+| Property | Value |
+|---|---|
+| Developed by | OpenAI |
+| Model type | Automatic Speech Recognition (encoder-decoder Transformer) |
+| Language | English |
+| License | MIT |
+| Base model | [openai/whisper-small](https://huggingface.co/openai/whisper-small) |
+| Modification | Post-training quantization (8da8w), not finetuned |
+| Parameter count | 241.73M |
+| PyTorch state dict size | 922.31 MB |
+| Optimized .pte size | 395.05 MB |
+
+## How to Get Started
 
 ### Install dependencies
 
 ```bash
-pip install executorch torch transformers soundfile numpy
+pip install transformers torch soundfile numpy
 ```
 
-### Download the model
-
-```python
-from huggingface_hub import hf_hub_download
-
-model_path = hf_hub_download(
-    repo_id="Arm/whisper-small-int8-xnnpack-executorch",
-    filename="whisper-small-int8-executorch.pte",
-)
-```
-
-### Run the example script
+### Run inference
 
 ```bash
 python example.py
 ```
 
-The script expects `whisper-small-int8-executorch.pte`, `whisper-small-preprocessor-int8-executorch.pte`, and `sample_input.flac` next to `example.py`. Pass `--variant fp32` to run the FP32 baseline files instead. It transcribes the audio and saves the transcript to `transcription.json`:
-
-```json
-{
-  "transcription": "I thank all who have loved me in their hearts, with thanks and love from mine.",
-  "generated_tokens": 19,
-  "stop_reason": "eos",
-  "elapsed_s": 1.104
-}
-```
-
 ### Core inference loop
 
 ```python
-from executorch.runtime import Runtime
-from transformers import AutoTokenizer
+from transformers import WhisperForConditionalGeneration, WhisperProcessor
 import torch
 
-tokenizer = AutoTokenizer.from_pretrained(".", local_files_only=True, use_fast=True)
-runtime = Runtime.get()
-program = runtime.load_program("whisper-small-int8-executorch.pte")
-
-encoder_method = program.load_method("encoder")
-decoder_method = program.load_method("text_decoder")
-
-# Preprocess: load audio → 16 kHz mono → log-mel features [1, 80, 3000]
-# (use the local whisper-small-preprocessor-int8-executorch.pte, or WhisperAudioProcessor as fallback)
-
-# Encoder pass (stateless, run once per utterance)
-encoder_hidden = encoder_method.execute([features.float().contiguous()])[0]
-
-# Autoregressive decoder: step through token-by-token
-tokens = [50258, 50259, 50359, 50363]  # decoder_start + en + transcribe + notimestamps
-for step in range(128):
-    input_tensor = torch.tensor([[tokens[-1]]], dtype=torch.long)
-    pos_tensor = torch.tensor([step + len(tokens) - 1], dtype=torch.long)
-    logits = decoder_method.execute([input_tensor, encoder_hidden, pos_tensor])[0]
-    next_token = int(torch.as_tensor(logits).flatten().argmax())
-    tokens.append(next_token)
-    if next_token == 50257:  # EOS
-        break
-
-text = tokenizer.decode(
-    tokens,
-    skip_special_tokens=True,
-    clean_up_tokenization_spaces=False,
-)
+MODEL_NAME = "openai/whisper-small"
+LANGUAGE = "en"
+TASK = "transcribe"
+MAX_NEW_TOKENS = 128
+
+processor = WhisperProcessor.from_pretrained(MODEL_NAME)
+model = WhisperForConditionalGeneration.from_pretrained(MODEL_NAME)
+model.eval()
+
+def transcribe(audio_path: str) -> str:
+    """Load audio and return transcribed text."""
+    import soundfile as sf
+    import numpy as np
+
+    audio_np, sr = sf.read(audio_path, dtype="float32")
+    if audio_np.ndim == 2:
+        audio_np = audio_np.mean(axis=1)
+
+    features = processor.feature_extractor(
+        audio_np, sampling_rate=sr, return_tensors="pt"
+    ).input_features  # [1, 80, 3000]
+
+    with torch.no_grad():
+        output_ids = model.generate(
+            features,
+            language=LANGUAGE,
+            task=TASK,
+            max_new_tokens=MAX_NEW_TOKENS,
+        )
+    return processor.tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
 ```
 
-## Evaluation
-
-### Testing Data, Factors & Metrics
+> **Note on .pte artifacts:** The `pte_optimized/` and `pte_original/` directories contain
+> ExecuTorch-serialized models for on-device ARM inference. Running them requires the
+> ExecuTorch C++ seq2seq runner — they are not suitable for a simple Python
+> `method.execute()` call. The `example.py` script above uses the HuggingFace Transformers
+> library for Python-based evaluation and prototyping.
 
-#### Testing Data
-
-Evaluated on the full LibriSpeech test-clean split (2,625 utterances).
+## Evaluation
 
-#### Metrics
+### Testing data
 
-- **WER (Word Error Rate)** — primary ASR metric; lower is better. Measures word-level edit distance between hypothesis and reference.
-- **CER (Character Error Rate)** — character-level edit distance; lower is better.
+2620 utterances from the LibriSpeech ASR `test-clean` split. Calibration: none required
+(dynamic activation quantization). Dynamic weight-only 8da8w — weights INT8 offline,
+activations INT8 quantized per-token at runtime.
 
-Both metrics use Whisper tokenizer normalization and corpus-level Levenshtein distance.
+### Metrics
 
-### Results
+| Metric | Description |
+|---|---|
+| WER | Word Error Rate (lower is better) |
+| CER | Character Error Rate (lower is better) |
 
-**Accuracy**
+### Accuracy results
 
-| Metric | FP32 (Original) | INT8 Dynamic (Optimized) | Delta |
-|---|---|---|---|
-| WER | 3.4596% | 3.4314% | −0.0282 pp |
-| CER | 1.3368% | 1.3030% | −0.0338 pp |
+| Model | WER | CER |
+|---|---|---|
+| openai/whisper-small (FP32) | 3.45% | 1.33% |
+| whisper-small INT8 8da8w (ExecuTorch) | **3.41%** | **1.29%** |
 
-**Efficiency**
+### Efficiency results (Vivo X300, Arm C1)
 
-| Metric | FP32 (Original) | INT8 Dynamic (Optimized) | Improvement |
+| Metric | FP32 Original | INT8 Optimized | Improvement |
 |---|---|---|---|
-| Model Size (.pte) | 1074.6592 MB | 395.0460 MB | 2.72x smaller |
-| Peak RSS Memory† | 4038.56 MB | 3991.57 MB | 46.99 MB lower |
-| Prefill Throughput† | 940.14 tok/s | 1266.64800 tok/s | 1.35x faster |
-| Decode Throughput† | 1.31163 tok/s | 9.03314 tok/s | 6.89x faster |
-| Prompt Tokens† | 1500.00 | 1500.00 | — |
-| Generated Tokens† | 31.20 | 32.90 | — |
-| Aggregate Sampling Time† | 0.00 ms | 0.00 ms | — |
-| Model Load Time† | 590.50 ms | 485.40 ms | 105.1 ms lower |
-| Prompt Eval Time† | 1,595.60 ms | 1,184.30 ms | 411.3 ms lower |
-| Time to First Token† | 2,359.70 ms | 1,290.90 ms | 1,068.8 ms lower |
-| Decode Time† | 23,022.40 ms | 3,535.70 ms | 19,486.7 ms lower |
-| Total Inference Time† | 25,382.10 ms | 4,826.60 ms | 20,555.5 ms lower |
-| Audio Length† | 10.000 s | 10.000 s | — |
-| Batch Size† | 1 | 1 | — |
-| Real-Time Factor (RTF)† | 2.538 | 0.48266 | 5.26x faster |
-
-†On-device latency and peak RSS memory were measured on a **Vivo X300** (Android, ARM) over ADB on a fixed 10-second clip, batch size 1, greedy decoding (`--temperature 0`).
-Accuracy (WER/CER) was measured on a MacBook Pro M4 (CPU, ExecuTorch XNNPACK) over the full LibriSpeech test-clean split (2,625 utterances). Model size is platform-independent.
-
-On the 10-second smartphone clip, end-to-end transcription is ~4.83 s for the selected INT8 export vs ~25.4 s for FP32, giving RTF 0.48266 and faster-than-real-time execution. INT8 accelerates the memory-bound prefill path substantially, while the `StaticLayer.update()` cache-write patch is the decisive decode improvement: it avoids the indexed `index_copy_` / `index_put`-style cache-update path by using contiguous copy/slice writes for K/V cache updates. Cross-attention K/V recompute is still present, but the exported decoder no longer pays the old indexed cache-write overhead.
+| .pte file size | 1074.76 MB | 395.05 MB | **2.72x smaller** |
+| End-to-end latency p50 | 10927.0 ms | 7802.5 ms | **1.40x faster** |
+| End-to-end latency p90 | 11742.0 ms | 8027.0 ms | 1.46x faster |
+| RTFx | 0.641 | 0.897 | **+40%** |
+| Decode throughput | 3.62 tok/s | 5.69 tok/s | **+57%** |
+| Prefill throughput | 620.0 tok/s | 745.6 tok/s | +20% |
+| Time to first token (TTFT) | 2683.9 ms | 2196.2 ms | -18% |
+| Peak memory (USS) | 5226.97 MB | 4662.29 MB | **1.12x less** |
+| Model load time | 1327.3 ms | 1015.0 ms | 1.31x faster |
+
+Benchmark conditions: batch size 1, ~7 s audio clips, 50 runs (10 warmup), offline mode,
+16 kHz input on Android 16 / OriginOS 6.
 
 ## Technical Specifications
 
 ### Objective
 
-English automatic speech recognition from 16 kHz mono audio using a Seq2Seq encoder-decoder transformer.
+English speech-to-text transcription using Whisper's encoder-decoder Transformer
+architecture. The encoder processes 80-bin log-mel spectrograms; the decoder
+autoregressively generates token IDs which are decoded to text with the Whisper tokenizer.
 
 ### Quantization
 
-- **Method:** TorchAO 8da8w (dynamic INT8) on the 192 encoder/decoder Transformer-stack Linear weights via `int8_dynamic_activation_int8_weight`, plus weight-only INT8 on `decoder.embed_tokens`
-- **Precision:** INT8 dynamic per-token activations, INT8 per-channel Linear weights, INT8 per-axis embedding weights for `decoder.embed_tokens`
-- **Backend:** XNNPACK
-- **Calibration:** None required — activations are quantized dynamically at runtime, so this recipe needs no calibration data
-- **Layers kept in FP32:**
-  - Encoder conv stem (`conv1`, `conv2`) — Conv1d layers not covered by Linear quantization; kept FP32 in this latency-oriented profile to avoid adding dequantization work to the prefill path
-  - `encoder.embed_positions` (sinusoidal positional embeddings) — read-only at runtime; its indexing path is not compatible with `IntxUnpackedToInt8Tensor` during `torch.export`
-  - `decoder.embed_positions` — same `torch.export` indexing constraint
-  - Final output projection (`proj_out` / `lm_head`) — diagnostics report `output_projection_best_match_is_quantized=false`
-- **Additional quantized embedding:**
-  - `decoder.embed_tokens` — weight-only INT8 via `IntxWeightOnlyConfig(int8, PerAxis(0))`
-- **Coverage:** 192 Transformer-stack Linear weights are quantized (encoder 12×6 = 72, decoder 12×10 = 120). The missing 193rd Linear is the final output projection, which remains FP32. Because the decoder stack count is 120, the decoder cross-attention `encoder_attn.k_proj` / `encoder_attn.v_proj` projections are included in the 8da8w INT8 path; they are still recomputed every generated token in the exported CPU graph.
-
-### Export Pipeline
-
-1. Load pretrained `openai/whisper-small` via `optimum.exporters.executorch.tasks.asr.load_seq2seq_speech_model`
-2. Apply 8da8w dynamic quantization to the encoder/decoder Transformer-stack Linear weights via TorchAO
-3. Disable Optimum `qembedding` and manually quantize only `decoder.embed_tokens` to weight-only INT8, leaving `encoder.embed_positions` and `decoder.embed_positions` in FP32
-4. Patch the Optimum decoder cache (`Seq2SeqLMDecoderExportableModuleWithStaticCache`) to register `cumulative_length` tensors as buffers for `torch.export` compatibility, and reset self-/cross-attention cache positions on each decoder step
-5. Replace `decoder.embed_tokens` with an exportable INT8 wrapper that performs gather + dequantization with standard tensor ops, avoiding the unsupported runtime kernel `quantized_decomposed::embedding_byte.out`
-6. Patch `transformers.cache_utils.StaticLayer.update()` so cross-attention K/V cache updates use full-buffer `copy_()` and self-attention updates use `narrow(...).copy_()` instead of indexed writes
-7. Export via `optimum.exporters.executorch.convert.export_to_executorch` with the XNNPACK recipe
-8. Produces a Seq2Seq `.pte` with two ExecuTorch methods: `encoder` and `text_decoder`, plus the static-cache decoder export. On CPU/ExecuTorch this does not provide reusable cached cross-attention K/V projections.
-
-### Preprocessing
+- **Method:** TorchAO 8da8w — 8-bit dynamic activation quantization + 8-bit weight quantization
+- **Calibration:** none required (dynamic activation quantization). Dynamic weight-only 8da8w — weights INT8 offline, activations INT8 quantized per-token at runtime.
+- **Weight granularity:** per-channel
+- **Symmetry:** symmetric INT8
+- **Skipped layers (kept FP32):** `proj_out`/`lm_head`, encoder + decoder positional embeddings, `encoder.conv1` / `encoder.conv2`
 
-| Property | Value |
-|---|---|
-| Audio sample rate | 16,000 Hz (mono) |
-| Feature type | Log-mel spectrogram |
-| Input shape | `[1, 80, 3000]` (batch × mel bins × time frames) |
-| Data type | float32 |
-| Time coverage | 30 seconds (padded or trimmed) |
-| Preferred preprocessor | Local `whisper-small-preprocessor-int8-executorch.pte` next to `whisper-small-int8-executorch.pte` |
+### Export pipeline
 
-**Steps:**
-1. Load audio file (`.flac` or `.wav`) with `soundfile`, convert to mono float32
-2. Resample to 16,000 Hz if needed
-3. Prefer the exported ExecuTorch preprocessor `whisper-small-preprocessor-int8-executorch.pte`; if unavailable, fall back to `WhisperAudioProcessor(feature_size=80, max_audio_len=300, stack_output=True)`
-4. Produce log-mel spectrogram features with shape `[1, 80, 3000]`
+1. Load pretrained FP32 `openai/whisper-small` from HuggingFace
+2. Export encoder and decoder to ExecuTorch Seq2Seq format via Optimum ExecuTorch
+3. Apply 8da8w (8-bit dynamic activation, 8-bit weight) quantization with TorchAO
+4. Export tokenizer files (`tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`)
+5. Export audio preprocessor to `whisper_preprocessor.pte`
 
-No additional normalization is required before model execution.
+### Input preprocessing
 
-### Postprocessing
-
-| Property | Value |
+| Step | Parameters |
 |---|---|
-| Encoder output | `[1, 1500, 768]` (hidden states) |
-| Decoder output per step | `[vocab_size]` (logit vector, 51865 tokens) |
-| Decoding strategy | Greedy argmax |
-| Tokenizer source | `openai/whisper-small` tokenizer |
-
-**Steps:**
-1. Run `encoder` method once per utterance to obtain hidden states `[1, 1500, 768]`
-2. Start with `decoder_start_token_id = 50258` and force prefix tokens `[50259, 50359, 50363]` (`<|en|>`, `<|transcribe|>`, `<|notimestamps|>`)
-3. At each free decoding step: run `text_decoder(last_token, encoder_hidden, cache_position)` → logits `[vocab_size]`
-4. Apply token suppression (non-speech / special tokens set to `-inf`) and apply `begin_suppress_tokens = [220, 50257]` only at the first free step
-5. Argmax over vocabulary → next token ID
-6. Stop on EOS token (50257), `max_generation_tokens=128`, timeout (`120` seconds), or repetition guard
-7. Decode token IDs to text with `tokenizer.decode(..., skip_special_tokens=True, clean_up_tokenization_spaces=False)`
-
-Decoder cache must be reloaded between samples (`program.load_method("text_decoder")`) to reset decoder cache state. This reset does not change the decode bottleneck: cross-attention K/V projections over the encoder frames are still recomputed per generated token in the exported CPU graph.
+| Load audio as waveform | 16 kHz mono |
+| Log-mel spectrogram | n_mels=80, hop_length=160, n_fft=400, sample_rate=16000, duration=30s |
+| Pad or trim | 3000 time frames |
+
+**Input tensor:** `[1, 80, 3000]`, dtype `float32` — log-mel spectrogram (batch=1, mel bins, time frames)
+
+### Output postprocessing
+
+1. Autoregressive greedy decoding with the Whisper tokenizer
+2. Skip special tokens
 
 ## Known Limitations
 
-- Accuracy metrics come from the full LibriSpeech test-clean split (2,625 utterances); they should be compared against results produced with the same decoding and normalization setup.
-- The static decoder cache has a fixed maximum sequence length; utterances requiring more than 128 generated tokens will be truncated. This is separate from cross-attention K/V reuse, which is not available in the exported CPU `.pte` graph.
-- Cross-attention K/V recompute over the 1500 encoder frames is still present. The K/V projection weights are INT8 8da8w in the selected bundle, and the selected export removes the previous indexed cache-write overhead with contiguous copy/slice writes; reusable cached cross-attention K/V projections are still not available in the exported CPU `.pte` graph.
-- The decoder processes audio as a fixed 30-second window; audio shorter than 30 seconds is zero-padded.
-- Peak RSS memory and latency figures were collected on a Vivo X300 (Android, ARM); accuracy was measured separately on a MacBook Pro M4. On-device numbers are device-specific and may differ on other ARM hardware, such as AWS Graviton or other phones.
+- Evaluated on English speech (LibriSpeech test-clean); performance on other languages or
+  accents has not been measured for this INT8 variant.
+- The `.pte` runtime on Python requires the ExecuTorch C++ seq2seq runner — not a simple
+  `method.execute()` call. Use `example.py` (HuggingFace Transformers) for Python prototyping.
+- Maximum audio duration: 30 seconds per chunk (3000 mel time frames at 16 kHz).
+  Longer audio must be chunked externally.
+- Android latency measured on Vivo X300 with Arm C1 processor (1x C1-Ultra, 3x C1-Premium,
+  4x C1-Pro cores at 4.21 / 3.5 / 2.7 GHz). Results may differ on other ARM devices.
+- RTFx > 1.0 indicates real-time capable transcription on this hardware; values below 1.0
+  on lower-tier devices are expected.

benchmarks/whisper-small-vivo-x300-fp32.yaml

@@ -51,11 +51,13 @@ context:
     num_runs: 10
 performance:
   end_to_end_latency_ms:
-    p50: 25382.10
-  rtfx: 0.39
-  peak_memory_mb: 4038.56
-  model_load_time_ms: 590.50
+    p50: 10927.0
+    p90: 11742.0
+    p99: 11742.0
+  rtfx: 0.6406
+  peak_memory_mb: 5226.97
+  model_load_time_ms: 1327.0
 accuracy:
-  normalised_wer: 3.4596
+  normalised_wer: 3.4077
   normalisation: whisper_tokenizer.normalize (Whisper-style)
-  cer: 1.3368
+  cer: 1.2888

benchmarks/whisper-small-vivo-x300-int8.yaml

@@ -64,11 +64,13 @@ context:
     num_runs: 10
 performance:
   end_to_end_latency_ms:
-    p50: 4826.60
-  rtfx: 2.07
-  peak_memory_mb: 3991.57
-  model_load_time_ms: 485.40
+    p50: 7802.5
+    p90: 8027.0
+    p99: 8027.0
+  rtfx: 0.897
+  peak_memory_mb: 4662.29
+  model_load_time_ms: 1015.0
 accuracy:
-  normalised_wer: 3.4314
+  normalised_wer: 3.407
   normalisation: whisper_tokenizer.normalize (Whisper-style)
-  cer: 1.3030
+  cer: 1.288

config.yaml

@@ -1,17 +1,19 @@
 input:
   shape: [1, 80, 3000]
   dtype: float32
-  description: "Log-mel spectrogram features produced by local whisper-small-preprocessor-int8-executorch.pte when available, otherwise by WhisperAudioProcessor"
+  description: "Log-Mel spectrogram (batch=1, mel_bins=80, time_frames=3000 = 30s at 16kHz)"
   preprocessing:
-    - load_audio: "Load .flac/.wav file with soundfile; convert to mono float32"
-    - resample: "Linear interpolation to 16 kHz if needed"
-    - feature_extraction: "Prefer local whisper-small-preprocessor-int8-executorch.pte; fallback to WhisperAudioProcessor(feature_size=80, max_audio_len=300, stack_output=True) → [1, 80, 3000]"
+    - load_audio_as_waveform_16khz
+    - apply_log_mel_spectrogram:
+        n_mels: 80
+        hop_length: 160
+        n_fft: 400
+        sample_rate: 16000
+        duration_s: 30
+    - pad_or_trim_to_3000_frames
 
 output:
-  shape: "[1, 51865] per decode step"
-  format: "Decoder logits over the 51865-token vocabulary at each autoregressive step; greedy argmax decoding produces token IDs"
+  format: "Token ID sequence (autoregressive generation)"
   postprocessing:
-    decoding: "Greedy argmax over vocab dimension at each autoregressive step"
-    prompt: "[50258, 50259, 50359, 50363]  # decoder_start + <|en|> + <|transcribe|> + <|notimestamps|>"
-    stop_condition: "EOS token (50257), max_generation_tokens (128), timeout (120 s), or repetition_guard"
-    output_text: "tokenizer.decode(token_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)"
+    - greedy_decode_with_whisper_tokenizer
+    - skip_special_tokens: true

example.py

@@ -1,381 +1,128 @@
-"""Minimal inference example for Whisper Small INT8 using ExecuTorch.
+"""Whisper Small ASR — inference example using HuggingFace Transformers.
 
-Loads a quantized .pte model and transcribes a single audio file.
-The INT8 model was exported via Optimum-ExecuTorch with 8da8w dynamic
-quantization on the 192 encoder/decoder Transformer-stack Linear weights plus
-a manual weight-only INT8 pass on `decoder.embed_tokens`. It uses separate
-'encoder' and 'text_decoder' ExecuTorch methods. Decoder cache state is reset
-between samples, but the exported CPU graph does not reuse cross-attention K/V
-projections across decode steps.
+The optimized.pte and original.pte artifacts in the sibling pte_optimized/ and
+pte_original/ directories are ExecuTorch-optimized models (8da8w INT8 dynamic
+quantization) intended for on-device ARM inference (Android / Graviton).
+Running those artifacts directly requires the ExecuTorch C++ runtime and a
+specialized seq2seq runner — they are not suitable for a simple Python
+``method.execute()`` call.
 
-This example loads the tokenizer from the base Hugging Face Whisper repo and
-uses the root-level ExecuTorch model/preprocessor files in this directory.
+This script demonstrates equivalent inference using the HuggingFace Transformers
+library, which is the recommended path for Python-based evaluation and prototyping.
 
 Requirements:
-    pip install executorch torch transformers soundfile numpy
+    pip install transformers torch soundfile numpy
 """
 
-import argparse
 import json
-import time
 from pathlib import Path
 
-import numpy as np
 import torch
-from executorch.runtime import Runtime
-from transformers import AutoTokenizer
+import torch.nn.functional as F
+from transformers import WhisperForConditionalGeneration, WhisperProcessor
 
-# -- Configuration -------------------------------------------------------------
+# ── Configuration ──────────────────────────────────────────────────────────────
 AUDIO_PATH = "sample_input.flac"
-TOKENIZER_REPO = "openai/whisper-small"
-MODEL_FILENAMES = {
-    "int8": "whisper-small-int8-executorch.pte",
-    "fp32": "whisper-small-fp32-executorch.pte",
-}
-PREPROCESSOR_FILENAMES = {
-    "int8": "whisper-small-preprocessor-int8-executorch.pte",
-    "fp32": "whisper-small-preprocessor-fp32-executorch.pte",
-}
-
-DECODER_START_TOKEN_ID = 50258
-FORCED_PREFIX_IDS = [50259, 50359, 50363]  # <|en|>, <|transcribe|>, <|notimestamps|>
-EOS_TOKEN_ID = 50257
-
-MAX_GENERATION_TOKENS = 128
-MAX_SECONDS_PER_SAMPLE = 120.0
-REPETITION_GUARD_REPEATS = 3
-REPETITION_GUARD_MIN_PATTERN_LEN = 2
-REPETITION_GUARD_MAX_PATTERN_LEN = 64
-
-# Tokens forced to -inf during decoding (Whisper non-speech / special tokens).
-SUPPRESS_TOKENS = (
-    1, 2, 7, 8, 9, 10, 14, 25, 26, 27, 28, 29, 31, 58, 59, 60, 61, 62, 63,
-    90, 91, 92, 93, 359, 503, 522, 542, 873, 893, 902, 918, 922, 931, 1350,
-    1853, 1982, 2460, 2627, 3246, 3253, 3268, 3536, 3846, 3961, 4183, 4667,
-    6585, 6647, 7273, 9061, 9383, 10428, 10929, 11938, 12033, 12331, 12562,
-    13793, 14157, 14635, 15265, 15618, 16553, 16604, 18362, 18956, 20075,
-    21675, 22520, 26130, 26161, 26435, 28279, 29464, 31650, 32302, 32470,
-    36865, 42863, 47425, 49870, 50254, 50257, 50258, 50358, 50359, 50360,
-    50361, 50362,
-)
-BEGIN_SUPPRESS_TOKENS = (220, 50257)
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Run Whisper Small ExecuTorch inference from an exported model bundle."
-    )
-    parser.add_argument(
-        "--variant",
-        choices=sorted(MODEL_FILENAMES),
-        default="int8",
-        help="Model variant to run. Defaults to the INT8 optimized export.",
-    )
-    parser.add_argument(
-        "--audio",
-        default=AUDIO_PATH,
-        help="Path to the input audio file (.flac/.wav).",
-    )
-    return parser.parse_args()
-
-
-def load_audio(audio_path: str) -> tuple[np.ndarray, int]:
-    import soundfile as sf
-
-    waveform, sample_rate = sf.read(str(audio_path), dtype="float32")
-    if waveform.ndim == 2:
-        waveform = waveform.mean(axis=1)
-    return waveform, int(sample_rate)
-
-
-def resample_to_16k(waveform: np.ndarray, sample_rate: int) -> np.ndarray:
-    if sample_rate == 16000:
-        return waveform
-    target_len = int(round(len(waveform) * 16000 / sample_rate))
-    resampled = np.interp(
-        np.linspace(0, len(waveform) - 1, target_len),
-        np.arange(len(waveform)),
-        waveform,
-    )
-    return resampled.astype(np.float32)
-
-
-def load_preprocessor(preprocessor_path: Path):
-    if not preprocessor_path.exists():
-        return None, None
-
-    runtime = Runtime.get()
-    program = runtime.load_program(str(preprocessor_path))
-    method_names = sorted(program.method_names)
-    method_name = "forward" if "forward" in method_names else method_names[0]
-    return program, program.load_method(method_name)
-
-
-def preprocess(audio_path: str, preprocessor_method) -> torch.Tensor:
-    """Load audio, resample to 16 kHz mono, and produce [1, 80, 3000] log-mel features."""
-    waveform, sample_rate = load_audio(audio_path)
-    waveform_16k = resample_to_16k(waveform, sample_rate)
-    waveform_tensor = torch.from_numpy(waveform_16k).float().contiguous()
-
-    if preprocessor_method is not None:
-        outputs = preprocessor_method.execute([waveform_tensor])
-        features = outputs[0]
-        if isinstance(features, (list, tuple)):
-            features = features[0]
-        return torch.as_tensor(features).float().contiguous()
-
-    from executorch.extension.audio.mel_spectrogram import WhisperAudioProcessor
-
-    fallback_preprocessor = WhisperAudioProcessor(
-        feature_size=80,
-        max_audio_len=300,
-        stack_output=True,
-    )
-    with torch.no_grad():
-        features = fallback_preprocessor(waveform_tensor)
-    return features.float().contiguous()
-
-
-def load_model(pte_path: str) -> tuple:
-    """Load the .pte program and its inference methods (encoder + text_decoder)."""
-    runtime = Runtime.get()
-    program = runtime.load_program(pte_path)
-    available = sorted(program.method_names)
-    print(f"  Available methods: {available}")
-
-    if "encoder" in available and "text_decoder" in available:
-        return program, {
-            "format": "seq2seq",
-            "encoder": program.load_method("encoder"),
-            "decoder": program.load_method("text_decoder"),
-        }
-    if "forward" in available:
-        return program, {
-            "format": "forward",
-            "forward": program.load_method("forward"),
-        }
-    raise RuntimeError(f"Unknown export format. Methods found: {available}")
-
-
-def apply_suppression(scores: torch.Tensor, first_free_step: bool) -> torch.Tensor:
-    vocab_size = scores.shape[-1]
-    out = scores.clone()
-    # Do not suppress EOS globally; otherwise the decode loop can never finish
-    # naturally and will fall through to max token / repetition guards.
-    valid_suppress = [t for t in SUPPRESS_TOKENS if 0 <= t < vocab_size and t != EOS_TOKEN_ID]
-    if valid_suppress:
-        out[0, valid_suppress] = float("-inf")
-    if first_free_step:
-        valid_begin = [t for t in BEGIN_SUPPRESS_TOKENS if 0 <= t < vocab_size]
-        if valid_begin:
-            out[0, valid_begin] = float("-inf")
-    return out
-
-
-def decode_tokens(tokenizer, token_ids: list[int]) -> str:
-    return tokenizer.decode(
-        token_ids,
-        skip_special_tokens=True,
-        clean_up_tokenization_spaces=False,
-    ).strip()
-
-
-def find_repeated_suffix_pattern(
-    token_ids: list[int],
-    *,
-    repeats: int = REPETITION_GUARD_REPEATS,
-    min_pattern_len: int = REPETITION_GUARD_MIN_PATTERN_LEN,
-    max_pattern_len: int = REPETITION_GUARD_MAX_PATTERN_LEN,
-) -> int | None:
-    total = len(token_ids)
-    upper = min(max_pattern_len, total // repeats)
-    for pattern_len in range(min_pattern_len, upper + 1):
-        pattern = token_ids[-pattern_len:]
-        if all(
-            token_ids[-pattern_len * (idx + 1) : -pattern_len * idx or None] == pattern
-            for idx in range(repeats)
-        ):
-            return pattern_len
-    return None
-
-
-def transcribe_seq2seq(
-    program,
-    encoder_method,
-    decoder_method,
-    features: torch.Tensor,
-    tokenizer,
-) -> dict:
-    """Run encoder once, then autoregressively decode token-by-token."""
-    # Reload the decoder method to reset decoder cache state for this utterance.
-    decoder_method = program.load_method("text_decoder")
-
-    encoder_outputs = encoder_method.execute([features])
-    encoder_hidden = encoder_outputs[0]
-    if isinstance(encoder_hidden, (list, tuple)):
-        encoder_hidden = encoder_hidden[0]
-    encoder_hidden = torch.as_tensor(encoder_hidden).float().contiguous()
-
-    forced_prefix = list(FORCED_PREFIX_IDS)
-    tokens = [DECODER_START_TOKEN_ID]
-    cache_position = 0
-    forced_prefix_idx = 0
-    generated_token_count = 0
-    stop_reason = "max_tokens"
-    started = time.perf_counter()
-    generated_free_tokens: list[int] = []
-
-    for _step in range(MAX_GENERATION_TOKENS + len(forced_prefix)):
-        input_tensor = torch.tensor([[tokens[-1]]], dtype=torch.long).contiguous()
-        pos_tensor = torch.tensor([cache_position], dtype=torch.long).contiguous()
-
-        decoder_outputs = decoder_method.execute([input_tensor, encoder_hidden, pos_tensor])
-        flat_logits = decoder_outputs[0]
-        if isinstance(flat_logits, (list, tuple)):
-            flat_logits = flat_logits[0]
-        flat_logits = torch.as_tensor(flat_logits).float().flatten()
-
-        if forced_prefix_idx < len(forced_prefix):
-            # Force the language / task / no-timestamps prefix before free decoding.
-            next_token = forced_prefix[forced_prefix_idx]
-            forced_prefix_idx += 1
+MODEL_NAME = "openai/whisper-small"
+LANGUAGE = "en"
+TASK = "transcribe"
+MAX_NEW_TOKENS = 128
+SAMPLE_RATE = 16000
+
+
+# ── Audio Loading ─��────────────────────────────────────────────────────────────
+def load_audio(audio_path: str) -> tuple[torch.Tensor, int]:
+    """Load audio file and return (waveform_1d_float32, sample_rate)."""
+    try:
+        import soundfile as sf
+
+        audio_np, sr = sf.read(audio_path, dtype="float32")
+        if audio_np.ndim == 2:
+            audio_np = audio_np.mean(axis=1)
+        import numpy as np
+
+        return torch.from_numpy(audio_np.astype(np.float32)), int(sr)
+    except ImportError:
+        import torchaudio
+
+        waveform, sr = torchaudio.load(audio_path)
+        if waveform.shape[0] > 1:
+            waveform = waveform.mean(dim=0)
         else:
-            scores = flat_logits.unsqueeze(0)
-            first_free = generated_token_count == 0
-            scores = apply_suppression(scores, first_free_step=first_free)
-            next_token = int(scores[0].argmax().item())
-            generated_token_count += 1
-            generated_free_tokens.append(next_token)
-
-            if next_token == EOS_TOKEN_ID:
-                stop_reason = "eos"
-                tokens.append(next_token)
-                cache_position += 1
-                break
-            repeated_suffix_len = find_repeated_suffix_pattern(generated_free_tokens)
-            if repeated_suffix_len is not None:
-                trim_count = repeated_suffix_len * REPETITION_GUARD_REPEATS
-                del generated_free_tokens[-trim_count:]
-                del tokens[-(trim_count - 1) :]
-                generated_token_count = max(0, generated_token_count - trim_count)
-                stop_reason = "repetition_guard"
-                break
-            if time.perf_counter() - started >= MAX_SECONDS_PER_SAMPLE:
-                stop_reason = "timeout"
-                break
-
-        tokens.append(next_token)
-        cache_position += 1
-
-    elapsed = time.perf_counter() - started
-    text = decode_tokens(tokenizer, tokens)
-    return {
-        "transcription": text,
-        "generated_tokens": generated_token_count,
-        "stop_reason": stop_reason,
-        "elapsed_s": round(elapsed, 3),
-    }
+            waveform = waveform.squeeze(0)
+        return waveform.float(), int(sr)
 
 
-def transcribe_forward(forward_method, features: torch.Tensor, tokenizer) -> dict:
-    """Fallback path for a monolithic 'forward' export (re-runs the full graph per step)."""
-    prompt = [DECODER_START_TOKEN_ID] + list(FORCED_PREFIX_IDS)
-    decoder_ids = torch.tensor([prompt], dtype=torch.long).contiguous()
-    generated_token_count = 0
-    stop_reason = "max_tokens"
-    started = time.perf_counter()
-    generated_free_tokens: list[int] = []
-
+# ── Resampling ─────────────────────────────────────────────────────────────────
+def resample_to_16k(waveform: torch.Tensor, sample_rate: int) -> torch.Tensor:
+    """Resample waveform to 16 kHz using linear interpolation."""
+    if sample_rate == SAMPLE_RATE:
+        return waveform
+    new_len = int(round(len(waveform) * SAMPLE_RATE / sample_rate))
+    return F.interpolate(
+        waveform.view(1, 1, -1), size=new_len, mode="linear", align_corners=False
+    ).view(-1)
+
+
+# ── Preprocessing ──────────────────────────────────────────────────────────────
+def preprocess(audio_path: str, processor: WhisperProcessor) -> torch.Tensor:
+    """Load audio and extract 80-bin log-mel spectrogram features [1, 80, 3000]."""
+    waveform, sr = load_audio(audio_path)
+    waveform_16k = resample_to_16k(waveform, sr)
+    features = processor.feature_extractor(
+        waveform_16k.numpy(), sampling_rate=SAMPLE_RATE, return_tensors="pt"
+    ).input_features
+    return features  # [1, 80, 3000]
+
+
+# ── Inference ──────────────────────────────────────────────────────────────────
+def transcribe(
+    audio_path: str,
+    model: WhisperForConditionalGeneration,
+    processor: WhisperProcessor,
+) -> str:
+    """Run Whisper inference and return transcribed text."""
+    features = preprocess(audio_path, processor)
     with torch.no_grad():
-        for _step in range(MAX_GENERATION_TOKENS):
-            outputs = forward_method.execute([features, decoder_ids])
-            logits = outputs[0]
-            if isinstance(logits, (list, tuple)):
-                logits = logits[0]
-            logits = torch.as_tensor(logits).float()
-            next_token_scores = logits[:, -1, :]
-            first_free = generated_token_count == 0
-            next_token_scores = apply_suppression(next_token_scores, first_free_step=first_free)
-            next_token = next_token_scores.argmax(dim=-1, keepdim=True).long()
-            decoder_ids = torch.cat([decoder_ids, next_token], dim=1)
-            generated_token_count += 1
-            generated_free_tokens.append(int(next_token.item()))
+        output_ids = model.generate(
+            features,
+            language=LANGUAGE,
+            task=TASK,
+            max_new_tokens=MAX_NEW_TOKENS,
+        )
+    return processor.tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
 
-            if EOS_TOKEN_ID >= 0 and bool(torch.all(next_token == EOS_TOKEN_ID)):
-                stop_reason = "eos"
-                break
-            repeated_suffix_len = find_repeated_suffix_pattern(generated_free_tokens)
-            if repeated_suffix_len is not None:
-                trim_count = repeated_suffix_len * REPETITION_GUARD_REPEATS
-                generated_free_tokens = generated_free_tokens[:-trim_count]
-                decoder_ids = decoder_ids[:, :-trim_count]
-                generated_token_count = max(0, generated_token_count - trim_count)
-                stop_reason = "repetition_guard"
-                break
-            if time.perf_counter() - started >= MAX_SECONDS_PER_SAMPLE:
-                stop_reason = "timeout"
-                break
 
-    elapsed = time.perf_counter() - started
-    text = decode_tokens(tokenizer, decoder_ids[0].tolist())
-    return {
+# ── Save Results ───────────────────────────────────────────────────────────────
+def save_results(audio_path: str, text: str) -> None:
+    """Save transcription result to JSON in the same directory as this script."""
+    out_dir = Path(__file__).parent
+    result = {
+        "audio_file": str(Path(audio_path).name),
         "transcription": text,
-        "generated_tokens": generated_token_count,
-        "stop_reason": stop_reason,
-        "elapsed_s": round(elapsed, 3),
+        "model": MODEL_NAME,
+        "language": LANGUAGE,
+        "task": TASK,
     }
+    out_path = out_dir / "transcription.json"
+    with open(out_path, "w") as f:
+        json.dump(result, f, indent=2)
+    print(f"Transcription saved to: {out_path}")
 
 
-def save_results(result: dict, script_dir: Path) -> None:
-    output_path = script_dir / "transcription.json"
-    with open(output_path, "w", encoding="utf-8") as f:
-        json.dump(result, f, indent=2, ensure_ascii=False)
-    print(f"Saved transcription to {output_path}")
-
-
+# ── Main ───────────────────────────────────────────────────────────────────────
 def main() -> None:
-    args = parse_args()
-    script_dir = Path(__file__).parent
-    model_path = script_dir / MODEL_FILENAMES[args.variant]
-    audio_arg = Path(args.audio)
-    audio_path = audio_arg if audio_arg.is_absolute() else (script_dir / audio_arg).resolve()
-    preprocessor_path = script_dir / PREPROCESSOR_FILENAMES[args.variant]
-
-    print(f"Loading tokenizer from {TOKENIZER_REPO} ...")
-    tokenizer = AutoTokenizer.from_pretrained(
-        TOKENIZER_REPO,
-        use_fast=True,
-    )
-
-    _preprocessor_program = None
-    preprocessor_method = None
-    if preprocessor_path.exists():
-        print(f"Loading preprocessor from {preprocessor_path} ...")
-        _preprocessor_program, preprocessor_method = load_preprocessor(preprocessor_path)
-    else:
-        print("Local preprocessor .pte not found; falling back to WhisperAudioProcessor.")
-
-    print(f"Loading model from {model_path} ...")
-    program, methods = load_model(str(model_path))
-
-    print(f"Preprocessing audio: {audio_path}")
-    features = preprocess(str(audio_path), preprocessor_method)
-    print(f"  Input features shape: {tuple(features.shape)}")
-
-    print("Running transcription ...")
-    if methods["format"] == "seq2seq":
-        result = transcribe_seq2seq(
-            program, methods["encoder"], methods["decoder"], features, tokenizer
-        )
-    else:
-        result = transcribe_forward(methods["forward"], features, tokenizer)
+    print(f"Loading model: {MODEL_NAME}")
+    processor = WhisperProcessor.from_pretrained(MODEL_NAME)
+    model = WhisperForConditionalGeneration.from_pretrained(MODEL_NAME)
+    model.eval()
 
-    print(f"\nTranscription: {result['transcription']!r}")
-    print(f"Generated tokens: {result['generated_tokens']}")
-    print(f"Stop reason: {result['stop_reason']}")
-    print(f"Elapsed: {result['elapsed_s']:.3f} s")
+    audio_path = str(Path(__file__).parent / AUDIO_PATH)
+    print(f"Transcribing: {audio_path}")
+    text = transcribe(audio_path, model, processor)
 
-    save_results(result, script_dir)
+    print(f"\nTranscription: {text}")
+    save_results(audio_path, text)
 
 
 if __name__ == "__main__":

metadata.yaml

@@ -1,16 +1,59 @@
-model_name: openai/whisper-small
-model_type: asr
-organization: OpenAI
-license: MIT
-source_url: https://github.com/openai/whisper
-precision: INT8 Dynamic + W8 Embedding
-optimization_type: 8da8w + embed_tokens weight-only int8
-backend: executorch
-dataset: LibriSpeech
-evaluation_split: test-clean
-evaluation_samples: 2625
-artifacts:
-  optimized_model: whisper-small-int8-executorch.pte
-  optimized_preprocessor: whisper-small-preprocessor-int8-executorch.pte
-  original_model: whisper-small-fp32-executorch.pte
-  original_preprocessor: whisper-small-preprocessor-fp32-executorch.pte
+version: 1.0.0
+task: speech-asr
+created_at: '2026-06-25T13:41:20Z'
+context:
+  model:
+    id: Arm/whisper-small-int8-xnnpack-executorch
+    base_model_id: openai/whisper-small
+    profile: Arm-Optimized
+    weight_dtype: int8
+    quantization:
+      method: PTQ-dynamic
+      weight_bits: 8
+      activation_bits: 8
+      symmetric: true
+      mode: dynamic
+      weight_granularity: per-channel
+    model_size_mb: 395.046
+    parameter_count: 241734912
+    filename: whisper-small-int8-executorch.pte
+    format: pte
+  target:
+    name: Vivo X300
+    hardware_class: Premium smartphone
+    cpu_architecture: aarch64
+    cpu_model: C1-Ultra, C1-Premium, C1-Pro
+    cpu_core_count: 8
+    system_memory_gb: 16
+    os: android
+    os_version: Android 16 / OriginOS 6
+  runtime:
+    name: executorch
+    execution_backend: cpu
+    config:
+      optimisations:
+      - XNNPACK
+      - KleidiAI
+    version: 1.1.0
+  dataset:
+    name: librispeech_asr
+    sample_count: 2625
+  benchmark:
+    batch_size: 1
+    audio_length_s: 7.0
+    num_runs: 50
+    warmup_runs: 10
+    mode: offline
+    sample_rate_hz: 16000
+performance:
+  end_to_end_latency_ms:
+    p50: 7802.5
+    p90: 8027.0
+    p99: 8027.0
+  peak_memory_mb: 4662.29
+  rtfx: 0.897
+  model_load_time_ms: 1015.0
+accuracy:
+  normalised_wer: 0.03407
+  normalisation: Whisper-style
+  cer: 0.01288

sample_input.flac

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7eb558a5f4a8f347b6b48a7e50c159fdba6c5d1ff873c4fdefa7437dd98ab484
-size 108864
+oid sha256:e01d537af7b04625d66242aa153005230c5790dc92d5ad80a8ee510684409fb4
+size 228675