---
library_name: executorch
license: mit
pipeline_tag: automatic-speech-recognition
base_model: openai/whisper-small
base_model_relation: quantized
tags:
  - automatic-speech-recognition
  - whisper
  - int8
  - quantized
  - xnnpack
  - arm
  - executorch
  - edge-ai
  - librispeech
datasets:
  - librispeech_asr
metrics:
  - wer
  - cer
model-index:
  - name: whisper-small-int8-xnnpack-executorch
    results:
      - task:
          type: automatic-speech-recognition
        dataset:
          type: librispeech_asr
          name: LibriSpeech ASR (test-clean)
          split: test
          args:
            evaluation_samples: 2620
        metrics:
          - type: wer
            value: 3.41
            name: WER (ExecuTorch)
          - type: cer
            value: 1.29
            name: CER (ExecuTorch)
---

# Whisper Small INT8 (ExecuTorch + XNNPACK + KleidiAI)

INT8 8da8w quantized version of [openai/whisper-small](https://huggingface.co/openai/whisper-small),
optimized for ARM deployment using ExecuTorch, XNNPACK, and KleidiAI. The model targets
on-device English speech-to-text transcription on Android and ARM-based edge hardware,
delivering near-identical accuracy at substantially lower latency and memory footprint.

## Key Highlights

Compared to the FP32 baseline on ARM hardware (Vivo X300, Arm C1):

- **2.72x smaller** — .pte artifact shrinks from 1074.76 MB to 395.05 MB
- **1.40x faster inference** — end-to-end latency drops from 10927.0 ms to 7802.5 ms (p50)
- **WER preserved** — 3.45% to 3.41% on LibriSpeech test-clean (2620 utterances)

## Model Details

| Property | Value |
|---|---|
| Developed by | OpenAI |
| Model type | Automatic Speech Recognition (encoder-decoder Transformer) |
| Language | English |
| License | MIT |
| Base model | [openai/whisper-small](https://huggingface.co/openai/whisper-small) |
| Modification | Post-training quantization (8da8w), not finetuned |
| Parameter count | 241.73M |
| PyTorch state dict size | 922.31 MB |
| Optimized .pte size | 395.05 MB |

## How to Get Started

### Install dependencies

```bash
pip install transformers torch soundfile numpy
```

### Run inference

```bash
python example.py
```

### Core inference loop

```python
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

MODEL_NAME = "openai/whisper-small"
LANGUAGE = "en"
TASK = "transcribe"
MAX_NEW_TOKENS = 128

processor = WhisperProcessor.from_pretrained(MODEL_NAME)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_NAME)
model.eval()

def transcribe(audio_path: str) -> str:
    """Load audio and return transcribed text."""
    import soundfile as sf
    import numpy as np

    audio_np, sr = sf.read(audio_path, dtype="float32")
    if audio_np.ndim == 2:
        audio_np = audio_np.mean(axis=1)

    features = processor.feature_extractor(
        audio_np, sampling_rate=sr, return_tensors="pt"
    ).input_features  # [1, 80, 3000]

    with torch.no_grad():
        output_ids = model.generate(
            features,
            language=LANGUAGE,
            task=TASK,
            max_new_tokens=MAX_NEW_TOKENS,
        )
    return processor.tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
```

> **Note on .pte artifacts:** The `pte_optimized/` and `pte_original/` directories contain
> ExecuTorch-serialized models for on-device ARM inference. Running them requires the
> ExecuTorch C++ seq2seq runner — they are not suitable for a simple Python
> `method.execute()` call. The `example.py` script above uses the HuggingFace Transformers
> library for Python-based evaluation and prototyping.

## Evaluation

### Testing data

2620 utterances from the LibriSpeech ASR `test-clean` split. Calibration: none required
(dynamic activation quantization). Dynamic weight-only 8da8w — weights INT8 offline,
activations INT8 quantized per-token at runtime.

### Metrics

| Metric | Description |
|---|---|
| WER | Word Error Rate (lower is better) |
| CER | Character Error Rate (lower is better) |

### Accuracy results

| Model | WER | CER |
|---|---|---|
| openai/whisper-small (FP32) | 3.45% | 1.33% |
| whisper-small INT8 8da8w (ExecuTorch) | **3.41%** | **1.29%** |

### Efficiency results (Vivo X300, Arm C1)

| Metric | FP32 Original | INT8 Optimized | Improvement |
|---|---|---|---|
| .pte file size | 1074.76 MB | 395.05 MB | **2.72x smaller** |
| End-to-end latency p50 | 10927.0 ms | 7802.5 ms | **1.40x faster** |
| End-to-end latency p90 | 11742.0 ms | 8027.0 ms | 1.46x faster |
| RTFx | 0.641 | 0.897 | **+40%** |
| Decode throughput | 3.62 tok/s | 5.69 tok/s | **+57%** |
| Prefill throughput | 620.0 tok/s | 745.6 tok/s | +20% |
| Time to first token (TTFT) | 2683.9 ms | 2196.2 ms | -18% |
| Peak memory (USS) | 5226.97 MB | 4662.29 MB | **1.12x less** |
| Model load time | 1327.3 ms | 1015.0 ms | 1.31x faster |

Benchmark conditions: batch size 1, ~7 s audio clips, 50 runs (10 warmup), offline mode,
16 kHz input on Android 16 / OriginOS 6.

## Technical Specifications

### Objective

English speech-to-text transcription using Whisper's encoder-decoder Transformer
architecture. The encoder processes 80-bin log-mel spectrograms; the decoder
autoregressively generates token IDs which are decoded to text with the Whisper tokenizer.

### Quantization

- **Method:** TorchAO 8da8w — 8-bit dynamic activation quantization + 8-bit weight quantization
- **Calibration:** none required (dynamic activation quantization). Dynamic weight-only 8da8w — weights INT8 offline, activations INT8 quantized per-token at runtime.
- **Weight granularity:** per-channel
- **Symmetry:** symmetric INT8
- **Skipped layers (kept FP32):** `proj_out`/`lm_head`, encoder + decoder positional embeddings, `encoder.conv1` / `encoder.conv2`

### Export pipeline

1. Load pretrained FP32 `openai/whisper-small` from HuggingFace
2. Export encoder and decoder to ExecuTorch Seq2Seq format via Optimum ExecuTorch
3. Apply 8da8w (8-bit dynamic activation, 8-bit weight) quantization with TorchAO
4. Export tokenizer files (`tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`)
5. Export audio preprocessor to `whisper_preprocessor.pte`

### Input preprocessing

| Step | Parameters |
|---|---|
| Load audio as waveform | 16 kHz mono |
| Log-mel spectrogram | n_mels=80, hop_length=160, n_fft=400, sample_rate=16000, duration=30s |
| Pad or trim | 3000 time frames |

**Input tensor:** `[1, 80, 3000]`, dtype `float32` — log-mel spectrogram (batch=1, mel bins, time frames)

### Output postprocessing

1. Autoregressive greedy decoding with the Whisper tokenizer
2. Skip special tokens

## Known Limitations

- Evaluated on English speech (LibriSpeech test-clean); performance on other languages or
  accents has not been measured for this INT8 variant.
- The `.pte` runtime on Python requires the ExecuTorch C++ seq2seq runner — not a simple
  `method.execute()` call. Use `example.py` (HuggingFace Transformers) for Python prototyping.
- Maximum audio duration: 30 seconds per chunk (3000 mel time frames at 16 kHz).
  Longer audio must be chunked externally.
- Android latency measured on Vivo X300 with Arm C1 processor (1x C1-Ultra, 3x C1-Premium,
  4x C1-Pro cores at 4.21 / 3.5 / 2.7 GHz). Results may differ on other ARM devices.
- RTFx > 1.0 indicates real-time capable transcription on this hardware; values below 1.0
  on lower-tier devices are expected.