R321-hf/whisper-small-hybrid-int8

Whisper Small Hybrid INT8 Quantized ONNX Model

This is a quantized version of openai/whisper-small optimized for mobile deployment with ONNX Runtime.

Quantization Details

• Method: Hybrid INT8 Dynamic Quantization
• MatMul/Gemm layers: INT8 quantized (for size reduction)
• Conv layers: FP32 (excluded to avoid ConvInteger operations)
• Framework: ONNX Runtime

This hybrid approach ensures compatibility with ONNX Runtime Mobile, which does not support ConvInteger operations.

Model Specifications

Parameter	Value
Hidden dimension	768
Attention heads	12
Encoder layers	12
Vocabulary size	51865
Max source positions	1500
Sample rate	16000 Hz

Files

File	Description	Size
encoder_model.onnx	Quantized encoder	~100 MB
decoder_model.onnx	Quantized decoder	~100 MB
tokenizer.json	Tokenizer configuration	~2 MB
vocab.json	Vocabulary file	~1 MB
merges.txt	BPE merges file	~1 MB

Usage

Python (ONNX Runtime)

import onnxruntime as ort
import numpy as np
from transformers import WhisperProcessor

# Load processor
processor = WhisperProcessor.from_pretrained("R321-hf/whisper-small-hybrid-int8")

# Load models
encoder_session = ort.InferenceSession("encoder_model.onnx")
decoder_session = ort.InferenceSession("decoder_model.onnx")

# Prepare input (mel spectrogram)
mel_spectrogram = np.random.randn(1, 80, 3000).astype(np.float32)

# Run encoder
encoder_output = encoder_session.run(None, {"input_features": mel_spectrogram})

# Run decoder with initial tokens
decoder_input_ids = np.array([[50258]])  # BOS token
logits = decoder_session.run(None, {
    "decoder_input_ids": decoder_input_ids,
    "encoder_hidden_states": encoder_output[0]
})

Android (ONNX Runtime Mobile)

// Load encoder
val encoderSession = ortEnvironment.createSession("encoder_model.onnx", sessionOptions)

// Load decoder
val decoderSession = ortEnvironment.createSession("decoder_model.onnx", sessionOptions)

// Run inference
val melTensor = OnnxTensor.createTensor(ortEnvironment, melSpectrogram, longArrayOf(1, 80, 3000))
val encoderOutput = encoderSession.run(mapOf("input_features" to melTensor))

Compatibility

• ONNX Runtime (Python)
• ONNX Runtime Mobile (Android/iOS)
• No ConvInteger operations
• NNAPI compatible

License

MIT License (same as original Whisper model)

Credits

• Original model: openai/whisper-small
• ONNX export: onnx-community/whisper-small
• Quantization: Hybrid INT8 with ONNX Runtime