iwillsolvehardestproblem/wav2vec2-xlsr-53-espeak-cv-ft-onnx

wav2vec2-xlsr-53-espeak-cv-ft, ONNX for CPU

CPU-ready ONNX export of facebook/wav2vec2-xlsr-53-espeak-cv-ft, the multilingual phoneme recognizer (eSpeak phoneme vocabulary, CTC head). Same weights, no torch, no GPU: load with onnxruntime (or the Rust ort crate) and decode speech into phonemes at several times realtime on a laptop CPU.

Built for pronunciation scoring on call-center telephony audio, where it replaced a GPU/SageMaker deployment of the same model. Published because the conversion is NOT a one-liner: two silent accuracy traps cost us a day each, and the benchmarks below are the map around them.

Files

file	size	what
model.fp32.onnx	1.26 GB	full precision, matches torch output (default)
model.int8.onnx	355 MB	dynamic int8, MatMul-only (see trap 2)
vocab.json	6 KB	{blank_id, sample_rate, id_to_phoneme} for CTC decode

Quickstart: hear what the model heard, in 10 lines

# pip install onnxruntime huggingface_hub numpy soundfile
import json, numpy as np, onnxruntime as ort, soundfile as sf
from huggingface_hub import hf_hub_download

repo = "iwillsolvehardestproblem/wav2vec2-xlsr-53-espeak-cv-ft-onnx"
model_path = hf_hub_download(repo, "model.fp32.onnx")
vocab = json.load(open(hf_hub_download(repo, "vocab.json")))

sig, sr = sf.read("speech.wav", dtype="float32")   # 16 kHz mono; resample first if not
if sig.ndim > 1: sig = sig.mean(axis=1)
x = ((sig - sig.mean()) / np.sqrt(sig.var() + 1e-7)).astype(np.float32)[None, :]

sess = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
ids = sess.run(None, {"input_values": x})[0][0].argmax(-1)

id2p, blank, prev, phs = vocab["id_to_phoneme"], vocab["blank_id"], -1, []
for i in ids:
    if i != prev and i != blank:
        p = id2p[str(int(i))]
        if p not in {"<unk>", "|", "<s>", "</s>"}: phs.append(p)
    prev = int(i)
print(" ".join(phs))   # e.g. h ə l oʊ w ɜː l d

Fine for clips up to ~30 s. For minutes-long audio use the chunked decode_full below (a single pass goes quadratic in the attention and ends up slower than realtime).

Measured accuracy (real telephony speech, not synthetic)

variant	phoneme drift vs torch fp32	speed (M-series CPU, 4 threads)
fp32 ONNX	byte-identical on 2-6 s windows	~9x realtime
int8 MatMul-only	4.3% mean (max 9.5%)	~9x realtime, NO gain
int8 default op set	~20% of phonemes corrupted	do not use

Against the torch production deployment of the same weights, on scored words from a real call: 75% of words decode identically, and pronunciation-flag decisions agree on 27-28 of 32 words. The residue is argmax flips on acoustically ambiguous frames, the same noise you get between any two runtimes of this model.

Trap 1: never decode short windows in isolation

wav2vec2 is a transformer: per-frame output depends on surrounding audio. Decoding an isolated sub-second word window (as a naive per-word loop does) corrupts ~50% of phonemes. Decode long context ONCE, then slice frames:

import json, numpy as np, onnxruntime as ort

sess  = ort.InferenceSession("model.fp32.onnx", providers=["CPUExecutionProvider"])
vocab = json.load(open("vocab.json"))
id2p, blank = vocab["id_to_phoneme"], vocab["blank_id"]

def decode_full(signal_16k: np.ndarray, core_s=30.0, margin_s=8.0):
    """Frame ids for the whole signal: normalize once, run stride-aligned
    chunks with context margins, keep core frames. Near-linear time (a single
    giant pass is O(n^2) in frames and slower than realtime on minutes of audio)."""
    x = (signal_16k - signal_16k.mean()) / np.sqrt(signal_16k.var() + 1e-7)
    stride, sr = 320, 16000
    core, margin = int(core_s*sr)//stride*stride, int(margin_s*sr)//stride*stride
    total = len(x)//stride
    ids = []
    start = 0
    while start < len(x) and len(ids) < total:
        end  = min(start + core, len(x))
        lead = min(start, margin)
        chunk = x[start-lead : min(end+margin, len(x))].astype(np.float32)[None, :]
        out = sess.run(None, {"input_values": chunk})[0][0].argmax(-1)
        want = min((end-start)//stride, total-len(ids))
        ids.extend(out[lead//stride : lead//stride + want])
        start = end
    return np.array(ids), len(x)/sr/len(ids)   # ids, frame_dur

SKIP = {"<unk>", "|", "<s>", "</s>"}
def decode_window(ids, frame_dur, start_s, end_s, pad_s=0.04):
    f0, f1 = int(max(0, start_s-pad_s)/frame_dur), int((end_s+pad_s)/frame_dur)
    phs, prev = [], -1
    for i in ids[f0:f1]:
        if i != prev and i != blank:
            p = id2p[str(int(i))]
            if p not in SKIP: phs.append(p)
        prev = int(i)
    return phs

Chunk boundaries MUST be multiples of the 320-sample conv stride: a one-frame accounting slip shifts every later window by 20 ms and quietly corrupts everything after it.

Trap 2: default dynamic quantization corrupts the model

quantize_dynamic with the default op set corrupted ~20% of phonemes while a synthetic-signal parity check still passed (it decoded zero phonemes on a test tone, so "no diff" was vacuous). Restricting to op_types_to_quantize=["MatMul"] brings int8 to 4.3% mean drift. On Apple Silicon int8 was NOT faster than fp32 (memory-bandwidth bound), so fp32 stays the default; int8 is for memory-tight hosts. Always run parity on real speech, never on synthetic signals.

Reproduce the export

import torch
from transformers import Wav2Vec2ForCTC
from onnxruntime.quantization import QuantType, quantize_dynamic

model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-xlsr-53-espeak-cv-ft").eval()
torch.onnx.export(model, (torch.randn(1, 32000),), "model.fp32.onnx",
                  input_names=["input_values"], output_names=["logits"],
                  dynamic_axes={"input_values": {0: "batch", 1: "samples"},
                                "logits": {0: "batch", 1: "frames"}},
                  opset_version=17, dynamo=False)
quantize_dynamic("model.fp32.onnx", "model.int8.onnx",
                 weight_type=QuantType.QInt8, op_types_to_quantize=["MatMul"])

The phoneme vocabulary is the source repo's vocab.json (392 eSpeak tokens, <pad> = CTC blank). Input is 16 kHz mono, zero-mean unit-variance normalized over the WHOLE signal (matching Wav2Vec2FeatureExtractor, do_normalize=true).

Credits and license

Weights: facebook/wav2vec2-xlsr-53-espeak-cv-ft (Apache-2.0), unchanged apart from format conversion and quantization. This repo: Apache-2.0.