wav2vec2-xlsr-53-espeak-cv-ft-ONNX

ONNX export of facebook/wav2vec2-xlsr-53-espeak-cv-ft, for running multilingual IPA phoneme recognition in the browser with transformers.js.

The upstream checkpoint ships only pytorch_model.bin — no safetensors, no ONNX — so transformers.js cannot load it as-is. This repo is that missing export.

Which file to use

file	size	notes
onnx/model.onnx	1205 MB	fp32 reference. Reproduces the PyTorch output exactly.
onnx/model_fp16.onnx	603 MB	Best accuracy/size, but fp16 in practice wants WebGPU.
onnx/model_q4.onnx	230 MB	Recommended. Blockwise 4-bit weight-only; runs on the wasm backend.
onnx/model_q4f16.onnx	188 MB	Smallest. fp16 activations, so same WebGPU caveat as fp16.

There are deliberately no 8-bit variants here. For this model they are strictly dominated — see below.

Accuracy

Phoneme error rate (token-level edit distance) against the fp32 PyTorch output of the original checkpoint, on two clips: a 6 s English one and a 60 s Lojban one. Lojban is the interesting column — its phonology is drawn from sounds common across natural languages, so it exercises the rare-phoneme tail of the 392-way output that a multilingual model exists for.

variant	size	English 6 s	Lojban 60 s
fp32	1205 MB	0.0%	0.0%
fp16	603 MB	1.8%	0.4%
q8	303 MB	0.0%	12.5%
int8	303 MB	5.4%	17.9%
uint8	303 MB	0.0%	12.5%
q4	230 MB	5.4%	3.3%
q4f16	188 MB	5.4%	3.7%
bnb4	212 MB	3.6%	5.1%

The 8-bit variants are both bigger and several times worse on non-English audio than the 4-bit ones. "8-bit" here means quantize_dynamic, which quantizes activations as well as weights; q4/q4f16 are blockwise weight-only (MatMulNBits, block size 32). Bits per weight is the wrong axis — what matters is whether activations survive.

Note that transformers.js defaults to q8 on the wasm backend, which for this model is the worst available choice. Set dtype explicitly.

Also note that on the English clip alone, q8 scores a perfect 0.0% and looks like the obvious pick. Only the non-English clip separates the variants.

Two clips is an illustration, not a benchmark — but the gap is large enough to act on.

Usage

This checkpoint declares Wav2Vec2PhonemeCTCTokenizer, which transformers.js does not implement, and ships no tokenizer.json — so pipeline() and AutoTokenizer both throw. Decoding a CTC phoneme model is a plain vocab lookup, so do it by hand over vocab.json:

import { AutoModelForCTC, Wav2Vec2FeatureExtractor } from '@huggingface/transformers';

const id = 'qnighy/wav2vec2-xlsr-53-espeak-cv-ft-ONNX';
const model = await AutoModelForCTC.from_pretrained(id, { dtype: 'q4' });
const extractor = await Wav2Vec2FeatureExtractor.from_pretrained(id);

// Index by token id. `vocab.json` maps the other way.
const vocab = [];
for (const [token, i] of Object.entries(await (await fetch(
  `https://huggingface.co/${id}/resolve/main/vocab.json`)).json())) vocab[i] = token;

/** @param {Float32Array} pcm mono, 16 kHz */
async function transcribe(pcm) {
  const { logits } = await model(await extractor(pcm));
  const [, frames, size] = logits.dims;
  const data = logits.data;

  const out = [];
  let prev = -1;
  for (let t = 0; t < frames; t++) {
    let best = 0;
    for (let v = 1; v < size; v++) {
      if (data[t * size + v] > data[t * size + best]) best = v;
    }
    // Collapse repeats *before* dropping blanks -- the other order merges two
    // genuinely repeated phonemes that the model separated with a blank.
    if (best === prev) continue;
    prev = best;
    if (vocab[best] !== '<pad>') out.push(vocab[best]);
  }
  return out.join(' ');
}

Input is mono 16 kHz float PCM; the feature extractor handles the zero-mean/ unit-variance normalisation. espeak-ng and phonemizer are not needed — they phonemize text at training time, and decoding is pure vocab lookup.

How this was exported

# 1. fp32 export. Note: `optimum[exporters]` no longer exists as of optimum 2.x.
uvx --with "optimum-onnx[onnxruntime]" --from optimum optimum-cli export onnx \
  --model facebook/wav2vec2-xlsr-53-espeak-cv-ft \
  --task automatic-speech-recognition out/

# 2. Constant-fold before quantizing. NOT optional: this checkpoint's positional
#    conv uses weight normalisation, which exports as a runtime Mul, so the Conv
#    weight is not an initializer and the quantizer fails with
#    "Expected .../pos_conv_embed/conv/weight/weight.0/Mul_output_0 to be an
#    initializer". Symbolic shape inference crashes on this graph and is skipped.
python -m onnxruntime.quantization.preprocess \
  --input out/model.onnx --output folded/model.onnx --skip_symbolic_shape True

# 3. Quantize with the transformers.js script (onnxruntime pinned to 1.20.1,
#    which is the last release before matmul_4bits_quantizer was renamed).
python quantize.py --input_folder folded --output_folder onnx

That step 2 is most likely why no ONNX build of this model existed before.

The fp32 export was verified to reproduce the original PyTorch model's output token-for-token on both evaluation clips.