PORTING_GUIDE.md

Porting Cohere Transcribe to Run Locally: A Complete Guide

What We Did (And Why It Matters)

We took a brand-new AI speech recognition model — Cohere Transcribe, released March 26, 2026 — and made it run completely offline on a regular laptop. No internet connection, no cloud API, no paid subscription.

The result: a 57-minute English interview transcribed in under 15 minutes, and a 35-minute German podcast in under 10 minutes. On CPU alone, no fancy GPU required.

This guide explains every step, every concept, and every decision along the way.

---

Background: What Is a Speech Recognition Model?

A speech recognition model (also called ASR — Automatic Speech Recognition) takes audio as input and produces text as output. Think of it like a very sophisticated dictation machine.

Modern ASR models are neural networks — software that learns patterns from millions of hours of audio paired with their correct transcriptions. Once trained, the model can listen to new audio it's never heard before and produce accurate text.

Why "Port" a Model?

The Cohere Transcribe model was released as a Python-only package that normally runs through HuggingFace's transformers library. That's fine for Python developers, but it means:

• You need Python installed
• You need ~15 GB of Python dependencies
• It's hard to use from other programming languages (C#, C++, mobile apps)
• It's slower than it could be

By "porting" the model to ONNX format (explained below), we make it usable from virtually any programming language, on any platform, with better performance.

---

Key Concepts Explained

Before diving into the process, here are the technical concepts you'll encounter. Each one is explained in plain language.

ONNX (Open Neural Network Exchange)

Think of ONNX as a universal file format for AI models — like how PDF is a universal format for documents. A model saved as ONNX can be loaded and run by any ONNX-compatible runtime, regardless of which framework (PyTorch, TensorFlow, etc.) originally created it.

ONNX Runtime is the engine that actually runs ONNX models. Microsoft maintains it, and it has bindings for C#, Python, C++, Java, JavaScript, and more.

Encoder-Decoder Architecture

Cohere Transcribe uses an encoder-decoder design, which is like a two-stage translation pipeline:

1. Encoder: Listens to the audio and builds an internal understanding of what was said. Think of this as a person listening very carefully and taking detailed notes — but in a form only they can read.

2. Decoder: Reads those internal notes and writes out the actual text, one word (actually one token) at a time. It works left-to-right, like a person typing out what was said.

The encoder runs once per audio clip. The decoder runs once per output token (word piece). For a 30-second clip that produces 50 words, the encoder runs 1 time and the decoder runs ~60 times.

Conformer (The Encoder Type)

The encoder in this model is a Conformer — a hybrid architecture that combines two different approaches:

• Convolutional layers: Good at detecting local patterns (specific sounds, phonemes, sharp transitions in audio). Think of these as recognizing individual letters.
• Transformer/Attention layers: Good at understanding global context (what the whole sentence means). Think of these as understanding how letters form words and sentences.

By alternating between convolution and attention, the Conformer captures both fine-grained sound details and big-picture meaning. This model has 48 of these combined layers stacked on top of each other.

Mel Spectrogram (How Audio Becomes Numbers)

Computers can't directly understand sound waves. The raw audio (a sequence of amplitude values, like the wiggly line you see in an audio editor) needs to be converted into a form the neural network can process.

A mel spectrogram is a visual representation of sound that shows:

• Time on the horizontal axis (left to right)
• Frequency on the vertical axis (low sounds at bottom, high sounds at top)
• Intensity as brightness (louder = brighter)

The "mel" part means the frequencies are spaced according to how human hearing works — we're more sensitive to differences between low frequencies than high ones. The mel scale matches this.

The conversion process: raw audio -> apply math (STFT/Fourier transform) -> group frequencies into mel bins -> take the logarithm. The result is a grid of numbers that the encoder can process.

STFT (Short-Time Fourier Transform)

The STFT is the mathematical tool that converts audio from the time domain (a wiggly line showing amplitude over time) to the frequency domain (showing which frequencies are present at each moment).

It works by sliding a window across the audio and, at each position, computing which frequencies are present in that small window. It's like analyzing a song not all at once, but moment-by-moment — "right now there's a bass note and a vocal, now it's just drums..."

Key parameters:

• n_fft (512): How many frequency bins to compute. More bins = finer frequency resolution.
• hop_length (160): How far the window slides between each analysis. At 16,000 samples per second with hop=160, you get 100 analysis frames per second of audio.
• window (hann, size 400): A smooth curve applied to each analysis window to prevent audio artifacts at the edges.

Tokens and Tokenization

Neural networks don't work with words directly. Instead, text is broken into tokens — small pieces that might be whole words, parts of words, or individual characters.

This model uses SentencePiece BPE (Byte Pair Encoding) with 16,384 tokens. Some examples:

• Common words are single tokens: "the" = one token
• Less common words get split: "transcription" might become "trans" + "cription"
• The special character _ (Unicode 2581) marks the start of a new word

The model also has special tokens that control its behavior:

• <|startoftranscript|> — signals "start producing text now"
• <|en|> — tells it to transcribe in English
• <|pnc|> — tells it to include punctuation
• <|endoftext|> — the model outputs this when it's done

Attention and Cross-Attention

Attention is the mechanism that lets the model focus on relevant parts of its input when producing each output.

Self-attention (used within both encoder and decoder): Each element in a sequence looks at every other element in the same sequence to understand context. In the encoder, each audio frame "pays attention to" every other audio frame — so a word can be understood in context.

Cross-attention (connecting encoder to decoder): When the decoder generates each word, it looks back at the encoder's representation of the audio. It's like asking "given what I've written so far, what part of the audio should I focus on next?"

Cross-attention works through three projections:

• Query (Q): "What am I looking for?" — comes from the decoder
• Key (K): "What do I contain?" — comes from the encoder
• Value (V): "What information do I provide?" — comes from the encoder

The decoder computes a similarity between Q and K to figure out where to focus, then retrieves the corresponding V. This happens at every decoder layer (8 layers in this model).

KV Cache

When the decoder generates text one token at a time, it would be wasteful to reprocess all previous tokens at every step. Instead, we save the Key and Value computations from previous steps in a KV cache.

Think of it like a growing notebook. At step 1, you write one line. At step 2, you don't rewrite the whole notebook — you just add a new line and re-read what you need. The KV cache is that notebook.

For the self-attention KV cache: it grows with each decoder step (we add new entries as we generate new tokens).

For the cross-attention K/V: these are computed once from the encoder output and then reused at every decoder step (the audio doesn't change — we're just looking at it from different angles).

Quantization

Neural network weights are normally stored as 32-bit floating point numbers (FP32). Each weight takes 4 bytes of memory.

Quantization means converting those weights to smaller number formats:

• INT8 (8-bit integers): 1 byte per weight instead of 4. Model becomes ~4x smaller.
• INT4 (4-bit integers): 0.5 bytes per weight. Model becomes ~8x smaller.

The tradeoff: some precision is lost. A weight that was 0.73291 might become 0.734. For most speech recognition tasks, this barely affects accuracy (we measured ~0.1-0.5% more errors for INT8).

Dynamic quantization (what we use) computes the conversion factors at runtime. No calibration data needed — just convert and go.

Pre-emphasis

Pre-emphasis is an audio preprocessing step that boosts high frequencies relative to low ones. The formula is simple: each audio sample has 97% of the previous sample subtracted from it.

Why? Human speech has more energy in low frequencies (the fundamental pitch of the voice) than high frequencies (the consonants like "s", "t", "f" that make words distinguishable). Pre-emphasis balances this out so the model can "hear" consonants better.

Per-Feature Normalization

After computing the mel spectrogram, each frequency bin (mel feature) is independently normalized to have mean=0 and standard deviation=1. This is like adjusting the brightness and contrast of each horizontal band in the spectrogram independently.

Why? Without normalization, some frequency bands might have much larger values than others, making it harder for the neural network to learn equally from all frequency ranges.

---

The Model's Architecture in Detail

Now that the concepts are covered, here's how Cohere Transcribe specifically works:

Raw Audio (16kHz, mono)
    |
    v
[Pre-emphasis] -- boost high frequencies
    |
    v
[STFT] -- convert to frequency domain (512 freq bins, 100 frames/sec)
    |
    v
[Mel Filterbank] -- compress 257 freq bins into 128 mel bins
    |
    v
[Log + Normalize] -- take log, normalize per mel bin
    |
    v
[Conformer Encoder] -- 48 layers of convolution + attention
  |  d_model = 1280 (each audio frame is a 1280-dimensional vector)
  |  8 attention heads
  |  ~1.8 billion parameters
    |
    v
[Linear Projection] -- shrink from 1280 to 1024 dimensions
    |
    v
[Transformer Decoder] -- 8 layers, generates text one token at a time
  |  d_model = 1024
  |  8 attention heads
  |  ~200 million parameters
  |  Uses cross-attention to look at encoder output
  |  Uses self-attention KV cache for efficiency
    |
    v
[Classifier Head] -- converts decoder output to probabilities over 16,384 tokens
    |
    v
Text Output (one token at a time until <|endoftext|>)

Total: 2 billion parameters, 4.13 GB in SafeTensors format.

---

Why a Simple Export Doesn't Work

Normally, converting a HuggingFace model to ONNX is one command:

optimum-cli export onnx --model model_name output_dir

This doesn't work here for four reasons:

1. Custom Code

The model uses trust_remote_code=True — it ships its own Python files that define the neural network architecture. The standard export tool doesn't know how to handle custom model classes.

2. Feature Extraction Mismatch

Cohere uses a unique mel spectrogram pipeline. If you use the standard one (from Whisper or any other model), the frequencies, normalization, and pre-emphasis are all different. The model was trained on these specific features — give it different features and it outputs nonsense.

What	Standard (Whisper)	What Cohere Needs
Mel frequency bins	80	128
FFT size	400	512
Pre-emphasis filter	None	0.97
Normalization	Global (whole spectrogram)	Per-feature (each frequency band separately)

3. STFT Can't Be Exported

PyTorch's built-in torch.stft function (the tool that converts audio to the frequency domain) uses complex numbers internally. The ONNX TorchScript exporter doesn't support complex number operations.

We solved this by reimplementing STFT from scratch using basic math operations (matrix multiplication via Conv1d filters). Same math, different code — and fully exportable.

4. The 2 GB File Size Limit

ONNX files use Google's Protocol Buffers format for serialization, which has a hard 2 GB limit. Our encoder at full precision is 7.1 GB. The solution: store the model graph (the computation recipe) in the .onnx file and the actual weights (the learned numbers) in separate files alongside it.

---

The Process, Step by Step

Step 1: Research the Model

Before writing any code, we needed to understand exactly how the model works internally. This meant reading through ~90 KB of custom Python source code:

File	Size	What's Inside
modeling_cohere_asr.py	64.6 KB	The neural network itself — all the layers, attention mechanisms, and the inference loop
processing_cohere_asr.py	20.6 KB	Audio preprocessing — how raw audio becomes mel spectrograms
tokenization_cohere_asr.py	6.6 KB	How text is split into tokens and reassembled
configuration_cohere_asr.py	1.7 KB	Model configuration (layer counts, dimensions, etc.)

Plus the JSON configuration files that specify exact dimensions and hyperparameters.

Key discoveries from reading the code:

1. Encoder and decoder have different sizes. The encoder works in 1280 dimensions, but the decoder works in 1024. There's a linear projection layer between them that squishes 1280 numbers down to 1024.

2. Weight tying. The decoder's final classification layer (which predicts the next token) shares its weights with the token embedding layer (which converts token IDs to vectors). This saves memory — the same 16,384 x 1024 weight matrix is used in both places.

3. Prompt format. The decoder doesn't just start generating text blindly. It receives a structured prompt of special tokens that tell it the language, whether to include punctuation, etc.

4. Long audio handling. Audio longer than 35 seconds is split into overlapping chunks, transcribed separately, and stitched back together.

Step 2: Design the ONNX Export Strategy

The key architectural decision: pre-compute cross-attention Key/Value tensors in the encoder.

In the original model, the encoder produces a hidden representation, and then each of the 8 decoder layers independently projects that representation into Key and Value tensors for cross-attention. This means those projections run at every single decoder step — wasteful, since the encoder output never changes.

Our approach: run those 8 Key projections and 8 Value projections once, inside the encoder, and pass the results to the decoder as pre-computed inputs. The decoder then only needs to compute the Query projection at each step.

This is mathematically identical (same weights, same inputs, same outputs) — we're just reorganizing when the computation happens.

Encoder ONNX graph (what we built):

Raw audio waveform (16kHz float32)
    -> Pre-emphasis
    -> STFT (via Conv1d DFT filters)
    -> Mel filterbank
    -> Log + Normalization
    -> 48 Conformer layers
    -> Linear projection (1280 -> 1024)
    -> 8x Key projection (one per decoder layer)
    -> 8x Value projection (one per decoder layer)
    -> Output: stacked cross_K and cross_V tensors

Decoder ONNX graph (what we built):

Prompt tokens + previously generated tokens
    -> Token embedding + positional encoding
    -> 8 Transformer layers, each doing:
        -> Self-attention (with KV cache for efficiency)
        -> Cross-attention (using pre-computed K/V from encoder)
        -> Feed-forward network
    -> Final layer norm
    -> Classifier (predict next token probabilities)
    -> Output: logits + updated KV cache

Step 3: Implement the ONNX-Compatible STFT

Since PyTorch's torch.stft doesn't export to ONNX, we reimplemented it using a Conv1d (1-dimensional convolution) with DFT basis filters.

How this works conceptually:

The Fourier Transform decomposes a signal into sine and cosine waves of different frequencies. We can express this as multiplying the audio signal by a matrix of sine/cosine patterns — which is equivalent to a convolution operation.

We pre-compute 257 cosine filters and 257 sine filters (for n_fft=512, you get 257 unique frequency bins), each 512 samples long, multiplied by the Hann window. These become the convolution kernels.

Running Conv1d with these filters and stride=160 (the hop length) produces the same result as torch.stft, but using only basic operations that ONNX supports.

Step 4: Export to ONNX

The actual export command:

python export_onnx_baked.py --model-dir . --output-dir ./onnx-baked

What happens internally:

1. Load the model from SafeTensors in bfloat16 format (saves memory — uses ~4 GB instead of ~8 GB during loading)
2. Convert to float32 for ONNX export (ONNX doesn't handle bfloat16 well)
3. Wrap the encoder in our custom BakedEncoderForONNX module
4. Trace the encoder with dummy input audio using torch.onnx.export(). PyTorch runs the model once, records every operation, and saves the computation graph as ONNX.
5. Wrap the decoder in our custom DecoderForONNX module
6. Trace the decoder similarly with dummy token inputs
7. Generate tokens.txt — a text file mapping each token ID to its string representation

Resource requirements:

• ~8 GB peak RAM (no GPU needed)
• ~2-3 minutes on a modern CPU
• Works on 24 GB shared RAM systems

Output files:

onnx-baked/
  cohere-encoder.onnx          (1.9 MB graph + ~7.1 GB external weight files)
  cohere-decoder.onnx          (581 MB, self-contained)
  tokens.txt                   (219 KB, 16,384 entries)

Step 5: Quantize

Quantization shrinks the model dramatically while preserving almost all accuracy:

python quantize.py --input-dir ./onnx-baked

What the quantization script does:

1. Scans the encoder for convolution and batch normalization layers (these process raw audio features and are sensitive to precision loss)
2. Protects those layers from quantization (keeps them at full 32-bit precision)
3. Quantizes everything else to INT8 using dynamic quantization
4. Quantizes the decoder entirely to INT8 (it's all attention and feed-forward layers, which handle INT8 well)

Results:

Component	Before (FP32)	After (INT8)	Size Reduction
Encoder	7.1 GB	2.6 GB	64% smaller
Decoder	581 MB	146 MB	75% smaller
Total	7.7 GB	2.75 GB	65% smaller

The encoder gets less compression because we protected the audio processing layers. The decoder compresses more because its weights (large matrix multiplications in the feed-forward layers) are ideal candidates for INT8.

For even more aggressive compression, add the --int4 flag to also produce INT4 models (~1.5 GB total, with slightly more accuracy risk).

Step 6: Run Inference

The inference loop has two phases:

Phase 1: Encode the audio

Send the raw audio waveform (as a flat array of float32 numbers at 16kHz) to the encoder. Get back two tensors representing the encoder's understanding of the audio.

cross_k, cross_v = encoder.run(None, {"audio": audio_array})

Phase 2: Decode text token by token

Start with the prompt tokens (language, punctuation settings, etc.). At each step:

1. Feed the current token(s) to the decoder along with the encoder's cross_k/cross_v
2. The decoder outputs a probability distribution over all 16,384 possible next tokens
3. Pick the most likely token (this is called greedy decoding)
4. If that token is <|endoftext|>, stop — the transcription is complete
5. Otherwise, add it to the output and repeat

for step in range(max_tokens):
    logits, kv_cache = decoder.run(inputs)
    next_token = argmax(logits)       # pick the most likely token
    if next_token == end_of_text:
        break
    # feed next_token back into the decoder for the next step

Phase 3: Convert tokens to text

The output is a list of token IDs. Look each one up in tokens.txt to get the text piece, replace the SentencePiece word-boundary marker (_) with a space, skip any special tokens, and join everything together.

Handling Long Audio

The model can only process about 35 seconds of audio at once (this is a limitation of the Conformer's positional encoding). For longer audio, we split it into overlapping chunks:

|---- chunk 1 (30s) ----|
                    |---- chunk 2 (30s) ----|
                                        |---- chunk 3 (30s) ----|
         5s overlap -->|  |<-- 5s overlap

• Chunk size: 30 seconds (leaving headroom below the 35s limit)
• Overlap: 5 seconds (so we don't cut words in half)
• Process: Transcribe each chunk independently, join results with spaces

For a 57-minute interview, this produces 137 chunks. Each chunk takes ~5-7 seconds on CPU.

Changing Language

The language is controlled by two tokens in the decoder prompt. To switch from English to German:

# English
prompt = [..., "<|en|>", "<|en|>", ...]

# German
prompt = [..., "<|de|>", "<|de|>", ...]

# French
prompt = [..., "<|fr|>", "<|fr|>", ...]

The first token is the source language (what language is being spoken), the second is the target language (what language to output). For transcription, these are always the same.

Supported: en, de, fr, es, it, pt, nl, pl, el, ar, ja, zh, vi, ko

---

Real-World Results

All tests run on a Windows 11 laptop, CPU only (no GPU), 24 GB shared RAM.

Audio File	Language	Duration	Processing Time	Speed	Tokens
Test recording	English	30.5s	5.4s	5.6x realtime	61
Interview podcast	English	57.1 min	14 min 53s	3.8x realtime	~14,500
German podcast	German	35.0 min	9 min 24s	3.7x realtime	~8,900

Peak RAM usage: ~3.5-4 GB regardless of audio length (only one chunk is in memory at a time).

With GPU (DirectML on any DirectX 12 GPU): expected 3-5x additional speedup, bringing it to 15-25x realtime.

---

What's Preserved, What's Changed

Mathematically identical (zero accuracy loss)

• The ONNX export itself — same weights, same computation graph
• Pre-computing cross-attention K/V in the encoder — same linear algebra, just reordered
• The baked feature extraction — same STFT, same mel filterbank, same normalization

Minor numerical differences (negligible accuracy impact)

• INT8 quantization: ~0.1-0.5% more word errors. For a 1000-word transcription, that's 1-5 extra errors.
• Dither removal: The original applies tiny random noise (scale 1e-5) during feature extraction. We skip this since it's a training-time augmentation with unmeasurable impact on accuracy.

Not available (never was in the model)

• Automatic language detection (you must specify the language)
• Timestamps (the model doesn't output when words were spoken)
• Speaker diarization (the model doesn't identify who is speaking)

---

Files Reference

File	Purpose
export_onnx_baked.py	Main export script — produces ONNX files with baked feature extraction
export_onnx.py	Alternative export without baked features (encoder takes mel spectrogram input)
quantize.py	Shrinks ONNX files via INT8/INT4 quantization
CohereTranscribe.cs	Complete C# inference example using Microsoft.ML.OnnxRuntime
tokens.txt (generated)	Maps token IDs to text strings (16,384 entries)
PORTING_GUIDE.md	This document

---

Quick Start Commands

# 1. Install dependencies (one time)
pip install torch transformers safetensors sentencepiece onnx onnxruntime onnxscript librosa soundfile protobuf

# 2. Export model to ONNX (one time, ~3 minutes, ~8 GB RAM)
python export_onnx_baked.py --model-dir . --output-dir ./onnx-baked

# 3. Quantize to INT8 (one time, ~2 minutes)
python quantize.py --input-dir ./onnx-baked

# 4. Transcribe (as many times as you want)
# Use the inference code from Step 6 above, or the C# example in CohereTranscribe.cs

ONNX Tensor Shapes Reference

For anyone implementing their own inference loop in any language:

Encoder

Input:   audio              shape (batch, samples)       dtype float32
                            Raw 16kHz mono waveform

Output:  n_layer_cross_k    shape (8, batch, T', 1024)   dtype float32
         n_layer_cross_v    shape (8, batch, T', 1024)   dtype float32
                            T' = encoder time steps, roughly (samples / 16000) * 12.7

Decoder

Input:   tokens                    shape (batch, n_tokens)         dtype int64
         in_n_layer_self_k_cache   shape (8, batch, 8, 1024, 128)  dtype float32
         in_n_layer_self_v_cache   shape (8, batch, 8, 1024, 128)  dtype float32
         n_layer_cross_k           shape (8, batch, T', 1024)      dtype float32
         n_layer_cross_v           shape (8, batch, T', 1024)      dtype float32
         offset                    shape () (scalar)                dtype int64

Output:  logits                    shape (batch, n_tokens, 16384)  dtype float32
         out_n_layer_self_k_cache  shape (8, batch, 8, 1024, 128)  dtype float32
         out_n_layer_self_v_cache  shape (8, batch, 8, 1024, 128)  dtype float32

Decoding loop:

1. First call: pass all prompt tokens (9 tokens), offset = 0
2. Each subsequent call: pass 1 token (the one you just generated), offset = previous offset + previous n_tokens
3. Take argmax of logits at the last token position to get the next token ID
4. Stop when the token ID is 3 (<|endoftext|>)
5. Initialize KV caches as all zeros before the first call