# Porting Cohere Transcribe to Run Locally: A Complete Guide ## What We Did (And Why It Matters) We took a brand-new AI speech recognition model — [Cohere Transcribe](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026), released March 26, 2026 — and made it run completely offline on a regular laptop. No internet connection, no cloud API, no paid subscription. The result: a 57-minute English interview transcribed in under 15 minutes, and a 35-minute German podcast in under 10 minutes. On CPU alone, no fancy GPU required. This guide explains every step, every concept, and every decision along the way. --- ## Background: What Is a Speech Recognition Model? A speech recognition model (also called ASR — Automatic Speech Recognition) takes audio as input and produces text as output. Think of it like a very sophisticated dictation machine. Modern ASR models are neural networks — software that learns patterns from millions of hours of audio paired with their correct transcriptions. Once trained, the model can listen to new audio it's never heard before and produce accurate text. ### Why "Port" a Model? The Cohere Transcribe model was released as a Python-only package that normally runs through HuggingFace's `transformers` library. That's fine for Python developers, but it means: - You need Python installed - You need ~15 GB of Python dependencies - It's hard to use from other programming languages (C#, C++, mobile apps) - It's slower than it could be By "porting" the model to **ONNX** format (explained below), we make it usable from virtually any programming language, on any platform, with better performance. --- ## Key Concepts Explained Before diving into the process, here are the technical concepts you'll encounter. Each one is explained in plain language. ### ONNX (Open Neural Network Exchange) Think of ONNX as a **universal file format for AI models** — like how PDF is a universal format for documents. A model saved as ONNX can be loaded and run by any ONNX-compatible runtime, regardless of which framework (PyTorch, TensorFlow, etc.) originally created it. **ONNX Runtime** is the engine that actually runs ONNX models. Microsoft maintains it, and it has bindings for C#, Python, C++, Java, JavaScript, and more. ### Encoder-Decoder Architecture Cohere Transcribe uses an **encoder-decoder** design, which is like a two-stage translation pipeline: 1. **Encoder**: Listens to the audio and builds an internal understanding of what was said. Think of this as a person listening very carefully and taking detailed notes — but in a form only they can read. 2. **Decoder**: Reads those internal notes and writes out the actual text, one word (actually one token) at a time. It works left-to-right, like a person typing out what was said. The encoder runs once per audio clip. The decoder runs once per output token (word piece). For a 30-second clip that produces 50 words, the encoder runs 1 time and the decoder runs ~60 times. ### Conformer (The Encoder Type) The encoder in this model is a **Conformer** — a hybrid architecture that combines two different approaches: - **Convolutional layers**: Good at detecting local patterns (specific sounds, phonemes, sharp transitions in audio). Think of these as recognizing individual letters. - **Transformer/Attention layers**: Good at understanding global context (what the whole sentence means). Think of these as understanding how letters form words and sentences. By alternating between convolution and attention, the Conformer captures both fine-grained sound details and big-picture meaning. This model has 48 of these combined layers stacked on top of each other. ### Mel Spectrogram (How Audio Becomes Numbers) Computers can't directly understand sound waves. The raw audio (a sequence of amplitude values, like the wiggly line you see in an audio editor) needs to be converted into a form the neural network can process. A **mel spectrogram** is a visual representation of sound that shows: - **Time** on the horizontal axis (left to right) - **Frequency** on the vertical axis (low sounds at bottom, high sounds at top) - **Intensity** as brightness (louder = brighter) The "mel" part means the frequencies are spaced according to how human hearing works — we're more sensitive to differences between low frequencies than high ones. The mel scale matches this. The conversion process: raw audio -> apply math (STFT/Fourier transform) -> group frequencies into mel bins -> take the logarithm. The result is a grid of numbers that the encoder can process. ### STFT (Short-Time Fourier Transform) The **STFT** is the mathematical tool that converts audio from the time domain (a wiggly line showing amplitude over time) to the frequency domain (showing which frequencies are present at each moment). It works by sliding a window across the audio and, at each position, computing which frequencies are present in that small window. It's like analyzing a song not all at once, but moment-by-moment — "right now there's a bass note and a vocal, now it's just drums..." Key parameters: - **n_fft** (512): How many frequency bins to compute. More bins = finer frequency resolution. - **hop_length** (160): How far the window slides between each analysis. At 16,000 samples per second with hop=160, you get 100 analysis frames per second of audio. - **window** (hann, size 400): A smooth curve applied to each analysis window to prevent audio artifacts at the edges. ### Tokens and Tokenization Neural networks don't work with words directly. Instead, text is broken into **tokens** — small pieces that might be whole words, parts of words, or individual characters. This model uses **SentencePiece BPE** (Byte Pair Encoding) with 16,384 tokens. Some examples: - Common words are single tokens: "the" = one token - Less common words get split: "transcription" might become "trans" + "cription" - The special character `_` (Unicode 2581) marks the start of a new word The model also has **special tokens** that control its behavior: - `<|startoftranscript|>` — signals "start producing text now" - `<|en|>` — tells it to transcribe in English - `<|pnc|>` — tells it to include punctuation - `<|endoftext|>` — the model outputs this when it's done ### Attention and Cross-Attention **Attention** is the mechanism that lets the model focus on relevant parts of its input when producing each output. **Self-attention** (used within both encoder and decoder): Each element in a sequence looks at every other element in the same sequence to understand context. In the encoder, each audio frame "pays attention to" every other audio frame — so a word can be understood in context. **Cross-attention** (connecting encoder to decoder): When the decoder generates each word, it looks back at the encoder's representation of the audio. It's like asking "given what I've written so far, what part of the audio should I focus on next?" Cross-attention works through three projections: - **Query (Q)**: "What am I looking for?" — comes from the decoder - **Key (K)**: "What do I contain?" — comes from the encoder - **Value (V)**: "What information do I provide?" — comes from the encoder The decoder computes a similarity between Q and K to figure out where to focus, then retrieves the corresponding V. This happens at every decoder layer (8 layers in this model). ### KV Cache When the decoder generates text one token at a time, it would be wasteful to reprocess all previous tokens at every step. Instead, we save the Key and Value computations from previous steps in a **KV cache**. Think of it like a growing notebook. At step 1, you write one line. At step 2, you don't rewrite the whole notebook — you just add a new line and re-read what you need. The KV cache is that notebook. For the **self-attention** KV cache: it grows with each decoder step (we add new entries as we generate new tokens). For the **cross-attention** K/V: these are computed once from the encoder output and then reused at every decoder step (the audio doesn't change — we're just looking at it from different angles). ### Quantization Neural network weights are normally stored as **32-bit floating point** numbers (FP32). Each weight takes 4 bytes of memory. **Quantization** means converting those weights to smaller number formats: - **INT8** (8-bit integers): 1 byte per weight instead of 4. Model becomes ~4x smaller. - **INT4** (4-bit integers): 0.5 bytes per weight. Model becomes ~8x smaller. The tradeoff: some precision is lost. A weight that was `0.73291` might become `0.734`. For most speech recognition tasks, this barely affects accuracy (we measured ~0.1-0.5% more errors for INT8). **Dynamic quantization** (what we use) computes the conversion factors at runtime. No calibration data needed — just convert and go. ### Pre-emphasis **Pre-emphasis** is an audio preprocessing step that boosts high frequencies relative to low ones. The formula is simple: each audio sample has 97% of the previous sample subtracted from it. Why? Human speech has more energy in low frequencies (the fundamental pitch of the voice) than high frequencies (the consonants like "s", "t", "f" that make words distinguishable). Pre-emphasis balances this out so the model can "hear" consonants better. ### Per-Feature Normalization After computing the mel spectrogram, each frequency bin (mel feature) is independently normalized to have mean=0 and standard deviation=1. This is like adjusting the brightness and contrast of each horizontal band in the spectrogram independently. Why? Without normalization, some frequency bands might have much larger values than others, making it harder for the neural network to learn equally from all frequency ranges. --- ## The Model's Architecture in Detail Now that the concepts are covered, here's how Cohere Transcribe specifically works: ``` Raw Audio (16kHz, mono) | v [Pre-emphasis] -- boost high frequencies | v [STFT] -- convert to frequency domain (512 freq bins, 100 frames/sec) | v [Mel Filterbank] -- compress 257 freq bins into 128 mel bins | v [Log + Normalize] -- take log, normalize per mel bin | v [Conformer Encoder] -- 48 layers of convolution + attention | d_model = 1280 (each audio frame is a 1280-dimensional vector) | 8 attention heads | ~1.8 billion parameters | v [Linear Projection] -- shrink from 1280 to 1024 dimensions | v [Transformer Decoder] -- 8 layers, generates text one token at a time | d_model = 1024 | 8 attention heads | ~200 million parameters | Uses cross-attention to look at encoder output | Uses self-attention KV cache for efficiency | v [Classifier Head] -- converts decoder output to probabilities over 16,384 tokens | v Text Output (one token at a time until <|endoftext|>) ``` **Total: 2 billion parameters, 4.13 GB in SafeTensors format.** --- ## Why a Simple Export Doesn't Work Normally, converting a HuggingFace model to ONNX is one command: ```bash optimum-cli export onnx --model model_name output_dir ``` This doesn't work here for four reasons: ### 1. Custom Code The model uses `trust_remote_code=True` — it ships its own Python files that define the neural network architecture. The standard export tool doesn't know how to handle custom model classes. ### 2. Feature Extraction Mismatch Cohere uses a unique mel spectrogram pipeline. If you use the standard one (from Whisper or any other model), the frequencies, normalization, and pre-emphasis are all different. The model was trained on these specific features — give it different features and it outputs nonsense. | What | Standard (Whisper) | What Cohere Needs | |------|-------------------|-------------------| | Mel frequency bins | 80 | **128** | | FFT size | 400 | **512** | | Pre-emphasis filter | None | **0.97** | | Normalization | Global (whole spectrogram) | **Per-feature** (each frequency band separately) | ### 3. STFT Can't Be Exported PyTorch's built-in `torch.stft` function (the tool that converts audio to the frequency domain) uses complex numbers internally. The ONNX TorchScript exporter doesn't support complex number operations. We solved this by reimplementing STFT from scratch using basic math operations (matrix multiplication via Conv1d filters). Same math, different code — and fully exportable. ### 4. The 2 GB File Size Limit ONNX files use Google's Protocol Buffers format for serialization, which has a hard 2 GB limit. Our encoder at full precision is 7.1 GB. The solution: store the model graph (the computation recipe) in the `.onnx` file and the actual weights (the learned numbers) in separate files alongside it. --- ## The Process, Step by Step ### Step 1: Research the Model Before writing any code, we needed to understand exactly how the model works internally. This meant reading through ~90 KB of custom Python source code: | File | Size | What's Inside | |------|------|---------------| | `modeling_cohere_asr.py` | 64.6 KB | The neural network itself — all the layers, attention mechanisms, and the inference loop | | `processing_cohere_asr.py` | 20.6 KB | Audio preprocessing — how raw audio becomes mel spectrograms | | `tokenization_cohere_asr.py` | 6.6 KB | How text is split into tokens and reassembled | | `configuration_cohere_asr.py` | 1.7 KB | Model configuration (layer counts, dimensions, etc.) | Plus the JSON configuration files that specify exact dimensions and hyperparameters. **Key discoveries from reading the code:** 1. **Encoder and decoder have different sizes.** The encoder works in 1280 dimensions, but the decoder works in 1024. There's a linear projection layer between them that squishes 1280 numbers down to 1024. 2. **Weight tying.** The decoder's final classification layer (which predicts the next token) shares its weights with the token embedding layer (which converts token IDs to vectors). This saves memory — the same 16,384 x 1024 weight matrix is used in both places. 3. **Prompt format.** The decoder doesn't just start generating text blindly. It receives a structured prompt of special tokens that tell it the language, whether to include punctuation, etc. 4. **Long audio handling.** Audio longer than 35 seconds is split into overlapping chunks, transcribed separately, and stitched back together. ### Step 2: Design the ONNX Export Strategy The key architectural decision: **pre-compute cross-attention Key/Value tensors in the encoder.** In the original model, the encoder produces a hidden representation, and then each of the 8 decoder layers independently projects that representation into Key and Value tensors for cross-attention. This means those projections run at every single decoder step — wasteful, since the encoder output never changes. Our approach: run those 8 Key projections and 8 Value projections once, inside the encoder, and pass the results to the decoder as pre-computed inputs. The decoder then only needs to compute the Query projection at each step. This is mathematically identical (same weights, same inputs, same outputs) — we're just reorganizing when the computation happens. **Encoder ONNX graph (what we built):** ``` Raw audio waveform (16kHz float32) -> Pre-emphasis -> STFT (via Conv1d DFT filters) -> Mel filterbank -> Log + Normalization -> 48 Conformer layers -> Linear projection (1280 -> 1024) -> 8x Key projection (one per decoder layer) -> 8x Value projection (one per decoder layer) -> Output: stacked cross_K and cross_V tensors ``` **Decoder ONNX graph (what we built):** ``` Prompt tokens + previously generated tokens -> Token embedding + positional encoding -> 8 Transformer layers, each doing: -> Self-attention (with KV cache for efficiency) -> Cross-attention (using pre-computed K/V from encoder) -> Feed-forward network -> Final layer norm -> Classifier (predict next token probabilities) -> Output: logits + updated KV cache ``` ### Step 3: Implement the ONNX-Compatible STFT Since PyTorch's `torch.stft` doesn't export to ONNX, we reimplemented it using a Conv1d (1-dimensional convolution) with DFT basis filters. **How this works conceptually:** The Fourier Transform decomposes a signal into sine and cosine waves of different frequencies. We can express this as multiplying the audio signal by a matrix of sine/cosine patterns — which is equivalent to a convolution operation. We pre-compute 257 cosine filters and 257 sine filters (for n_fft=512, you get 257 unique frequency bins), each 512 samples long, multiplied by the Hann window. These become the convolution kernels. Running Conv1d with these filters and stride=160 (the hop length) produces the same result as `torch.stft`, but using only basic operations that ONNX supports. ### Step 4: Export to ONNX The actual export command: ```bash python export_onnx_baked.py --model-dir . --output-dir ./onnx-baked ``` What happens internally: 1. **Load the model** from SafeTensors in bfloat16 format (saves memory — uses ~4 GB instead of ~8 GB during loading) 2. **Convert to float32** for ONNX export (ONNX doesn't handle bfloat16 well) 3. **Wrap the encoder** in our custom `BakedEncoderForONNX` module 4. **Trace the encoder** with dummy input audio using `torch.onnx.export()`. PyTorch runs the model once, records every operation, and saves the computation graph as ONNX. 5. **Wrap the decoder** in our custom `DecoderForONNX` module 6. **Trace the decoder** similarly with dummy token inputs 7. **Generate tokens.txt** — a text file mapping each token ID to its string representation **Resource requirements:** - ~8 GB peak RAM (no GPU needed) - ~2-3 minutes on a modern CPU - Works on 24 GB shared RAM systems **Output files:** ``` onnx-baked/ cohere-encoder.onnx (1.9 MB graph + ~7.1 GB external weight files) cohere-decoder.onnx (581 MB, self-contained) tokens.txt (219 KB, 16,384 entries) ``` ### Step 5: Quantize Quantization shrinks the model dramatically while preserving almost all accuracy: ```bash python quantize.py --input-dir ./onnx-baked ``` What the quantization script does: 1. **Scans the encoder** for convolution and batch normalization layers (these process raw audio features and are sensitive to precision loss) 2. **Protects those layers** from quantization (keeps them at full 32-bit precision) 3. **Quantizes everything else** to INT8 using dynamic quantization 4. **Quantizes the decoder** entirely to INT8 (it's all attention and feed-forward layers, which handle INT8 well) **Results:** | Component | Before (FP32) | After (INT8) | Size Reduction | |-----------|---------------|-------------|----------------| | Encoder | 7.1 GB | 2.6 GB | 64% smaller | | Decoder | 581 MB | 146 MB | 75% smaller | | **Total** | **7.7 GB** | **2.75 GB** | **65% smaller** | The encoder gets less compression because we protected the audio processing layers. The decoder compresses more because its weights (large matrix multiplications in the feed-forward layers) are ideal candidates for INT8. For even more aggressive compression, add the `--int4` flag to also produce INT4 models (~1.5 GB total, with slightly more accuracy risk). ### Step 6: Run Inference The inference loop has two phases: **Phase 1: Encode the audio** Send the raw audio waveform (as a flat array of float32 numbers at 16kHz) to the encoder. Get back two tensors representing the encoder's understanding of the audio. ```python cross_k, cross_v = encoder.run(None, {"audio": audio_array}) ``` **Phase 2: Decode text token by token** Start with the prompt tokens (language, punctuation settings, etc.). At each step: 1. Feed the current token(s) to the decoder along with the encoder's cross_k/cross_v 2. The decoder outputs a probability distribution over all 16,384 possible next tokens 3. Pick the most likely token (this is called **greedy decoding**) 4. If that token is `<|endoftext|>`, stop — the transcription is complete 5. Otherwise, add it to the output and repeat ```python for step in range(max_tokens): logits, kv_cache = decoder.run(inputs) next_token = argmax(logits) # pick the most likely token if next_token == end_of_text: break # feed next_token back into the decoder for the next step ``` **Phase 3: Convert tokens to text** The output is a list of token IDs. Look each one up in `tokens.txt` to get the text piece, replace the SentencePiece word-boundary marker (`_`) with a space, skip any special tokens, and join everything together. ### Handling Long Audio The model can only process about 35 seconds of audio at once (this is a limitation of the Conformer's positional encoding). For longer audio, we split it into overlapping chunks: ``` |---- chunk 1 (30s) ----| |---- chunk 2 (30s) ----| |---- chunk 3 (30s) ----| 5s overlap -->| |<-- 5s overlap ``` - **Chunk size**: 30 seconds (leaving headroom below the 35s limit) - **Overlap**: 5 seconds (so we don't cut words in half) - **Process**: Transcribe each chunk independently, join results with spaces For a 57-minute interview, this produces 137 chunks. Each chunk takes ~5-7 seconds on CPU. ### Changing Language The language is controlled by two tokens in the decoder prompt. To switch from English to German: ```python # English prompt = [..., "<|en|>", "<|en|>", ...] # German prompt = [..., "<|de|>", "<|de|>", ...] # French prompt = [..., "<|fr|>", "<|fr|>", ...] ``` The first token is the source language (what language is being spoken), the second is the target language (what language to output). For transcription, these are always the same. Supported: en, de, fr, es, it, pt, nl, pl, el, ar, ja, zh, vi, ko --- ## Real-World Results All tests run on a Windows 11 laptop, CPU only (no GPU), 24 GB shared RAM. | Audio File | Language | Duration | Processing Time | Speed | Tokens | |-----------|----------|----------|----------------|-------|--------| | Test recording | English | 30.5s | 5.4s | 5.6x realtime | 61 | | Interview podcast | English | 57.1 min | 14 min 53s | 3.8x realtime | ~14,500 | | German podcast | German | 35.0 min | 9 min 24s | 3.7x realtime | ~8,900 | **Peak RAM usage: ~3.5-4 GB** regardless of audio length (only one chunk is in memory at a time). **With GPU** (DirectML on any DirectX 12 GPU): expected 3-5x additional speedup, bringing it to 15-25x realtime. --- ## What's Preserved, What's Changed ### Mathematically identical (zero accuracy loss) - The ONNX export itself — same weights, same computation graph - Pre-computing cross-attention K/V in the encoder — same linear algebra, just reordered - The baked feature extraction — same STFT, same mel filterbank, same normalization ### Minor numerical differences (negligible accuracy impact) - **INT8 quantization**: ~0.1-0.5% more word errors. For a 1000-word transcription, that's 1-5 extra errors. - **Dither removal**: The original applies tiny random noise (scale 1e-5) during feature extraction. We skip this since it's a training-time augmentation with unmeasurable impact on accuracy. ### Not available (never was in the model) - Automatic language detection (you must specify the language) - Timestamps (the model doesn't output when words were spoken) - Speaker diarization (the model doesn't identify who is speaking) --- ## Files Reference | File | Purpose | |------|---------| | `export_onnx_baked.py` | Main export script — produces ONNX files with baked feature extraction | | `export_onnx.py` | Alternative export without baked features (encoder takes mel spectrogram input) | | `quantize.py` | Shrinks ONNX files via INT8/INT4 quantization | | `CohereTranscribe.cs` | Complete C# inference example using Microsoft.ML.OnnxRuntime | | `tokens.txt` (generated) | Maps token IDs to text strings (16,384 entries) | | `PORTING_GUIDE.md` | This document | --- ## Quick Start Commands ```bash # 1. Install dependencies (one time) pip install torch transformers safetensors sentencepiece onnx onnxruntime onnxscript librosa soundfile protobuf # 2. Export model to ONNX (one time, ~3 minutes, ~8 GB RAM) python export_onnx_baked.py --model-dir . --output-dir ./onnx-baked # 3. Quantize to INT8 (one time, ~2 minutes) python quantize.py --input-dir ./onnx-baked # 4. Transcribe (as many times as you want) # Use the inference code from Step 6 above, or the C# example in CohereTranscribe.cs ``` ## ONNX Tensor Shapes Reference For anyone implementing their own inference loop in any language: ### Encoder ``` Input: audio shape (batch, samples) dtype float32 Raw 16kHz mono waveform Output: n_layer_cross_k shape (8, batch, T', 1024) dtype float32 n_layer_cross_v shape (8, batch, T', 1024) dtype float32 T' = encoder time steps, roughly (samples / 16000) * 12.7 ``` ### Decoder ``` Input: tokens shape (batch, n_tokens) dtype int64 in_n_layer_self_k_cache shape (8, batch, 8, 1024, 128) dtype float32 in_n_layer_self_v_cache shape (8, batch, 8, 1024, 128) dtype float32 n_layer_cross_k shape (8, batch, T', 1024) dtype float32 n_layer_cross_v shape (8, batch, T', 1024) dtype float32 offset shape () (scalar) dtype int64 Output: logits shape (batch, n_tokens, 16384) dtype float32 out_n_layer_self_k_cache shape (8, batch, 8, 1024, 128) dtype float32 out_n_layer_self_v_cache shape (8, batch, 8, 1024, 128) dtype float32 ``` **Decoding loop:** 1. First call: pass all prompt tokens (9 tokens), offset = 0 2. Each subsequent call: pass 1 token (the one you just generated), offset = previous offset + previous n_tokens 3. Take argmax of logits at the last token position to get the next token ID 4. Stop when the token ID is 3 (`<|endoftext|>`) 5. Initialize KV caches as all zeros before the first call