README.md

---
base_model: microsoft/Phi-4-multimodal-instruct-onnx
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: onnx
tags:
  - onnx
  - onnxruntime-genai
  - int8
  - gptq
  - weight-only-quantization
  - vision-language
  - speech
  - phi4mm
license: mit
license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE
---

# Phi-4-multimodal-instruct — ONNX INT8 decoder + GPTQ INT4 vision

An **INT8 language decoder** paired with a **GPTQ-calibrated INT4 vision tower**, rebuilt from
[`microsoft/Phi-4-multimodal-instruct-onnx`](https://huggingface.co/microsoft/Phi-4-multimodal-instruct-onnx)
for [`onnxruntime-genai`](https://github.com/microsoft/onnxruntime-genai) on the **CUDA** execution
provider. 5.6 B parameters: a 3.8 B Phi-4-mini language backbone, a ~0.45 B SigLIP vision encoder,
a ~0.46 B conformer speech encoder, and the modality LoRA adapters.

**This is the byte-exact build.** Its sibling
[`jarvis-pet/Phi-4-multimodal-instruct-gptq-int4-onnx`](https://huggingface.co/jarvis-pet/Phi-4-multimodal-instruct-gptq-int4-onnx)
is 1.99 GiB smaller and buys a bigger context window plus both modalities in one process on a 12 GB
card, but it drops the occasional character in short strings. This one does not: pick it when
byte-exact short-field extraction matters more than VRAM headroom.

**Why it is not simply the published artifact.** Microsoft's GPU release quantizes both towers with
**INT4 RTN**, and both are damaged by it. On the vision tower, `image_features` measured against
FP16 have a **relative L2 error of 0.94–1.09** — error equal to signal — so OCR either degenerates
into repetition or confabulates. The INT4 decoder separately breaks EOS, so page transcription never
terminates. `int4` was the whole problem, in both components; this build replaces the tower with
data-aware GPTQ and the decoder with 8-bit.

**The name states the DECODER's precision**, matching the convention already used for this account's
OpenVINO uploads (where `-int4-ov` likewise names the language model while other submodels sit at
INT8). The vision tower here is GPTQ INT4, as the table below records.

Released under the **MIT License**, inherited from the base model
[`microsoft/Phi-4-multimodal-instruct`](https://huggingface.co/microsoft/Phi-4-multimodal-instruct).
This is a derivative quantization only; all model rights and terms follow the base model.

## Files

19 files, **6.78 GiB** total:

| Component | Precision | Notes |
|---|---|---|
| `phi-4-mm-text.onnx{,.data}` | **INT8** | the decoder, **4.55 GB** — 673 `MatMulNBits`, `bits: 8` |
| `phi-4-mm-vision.onnx{,.data}` | **INT4 GPTQ**, block 128 | vision tower, 370 MB — 80 `MatMulNBits` |
| `phi-4-mm-speech.onnx{,.data}` | as published | conformer speech encoder, 520 MB |
| `phi-4-mm-embedding.onnx{,.data}` | as published | shared embedding graph, 1.23 GB |
| `phi-4-mm-vision.onnx_adapter`, `phi-4-mm-speech.onnx_adapter` | as published | **present but not referenced** — see below |
| `genai_config.json`, `vision_processor.json`, `speech_processor.json` | — | `genai_config.json` differs from the published one |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json` | — | unchanged |

**The adapter files are vestigial here, deliberately.** The locally built decoder bakes the LoRAs in
(896 `lora_*` initializers, 0 adapter inputs — `Phi4MMModel.make_layer` makes **vision** the baked
default), so `adapter_filename` has been **removed** from the `vision` and `speech` sections of
`genai_config.json`. Leaving it in makes the runtime bind tensors the graph does not have. Losing the
*speech* LoRA was expected to cost audio quality — measured, it does not (see below).

## Audio input — 16 kHz required, 8 kHz optional

The speech encoder consumes **80-bin log-mel features at a 25 ms window and a 10 ms hop**.
`speech_processor.json` declares `AudioDecoderEx.target_sample_rates: [8000, 16000]`, and
`Phi4AudioEmbed` carries a complete STFT/log-mel parameter set for **each** rate:

| | **16 kHz — required / primary** | **8 kHz — optional** |
|---|---|---|
| `n_fft` | 512 | 256 |
| `frame_length` | 400 → 25 ms | 200 → 25 ms |
| `hop_length` | 160 → 10 ms | 80 → 10 ms |
| `n_mel` | 80 | 80 |

Both paths emit the same feature geometry — 80 mel bins at 100 frames/second — which is what the
encoder is built for; `audio_compression_rate: 8` then decimates that 8× along time. What differs is
bandwidth, not shape.

- **Supply 16 kHz mono PCM16 WAV.** This is the primary path and what every audio measurement below
  used.
- **8 kHz is natively supported**, not merely tolerated — a first-class second parameter set intended
  for telephony-band input. Nyquist is 4 kHz there, so the 4–8 kHz band is absent from the features;
  expect that to cost fricative and sibilant detail.
- **Any other rate is resampled by `AudioDecoderEx`**, not rejected. That path is untested here, so
  do not rely on its fidelity — resample to 16 kHz yourself if you care about the transcript. This
  differs from an OpenVINO serve of the same model, where nothing resamples and a wrong-rate waveform
  transcribes confidently wrong.

## Measured results

Measured on an **RTX 3060 12 GB** (Windows 11, driver 610.88, no CUDA toolkit — pip wheels only),
`onnxruntime-genai-cuda 0.15.2` + `onnxruntime-gpu 1.29.0`, at **`max_length = 4096`**:

| Probe | Result |
|---|---|
| `Ocr.Tests.ps1` | **PASS=4 FAIL=0** — `"JARVIS OCR OCRPASS-4713"` read **byte-perfect** in 1.3 s |
| `OrtGenai.Tests.ps1` | **36/36** |
| Dense 850×1100 invoice, full transcription | `finish_reason=stop`, 186 tokens, 7.8 s, longest identical-line repeat **1**, **6/6** gold fields including `Subtotal 2,355.21` and routing `021000021` |
| Short-answer extraction | `Total Due` → `2,549.51`, 6 tokens, 1.3 s |
| Audio, 10 s 16 kHz clip | **byte-perfect** transcription *including* `"at sunrise"` (which the published INT4 build got wrong), plus content Q&A and correct French translation |
| VRAM | 8,849 MiB idle → **11,683 MiB after a page** |

Against the published INT4-RTN artifact on the same invoice: `finish_reason` `length` → **`stop`**,
identical-line repeats 10 → **1**, gold fields 5/6 → **6/6**, `Subtotal` wrong → **exact**, routing
number MISS → **HIT**, and severe character doubling → **none**.

### Cost: 12 GB serves one modality per process

Weights are 7.4 GB, and ORT's BFC arena grows and never releases, so:

- **`max_length = 8192` does not fit** a 12 GB card at all. 4096 works, for OCR-only or audio-only.
- **An audio request after a page request dies** with
  `AllocateRawInternal Failed to allocate memory for requested buffer of size 1338028032` — the
  vision activation buffer — with ~1.4 GB nominally free. A serve that must do both in one process
  needs a bigger card.

Both limits are what the INT4-decoder sibling exists to lift: at 4.77 GB of weights it holds 8192
*and* takes audio after a page in the same process, on the same 12 GB card.

## How the quantization was produced

- **Decoder — INT8.** Upstream's `build_text()` hardcodes `precision = "int4"` and ignores
  `--precision`, so `create_model` was called directly with `precision="int8"` and
  `exclude_embeds=true`, `filename=phi-4-mm-text.onnx` to match what `genai_config.json` names.
- **Vision tower — GPTQ INT4, block 128.** `build_vision()` runs the nbits quantizer
  *unconditionally* at INT4 RTN, so the FP16 tower was exported with that step patched out and
  `torch.onnx.export(..., dynamo=False)` (torch ≥ 2.9's dynamo exporter fails to capture the SigLIP
  tower and silently rewrites opset 14 → 18), then quantized with
  `MatMulNBitsQuantizer(bits=4, block_size=128, is_symmetric=True)` under
  `GPTQWeightOnlyQuantConfig(percdamp=0.1, actorder=False, mse=False, perchannel=True)`.
- **GPTQ is Python-API-only:** `--quant_method gptq` from the CLI builds the config with no
  `calibration_data_reader`, so INC iterates `None` and dies with
  `TypeError: 'NoneType' object is not iterable`.

Fidelity of the vision tower against FP16, five page fixtures, CUDA EP:

| tower | size | cosine | rel L2 |
|---|---|---|---|
| INT8 block 32 | 536 MB | 0.99990–0.99992 | 0.012 |
| **INT4 GPTQ block 128 (this build)** | **370 MB** | **0.950–0.961** | 0.29–0.33 |
| INT4 GPTQ block 32 | 383 MB | 0.926–0.944 | 0.33–0.38 |
| INT4 RTN block 32 (published) | 383 MB | **0.414–0.594** | **0.94–1.09** |

Cosine 0.95 passes every end-to-end probe and scored **identically** to an INT8 tower while being
166 MB smaller, which is why the INT4 GPTQ tower is used here rather than an INT8 one. Note finer
groups made GPTQ *worse*, the opposite of the usual expectation.

### Gotchas worth knowing

- **`device_id` must be 0.** ORT's CUDA EP creates its cudnn handle before `cudaSetDevice`, so any
  non-zero `device_id` dies at model load with
  `CUDNN failure 2007: CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH`. To serve on a different physical
  card, mask it with `CUDA_VISIBLE_DEVICES=<n>` (plus `CUDA_DEVICE_ORDER=PCI_BUS_ID`) and keep
  `device_id=0`.
- **`max_length` reserves the whole KV cache up front** — `genai_config.json` sets
  `past_present_share_buffer: true`. The artifact declares `context_length: 131072`, which on this
  geometry (32 layers × 8 KV heads × 128 head size × 2 × 2 B) would ask ~17 GB.
- **ORT's arena never releases.** Peak VRAM is set by the widest activation the process has *ever*
  allocated, not by the current request — which is why modality order matters here.
- **`nvidia_awq` cannot quantize either half of this model.** `NVAWQWeightOnlyQuantConfig`
  synthesises `input_ids`/`attention_mask`/`position_ids`/past-KV, but the vision tower takes
  `pixel_values`/`image_attention_mask`/`image_sizes` and the decoder — built `exclude_embeds=true` —
  takes `inputs_embeds` with no `input_ids`. Structural, not a tuning problem.

## Usage

```python
import onnxruntime as ort
import onnxruntime_genai as og

# The pip CUDA wheels place their DLLs outside the process search path.
ort.preload_dlls()

cfg = og.Config("Phi-4-multimodal-instruct-int8-onnx")
cfg.clear_providers()
cfg.append_provider("cuda")
cfg.set_provider_option("cuda", "device_id", "0")   # must be 0 — see gotchas
model = og.Model(cfg)

tokenizer = og.Tokenizer(model)
params = og.GeneratorParams(model)
params.set_search_options(max_length=4096, do_sample=False)   # 8192 does not fit 12 GB here

gen = og.Generator(model, params)
gen.append_tokens(tokenizer.encode(
    "<|user|><|image_1|>\nTranscribe all text in this image exactly.<|end|><|assistant|>"))
```

Images and audio are passed as `og.Images.open_bytes(...)` / `og.Audios.open_bytes(...)` to
`model.create_multimodal_processor()`, and the prompt must carry the inline markers this
architecture expects — `<|image_1|>`, `<|audio_1|>`, numbered from 1 independently per modality.
**Use one modality per process on a 12 GB card**, per the arena limit above.

## Acknowledgements

Base model © Microsoft —
[`microsoft/Phi-4-multimodal-instruct`](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)
and its ONNX release
[`microsoft/Phi-4-multimodal-instruct-onnx`](https://huggingface.co/microsoft/Phi-4-multimodal-instruct-onnx),
whose embedding and speech graphs, adapters and tokenizer ship here unchanged. Quantization tooling:
ONNX Runtime, its bundled Intel Neural Compressor GPTQ implementation, and onnxruntime-genai's model
builder.