jarvis-pet/Phi-4-multimodal-instruct-gptq-int4-onnx

Phi-4-multimodal-instruct — ONNX GPTQ INT4 (calibrated, both towers)

A GPTQ-calibrated INT4 rebuild of the vision tower and the language decoder of microsoft/Phi-4-multimodal-instruct-onnx, for onnxruntime-genai on the CUDA execution provider. 5.6 B parameters: a 3.8 B Phi-4-mini language backbone, a ~0.45 B SigLIP vision encoder, a ~0.46 B conformer speech encoder, and the modality LoRA adapters.

Why this exists. The published GPU artifact quantizes both towers with INT4 RTN, and on the vision tower that is destructive: measured against the FP16 tower on five page fixtures, its image_features have a relative L2 error of 0.94–1.09 — the error equals the signal magnitude — and cosine similarity of only 0.41–0.59. In practice OCR either degenerates into repetition ("A C M E E L L O G S S T T T T…") or confabulates prose unrelated to the image. Its INT4 decoder separately breaks EOS, so page transcription never terminates. Replacing RTN with data-aware GPTQ on both components fixes all of it while staying at 4 bits.

Sibling build. jarvis-pet/Phi-4-multimodal-instruct-int8-onnx pairs the same GPTQ INT4 vision tower with an INT8 decoder: 1.99 GiB larger, capped at max_length = 4096 and one modality per process on a 12 GB card, but byte-exact on the short strings this build occasionally drops a character from. Pick that one when exactness matters more than headroom; pick this one for the window and for image+audio in a single process.

Released under the MIT License, inherited from the base model microsoft/Phi-4-multimodal-instruct. This is a derivative quantization only; all model rights and terms follow the base model.

Files

19 files, 4.79 GiB (5.14 GB) total — byte-for-byte the same footprint as the published INT4-RTN artifact (also 4.79 GiB), and 1.99 GiB smaller than the INT8-decoder build that was the previous working configuration (6.78 GiB). Same size as Microsoft's INT4 release, with OCR that works:

Component	Precision	Notes
phi-4-mm-text.onnx{,.data}	INT4 GPTQ, group 32	the decoder, 2.43 GB — 673 MatMulNBits
phi-4-mm-vision.onnx{,.data}	INT4 GPTQ, block 128	vision tower, 370 MB — 80 MatMulNBits
phi-4-mm-speech.onnx{,.data}	as published	conformer speech encoder, 520 MB
phi-4-mm-embedding.onnx{,.data}	as published	shared embedding graph, 1.23 GB
phi-4-mm-vision.onnx_adapter, phi-4-mm-speech.onnx_adapter	as published	present but not referenced — see below
genai_config.json, vision_processor.json, speech_processor.json	—	genai_config.json differs from the published one
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json	—	unchanged

The adapter files are vestigial here, deliberately. The locally built decoder bakes the LoRAs in (896 lora_* initializers, 0 adapter inputs — Phi4MMModel.make_layer makes vision the baked default), so adapter_filename has been removed from the vision and speech sections of genai_config.json. Leaving it in makes the runtime bind tensors the graph does not have. The adapters ship anyway so the directory stays a drop-in replacement and provenance is visible; nothing loads them. Losing the speech LoRA was expected to cost audio quality — measured, it does not.

Audio input — 16 kHz required, 8 kHz optional

The speech encoder consumes 80-bin log-mel features at a 25 ms window and a 10 ms hop. speech_processor.json declares AudioDecoderEx.target_sample_rates: [8000, 16000], and Phi4AudioEmbed carries a complete STFT/log-mel parameter set for each rate:

	16 kHz — required / primary	8 kHz — optional
n_fft	512	256
frame_length	400 → 25 ms	200 → 25 ms
hop_length	160 → 10 ms	80 → 10 ms
n_mel	80	80

Both paths therefore emit the same feature geometry — 80 mel bins at 100 frames/second — which is what the encoder is built for; audio_compression_rate: 8 then decimates that 8× along time. What differs is bandwidth, not shape.

• Supply 16 kHz mono PCM16 WAV. This is the primary path and what every audio measurement below used.
• 8 kHz is natively supported, not merely tolerated — it is a first-class second parameter set, intended for telephony-band input. Nyquist is 4 kHz there, so the 4–8 kHz band is simply absent from the features; expect that to cost fricative and sibilant detail.
• Any other rate is resampled by AudioDecoderEx, not rejected. That path is untested here, so do not rely on its fidelity — resample to 16 kHz yourself if you care about the transcript. Note this differs from an OpenVINO serve of the same model, where nothing resamples and a wrong-rate waveform transcribes confidently wrong.

Measured results

Verified on an RTX 3060 12 GB (Windows 11, driver 610.88, no CUDA toolkit — pip wheels only), onnxruntime-genai-cuda 0.15.2 + onnxruntime-gpu 1.29.0, at max_length = 8192:

Probe	Result
Dense 850×1100 invoice, full transcription	finish_reason=stop, 190 tokens, 4.7 s, longest identical-line repeat 1, 6/6 gold fields including Subtotal 2,355.21 and routing 021000021
Short-answer extraction from the same page	exact and terminating — KALEIDOSCOPE-4471, 2,549.51, 6–8 tokens, ~1.2 s
Audio, 10 s 16 kHz clip	near-verbatim transcription, plus content Q&A and translation — speech understanding, not just ASR
Decode throughput	~64 tok/s
VRAM	6,437 MiB idle → 10,937 MiB after a dense page → 11,457 MiB after an audio request in the same process

The INT8-decoder build could not do either of the last two things on a 12 GB card: 7.4 GB of weights capped it at max_length = 4096 (11,683 MiB after a page), and because ORT's BFC arena grows and never releases, an audio request after a page request died with AllocateRawInternal Failed to allocate memory for requested buffer of size 1338028032. At 4.77 GB of weights, 8192 fits and both modalities coexist in one process.

Known limitation — short strings lose characters

This is the honest cost of 4-bit on the decoder, and it is why we do not advertise this build for OCR of isolated fields. Page transcription is unaffected (see the invoice row above), but:

• a high-contrast nonce image reading JARVIS OCR / OCRPASS-4713 comes back as "OCPASS - 4713" — the digits are right, a letter is gone. The INT8 decoder returns it byte-perfect.
• an audio clip containing "kaleidoscope" comes back as "Kalidoscope", and a test sentence gains a doubled "the the". Again byte-perfect at INT8.

If byte-exact short-field extraction matters more than VRAM headroom, use an INT8 decoder instead. Two untried levers on this: block_size=128 for the decoder (coarser groups beat finer on the vision tower, unexplained), and --nodes_to_exclude mixed precision for whichever layers carry it.

How the quantization was produced

GPTQ is Python-API-only for this: --quant_method gptq from the CLI builds a GPTQWeightOnlyQuantConfig with no calibration_data_reader, so INC iterates None and dies with TypeError: 'NoneType' object is not iterable.

• Decoder — MatMulNBitsQuantizer(bits=4, block_size=32, is_symmetric=True) with GPTQWeightOnlyQuantConfig(block_size=128, percdamp=0.1, actorder=False, mse=False, perchannel=True) over 673 MatMuls. Upstream's build_text() hardcodes precision = "int4" and ignores --precision, so the FP16 decoder it quantizes had to be built by calling create_model directly with precision="fp16" and exclude_embeds=true.
• Vision tower — the same quantizer at block_size=128, 80 MatMuls. build_vision() runs the nbits quantizer unconditionally, so the FP16 tower had to be exported with that step patched out and torch.onnx.export(..., dynamo=False) (torch ≥ 2.9's dynamo exporter fails to capture the SigLIP tower and silently rewrites opset 14 → 18).
• Calibration is MULTIMODAL, and that is load-bearing. Four page fixtures spanning 448²/672²/896² and a dense invoice, each built the way the runtime builds it — vision tower → image_features → embedding graph → inputs_embeds (sequence lengths 558/1342/1342/1672). The failure being fixed happens while generating conditioned on ~1600 positions of image embeddings, so text-only calibration would miss the distribution that actually breaks.

Fidelity of the vision tower against FP16, five page fixtures, CUDA EP:

tower	size	cosine	rel L2
INT8 block 32	536 MB	0.99990–0.99992	0.012
INT4 GPTQ block 128 (this build)	370 MB	0.950–0.961	0.29–0.33
INT4 GPTQ block 32	383 MB	0.926–0.944	0.33–0.38
INT4 RTN block 32 (published)	383 MB	0.414–0.594	0.94–1.09

Cosine 0.95 passes every end-to-end probe; 0.41 is garbage. Note that finer groups made GPTQ worse here, the opposite of the usual expectation, and blk32 is also larger.

Gotchas worth knowing

• device_id must be 0. ORT's CUDA EP creates its cudnn handle before cudaSetDevice, so any non-zero device_id dies at model load with CUDNN failure 2007: CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH. To serve on a different physical card, mask it with CUDA_VISIBLE_DEVICES=<n> (plus CUDA_DEVICE_ORDER=PCI_BUS_ID) and keep device_id=0.
• max_length reserves the whole KV cache up front — genai_config.json sets past_present_share_buffer: true. The artifact declares context_length: 131072, which on this geometry (32 layers × 8 KV heads × 128 head size × 2 × 2 B) would ask ~17 GB. 8192 is what was measured to fit 12 GB with this decoder.
• ORT's arena never releases. Peak VRAM is set by the widest activation the process has ever allocated, not by the current request.
• nvidia_awq cannot quantize either half of this model. NVAWQWeightOnlyQuantConfig synthesises input_ids/attention_mask/position_ids/past-KV, but the vision tower takes pixel_values/image_attention_mask/image_sizes and the decoder — built exclude_embeds=true — takes inputs_embeds with no input_ids. Structural, not a tuning problem.

Usage

import onnxruntime as ort
import onnxruntime_genai as og

# The pip CUDA wheels place their DLLs outside the process search path.
ort.preload_dlls()

cfg = og.Config("Phi-4-multimodal-instruct-gptq-int4-onnx")
cfg.clear_providers()
cfg.append_provider("cuda")
cfg.set_provider_option("cuda", "device_id", "0")   # must be 0 — see gotchas
model = og.Model(cfg)

tokenizer = og.Tokenizer(model)
params = og.GeneratorParams(model)
params.set_search_options(max_length=8192, do_sample=False)

gen = og.Generator(model, params)
gen.append_tokens(tokenizer.encode(
    "<|user|><|image_1|>\nTranscribe all text in this image exactly.<|end|><|assistant|>"))

Images and audio are passed as og.Images.open_bytes(...) / og.Audios.open_bytes(...) to model.create_multimodal_processor(), and the prompt must carry the inline markers this architecture expects — <|image_1|>, <|audio_1|>, numbered from 1 independently per modality.

Acknowledgements

Base model © Microsoft — microsoft/Phi-4-multimodal-instruct and its ONNX release microsoft/Phi-4-multimodal-instruct-onnx, whose embedding and speech graphs, adapters and tokenizer ship here unchanged. Quantization tooling: ONNX Runtime, its bundled Intel Neural Compressor GPTQ implementation, and onnxruntime-genai's model builder.