justinchuby/qwen2.5-14b-instruct-int4-zp-onnx

Qwen2.5-14B-Instruct — int4 (zero-point) ONNX, external data repacked

An ONNX export of Qwen2.5-14B-Instruct with int4 weight quantization (zero-point, MatMulNBits-style packed UINT8 weights plus FLOAT16 scales), published so that a weight-offload / memory-residency measurement can be reproduced on hardware other than the machine it was first run on.

Why this exists

It is the fixture for justinchuby/onnx-genai#925, which asks whether a host-mapped ("zero-copy") weight read has an aggregate capacity ceiling on Linux. On one consumer GPU under Windows/WDDM, aggregate distinct host-mapped bytes read per decode step above ~0.44–0.65 GB began silently returning stale data — no error, just wrong tokens. Whether that ceiling is a property of the hardware or of the Windows display driver is unresolved, and it cannot be settled without re-measuring somewhere without WDDM.

A model of roughly this size is required: the question is about absolute host-mapped bytes per step, so the weights must be large enough to push past the ~0.65 GB band. A small model cannot reach the regime under test no matter how the budget is configured.

Contents

File	Size	Notes
model.onnx	0.66 MB	graph only; all initializers external
model.onnx.data	7.76 GiB	339 packed-int4 UINT8 tensors (7.397 GB) + 581 FLOAT16 scale tensors (0.933 GB)
inference_metadata.yaml	157 KB	canonical pipeline.workflow — see below
policies/*.onnx	small	ten token-policy graphs referenced by the workflow
genai_config.json, config.json, tokenizer.json, tokenizer_config.json	small	standard

48 layers, 40 attention heads, 8 KV heads, head size 128, hidden 5120, vocab 152064, context_length 8192.

External data was repacked — read this before comparing byte counts

The original export's model.onnx.data was 16.652 GB, of which only 8.330 GB (50.02%) was referenced by any initializer. The unreferenced part was a single contiguous prefix [0, 8322547712): an entire superseded generation of weights left in the file, referenced by nothing.

That is not harmless. It costs 2× disk and download, and 2× pinned host RAM on memory-mapped weight paths — the zero-copy path in onnx-genai registers the whole mapping and measured host_registered_bytes = 16,652,453,888 for 8.33 GB of live weights. It also produced a wrong derived number: sizing a weight budget from file length rather than referenced extents made the model look 2.00× larger than it is (#853, fixed in #856).

What is published here is the repacked model: the referenced byte ranges streamed into a fresh blob with the offsets rewritten, produced by scripts/repack_external_data.py.

source blob : 16,652,453,888 bytes (16.652 GB)
repacked    :  8,330,399,744 bytes (8.330 GB)
reduction   : 49.97%
tensors     : 920

Verified rather than assumed: the repacked model produced exactly the same 16 generated token IDs as the original — same prompt, same binary, same GPU — with captures=2 fallbacks=0 and oversubscribed=0 unchanged. Weights are bit-identical; only their file offsets moved.

Which code path produced the doubled blob is still unknown and is being tracked at onnxruntime/mobius#488. Do not assume other exports are affected without checking; the check is cheap:

import onnx, pathlib
m = onnx.load("model.onnx", load_external_data=False)
referenced = sum(
    int(dict((kv.key, kv.value) for kv in t.external_data)["length"])
    for t in m.graph.initializer
    if t.data_location == onnx.TensorProto.EXTERNAL
)
size = pathlib.Path("model.onnx.data").stat().st_size
print(f"{referenced:,} referenced of {size:,} bytes ({100 * referenced / size:.2f}%)")

inference_metadata.yaml

The metadata is a fully serialized canonical pipeline.workflow. It declares the decoder graph and the ten token-policy graphs under policies/ as workflow components, threads all 48 layers' key/value cache pairs through serving.state_service groups, and expresses sampling, termination, and length bookkeeping as ONNX graphs. The autoregressive loop is therefore data the generic workflow runtime executes — there is no decoder-specific lowering step.

There is no model.io section. That legacy form described a single decoder in a shape only a special-cased decoder loader could execute; it is not supported.

Exports produced by onnxruntime-genai ship a genai_config.json but not this file. It is included here so the directory works out of the box.

Usage

hf download justinchuby/qwen2.5-14b-instruct-int4-zp-onnx --local-dir qwen14b-zp

cargo build --release -p onnx-genai-bench --bin profile_native --features bench-native,cuda
./target/release/profile_native --model qwen14b-zp --ep cuda --tokens 16 --steady

Reference output on an RTX 4060 Laptop (8 GB, WDDM), greedy sampling off, prompt "The capital of France is" at 16 tokens:

generated_token_ids: [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752]

These token IDs are a property of this model on that build; do not treat them as a cross-machine expectation. For any measurement, establish your own baseline first and compare every arm against it. That is the gate that catches the silent-corruption failure mode above, where nothing errors and generation simply ends early with the wrong tokens.

Wall-clock on that box ranged 3.9–28 tok/s across identical configurations, so throughput from a single run is not evidence of anything.

License

Apache-2.0, inherited from Qwen/Qwen2.5-14B-Instruct. This is a quantized format conversion of that model; no weights were retrained or otherwise modified beyond quantization and the external-data repack described above.

Annotated inference metadata

Review inference_metadata.annotated.yaml for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. inference_metadata.yaml remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.