--- license: apache-2.0 base_model: Qwen/Qwen2.5-14B-Instruct library_name: onnx tags: - onnx - onnxruntime-genai - int4 - quantized - qwen2 --- # Qwen2.5-14B-Instruct — int4 (zero-point) ONNX, external data repacked An ONNX export of Qwen2.5-14B-Instruct with int4 weight quantization (zero-point, `MatMulNBits`-style packed `UINT8` weights plus `FLOAT16` scales), published so that a weight-offload / memory-residency measurement can be reproduced on hardware other than the machine it was first run on. ## Why this exists It is the fixture for [justinchuby/onnx-genai#925](https://github.com/justinchuby/onnx-genai/issues/925), which asks whether a host-mapped ("zero-copy") weight read has an aggregate capacity ceiling on Linux. On one consumer GPU under Windows/WDDM, aggregate *distinct* host-mapped bytes read per decode step above ~0.44–0.65 GB began **silently returning stale data** — no error, just wrong tokens. Whether that ceiling is a property of the hardware or of the Windows display driver is unresolved, and it cannot be settled without re-measuring somewhere without WDDM. A model of roughly this size is required: the question is about **absolute** host-mapped bytes per step, so the weights must be large enough to push past the ~0.65 GB band. A small model cannot reach the regime under test no matter how the budget is configured. ## Contents | File | Size | Notes | |---|---|---| | `model.onnx` | 0.66 MB | graph only; all initializers external | | `model.onnx.data` | 7.76 GiB | 339 packed-int4 `UINT8` tensors (7.397 GB) + 581 `FLOAT16` scale tensors (0.933 GB) | | `inference_metadata.yaml` | 157 KB | canonical `pipeline.workflow` — see below | | `policies/*.onnx` | small | ten token-policy graphs referenced by the workflow | | `genai_config.json`, `config.json`, `tokenizer.json`, `tokenizer_config.json` | small | standard | 48 layers, 40 attention heads, 8 KV heads, head size 128, hidden 5120, vocab 152064, `context_length` 8192. ## External data was repacked — read this before comparing byte counts The original export's `model.onnx.data` was **16.652 GB, of which only 8.330 GB (50.02%) was referenced by any initializer**. The unreferenced part was a single contiguous prefix `[0, 8322547712)`: an entire superseded generation of weights left in the file, referenced by nothing. That is not harmless. It costs 2× disk and download, and 2× pinned host RAM on memory-mapped weight paths — the zero-copy path in onnx-genai registers the whole mapping and measured `host_registered_bytes = 16,652,453,888` for 8.33 GB of live weights. It also produced a wrong derived number: sizing a weight budget from file length rather than referenced extents made the model look 2.00× larger than it is ([#853](https://github.com/justinchuby/onnx-genai/issues/853), fixed in [#856](https://github.com/justinchuby/onnx-genai/pull/856)). What is published here is the repacked model: the referenced byte ranges streamed into a fresh blob with the offsets rewritten, produced by [`scripts/repack_external_data.py`](https://github.com/justinchuby/onnx-genai/blob/main/scripts/repack_external_data.py). ``` source blob : 16,652,453,888 bytes (16.652 GB) repacked : 8,330,399,744 bytes (8.330 GB) reduction : 49.97% tensors : 920 ``` **Verified rather than assumed:** the repacked model produced *exactly* the same 16 generated token IDs as the original — same prompt, same binary, same GPU — with `captures=2 fallbacks=0` and `oversubscribed=0` unchanged. Weights are bit-identical; only their file offsets moved. Which code path produced the doubled blob is still unknown and is being tracked at [onnxruntime/mobius#488](https://github.com/onnxruntime/mobius/issues/488). Do not assume other exports are affected without checking; the check is cheap: ```python import onnx, pathlib m = onnx.load("model.onnx", load_external_data=False) referenced = sum( int(dict((kv.key, kv.value) for kv in t.external_data)["length"]) for t in m.graph.initializer if t.data_location == onnx.TensorProto.EXTERNAL ) size = pathlib.Path("model.onnx.data").stat().st_size print(f"{referenced:,} referenced of {size:,} bytes ({100 * referenced / size:.2f}%)") ``` ## `inference_metadata.yaml` The metadata is a fully serialized canonical `pipeline.workflow`. It declares the decoder graph and the ten token-policy graphs under `policies/` as workflow components, threads all 48 layers' key/value cache pairs through `serving.state_service` groups, and expresses sampling, termination, and length bookkeeping as ONNX graphs. The autoregressive loop is therefore data the generic workflow runtime executes — there is no decoder-specific lowering step. There is no `model.io` section. That legacy form described a single decoder in a shape only a special-cased decoder loader could execute; it is not supported. Exports produced by onnxruntime-genai ship a `genai_config.json` but not this file. It is included here so the directory works out of the box. ## Usage ```bash hf download justinchuby/qwen2.5-14b-instruct-int4-zp-onnx --local-dir qwen14b-zp cargo build --release -p onnx-genai-bench --bin profile_native --features bench-native,cuda ./target/release/profile_native --model qwen14b-zp --ep cuda --tokens 16 --steady ``` Reference output on an RTX 4060 Laptop (8 GB, WDDM), greedy sampling off, prompt `"The capital of France is"` at 16 tokens: ``` generated_token_ids: [96347, 3375, 724, 11, 358, 2776, 14589, 311, 6723, 429, 498, 3003, 2581, 6617, 315, 752] ``` **These token IDs are a property of this model on that build; do not treat them as a cross-machine expectation.** For any measurement, establish your own baseline first and compare every arm against it. That is the gate that catches the silent-corruption failure mode above, where nothing errors and generation simply ends early with the wrong tokens. Wall-clock on that box ranged 3.9–28 tok/s across *identical* configurations, so throughput from a single run is not evidence of anything. ## License Apache-2.0, inherited from [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct). This is a quantized format conversion of that model; no weights were retrained or otherwise modified beyond quantization and the external-data repack described above. ## Annotated inference metadata Review [`inference_metadata.annotated.yaml`](./inference_metadata.annotated.yaml) for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. [`inference_metadata.yaml`](./inference_metadata.yaml) remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.