justinchuby/qwen3-0.6b-onnx-genai

Qwen3-0.6B for onnx-genai

An optimized CPU ONNX package that runs directly with the nxrt Python API. It demonstrates onnx-genai's canonical inference_metadata.yaml format, which is the authoritative contract in this package. genai_config.json is retained only for ecosystem compatibility.

The metadata is a fully serialized pipeline.workflow. The autoregressive loop is data, not runtime-specific code: the decoder graph and the ten token-policy graphs under policies/ are declared as workflow components, all 28 key/value cache pairs are threaded through serving.state_service groups, and sampling, termination, and length bookkeeping are ONNX graphs the runtime executes like any other component. The package therefore runs on the generic workflow runtime with no decoder-specific lowering step.

There is no model.io section. That legacy form described a single decoder in a shape only a special-cased decoder loader could execute; it is not supported.

Run

This package requires an nxrt development build from onnx-genai main at or after session continuation support (c1d09e1b). Published PyPI builds through 0.1.0.dev4 predate the generic workflow runtime and cannot load this package.

# After building and installing nxrt from a compatible onnx-genai checkout:
ONNX_GENAI_KV_MAX_LEN=128 python3 - <<'PY'
import nxrt

engine = nxrt.genai.Engine.from_dir(".")
result = engine.generate(
    "Answer in one concise sentence: What is the Rust programming language?\nAnswer:",
    max_tokens=18,
    temperature=0.0,
)
print(result.text)
print(result.token_ids)
print(result.finish_reason)
PY

Example CPU output:

 The Rust programming language is a systems programming language that is known for its safety and performance.
[576, 33789, 15473, 4128, 374, 264, 5942, 15473, 4128, 429, 374, 3881, 369, 1181, 7149, 323, 5068, 382]
max_tokens

ONNX_GENAI_KV_MAX_LEN limits the shared KV allocation for this short example; the model metadata preserves the full 40,960-token context limit.

Multi-turn

The workflow declares a conversation state cell with scope: session and a session.continuation naming the prompt input it rejoins, so a session carries the conversation across turns: each turn's prompt is everything the session has heard followed by the caller's tokens, and the cell then absorbs both that prompt and the tokens the turn published. Independent sessions are isolated, and resetting or closing one releases what it held.

session = engine.create_session()
engine.generate_in_session(session, "My name is Ada. Remember it.", max_tokens=8)
engine.generate_in_session(session, " What is my name?", max_tokens=8)

Without this declaration a runtime executing the package as written has nothing to carry, and every turn restarts from its own prompt. A request with no session is unaffected: a session holding nothing contributes nothing, so a conversation's first turn and a stateless generation are the same execution.

Attribution and license

Derived from Qwen/Qwen3-0.6B. Qwen3 is distributed under the Apache License 2.0. See LICENSE.

Annotated inference metadata

Review inference_metadata.annotated.yaml for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. inference_metadata.yaml remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.