aweussom/Qwen3-30B-A3B-Thinking-2507-int4-ov

Qwen3-30B-A3B-Thinking-2507 — OpenVINO INT4

INT4 OpenVINO export of Qwen/Qwen3-30B-A3B-Thinking-2507, built to pair with Intel's own OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov: same architecture, same quantization recipe, so the two variants are directly comparable in an A/B — reasoning on/off is the only variable.

Why this export?

The 2507 generation split Qwen3's hybrid thinking mode into two committed models: Instruct (never thinks) and Thinking (always thinks). Intel published a pre-converted Instruct but, as of 2026-08, no Thinking counterpart. This fills that gap with the identical recipe (INT4 asymmetric, group size 128 — verified against the Instruct export's rt_info).

Export command

optimum-cli export openvino \
  --model Qwen/Qwen3-30B-A3B-Thinking-2507 \
  --weight-format int4 \
  --trust-remote-code \
  Qwen3-30B-A3B-Thinking-2507-int4-ov

Toolchain: OpenVINO 2026.1.0-21367, optimum-intel 1.27.0, transformers 4.57.6, NNCF weight compression (98% of layers int4_asym gs128, embeddings/head int8_asym per-channel). 15.2 GB on disk.

Geometry

• Qwen3MoeForCausalLM — 128 experts, 8 active per token (~3B active params)
• 48 layers, 262,144-token native context, ~96 KB/token KV cache
• A whole novel (~95k tokens) fits in one prompt with a ~9 GB KV pool

Verified (2026-08-09)

• Reproducible, independently confirmed: this conversion was run from scratch with the toolchain above, and the resulting openvino_model.bin (sha256 f6afe8bb73031387a1831dce0d987210311e47d1dbc38b3a47d642ef62697d09) is byte-identical to the independently published livrrr/Qwen3-30B-A3B-Thinking-2507-int4-ov. Two independent runs of the official export pipeline agree bit-for-bit — NNCF data-free int4 compression is deterministic — so neither repo asks for trust: verify the hash yourself.
• Integrity: weight blobs complete per the IR's own offset+size records.
• Loads and generates on CPU (Intel Core Ultra 9 285K, openvino-genai 2026.1, continuous-batching backend with prefix caching): coherent multi-step reasoning followed by a correct, concise answer. Lightly verified — one smoke generation, not a benchmark.

Notes for serving

• Output begins mid-thought: the chat template injects the opening <think> tag, so responses contain reasoning text terminated by </think>, then the answer. Parse accordingly.
• The thinking budget shares the output token cap. A small max_new_tokens can be consumed entirely by reasoning, returning a reply with no post-</think> content. Give it room (2k+ for short answers, much more for long-form tasks).
• Group-quantized INT4 targets GPU/CPU. It is not the channel-wise recipe Intel NPUs require, and this model is untested on NPU.
• On XMX-capable Intel GPUs (Arc, Lunar Lake), OpenVINO 2026.3's OFFLOAD_RATIO streams expert weights from disk to fit tight VRAM budgets; without XMX hardware that property is a silent no-op.

Serve with NoLlama

python nollama.py --model-dir Qwen3-30B-A3B-Thinking-2507-int4-ov --device CPU --cache-size-gb 12

NoLlama is an OpenAI-compatible LLM/VLM server for Intel hardware; --cache-size-gb sizes the KV pool (~12 GB holds a ~95k-token prompt for this geometry).