README.md

metadata

license: other
license_name: qwen-community-1.0
license_link: LICENSE
base_model:
  - Qwen/Qwen3.8-Flash-Next
pipeline_tag: image-text-to-text
tags:
  - radiance
  - rocm
  - amd
  - rdna4
  - moe
  - 4-bit
  - int8
  - gptq
  - speculative-decoding
  - mtp
  - vision

Qwen3.8-Flash-Next, 4-bit codebook experts + int8 trunk — radiance container

Qwen/Qwen3.8-Flash-Next quantised from its bf16 checkpoint into a single .rad container for the radiance inference engine (AMD RDNA4, ROCm), with the model's MTP head for speculative decoding and its vision tower.

File	qwen3.8-next-flash-fp8-iq4r-moe.rad — 113.6 GiB
Routed experts	4-bit codes into a 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation, one E4M3 scale per 64 weights; codes chosen by GPTQ (10M-token calibration, per-expert down-projection Hessians) and refined by three sweeps of coordinate descent
Protected experts	ten experts that carry most of their layer's down-projection energy, kept bf16
Trunk	attention, delta-net and shared-expert linears and the lm_head int8 (W8A8), a scale per 128 columns searched for least error; hyper-connection mixing matrices E4M3
Speculator	the model's MTP head (depth 3 by default)
Vision	the vision tower, bf16: images and video in chat requests
Context	262,144 tokens trained; served at 200K

Quality

Full-vocabulary KL divergence against the bf16 model, teacher-forced over ~75K positions of chat, tool-use and code transcripts:

	mean KL	median	p99	p99.9	top-1 agreement
all tokens	0.0789	0.0037	1.65	5.33	92.85%
text and assistant turns	0.0239	0.0029	0.32	1.03	94.57%
reference: a second bf16 implementation, all tokens	0.0399	0.0013	0.82	3.43	95.13%

Serve

The routed experts do not fit two 32 GB cards; the engine keeps the hottest in VRAM and streams the rest from a pinned host pool, so the host needs about 32 GB of free RAM.

radiance --model qwen3.8-next-flash-fp8-iq4r-moe.rad --tp 2 --tp-wire wht6 \
    --max-model-len 200000 --max-num-seqs 8 --kv-cache-dtype fp8 \
    --placement expert_tiered --host-pool-mib 12288 --gpu-headroom-mib 96 \
    --expert-vs-cache-ratio 0.82 --num-speculative-tokens 3 --max-num-batched-tokens 2048 \
    --host 0.0.0.0 --port 8000

On 2× Radeon AI PRO R9700 (gfx1201), one stream: about 15 ms per decode step at ~2.4 accepted tokens per step, and ~6,400 tokens/s prefill on a 26K-token prompt.

The server speaks the OpenAI API (/v1/chat/completions, /v1/completions), with tool calls, structured output, and image_url / video parts in chat messages.

How this file was made

The exact command, run on 2026-10-03 from a radiance build directory of the commit series merged to radiance's main branch as 2531f14 (radiance_home is that build's plugin directory):

CALIB=~/models/calib/w4nl-calib OMP_NUM_THREADS=12 nice -n 10 ./bin/rad-convert \
    ~/models/Qwen/Qwen3.8-Flash-Next --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe \
    --home radiance_home --tokenizer ~/models/Qwen/Qwen3.8-Flash-Next/tokenizer.json \
    --reuse ~/models/rad/w4nl-i8lm.rad -o ~/models/rad/w4nl64-i8lm.rad -v

• --reuse names an earlier container converted from the same checkpoint by an intermediate recipe. Every weight it already held under the same rule -- the int8 trunk and lm_head, the E4M3 connection and MTP matrices, the ten bf16 experts -- was copied from it; the routed experts were quantised by this run. Without --reuse the same command quantises those weights from the checkpoint by the same rules.
• CALIB is the calibration directory the GPTQ rule reads: each layer's routed-expert input Gram from ten million tokens of the bf16 model, and each expert's own down-projection Gram from one million, recorded with the engine's calibration mode on a mixed chat, tool-use and code corpus. It is not published.
• The published file is that output under this repository's name, with the local directory removed from the created by, recipe and calibration-path strings in its header; the weights are the converter's bytes.

The recipe file the command names is in this repository as qwen4exp-w4nl64-i8-hc8m.recipe, with its comments; its $CALIB is the calibration directory. Its rules, as rad-info --recipe qwen3.8-next-flash-fp8-iq4r-moe.rad prints them; the first rule that matches a weight decides it, and everything no rule names is the checkpoint's bf16:

blk.34.ffn_*_exps.407.weight cast dtype=bf16
blk.34.ffn_*_exps.496.weight cast dtype=bf16
blk.44.ffn_*_exps.292.weight cast dtype=bf16
blk.44.ffn_*_exps.350.weight cast dtype=bf16
blk.46.ffn_*_exps.290.weight cast dtype=bf16
blk.46.ffn_*_exps.392.weight cast dtype=bf16
blk.47.ffn_*_exps.122.weight cast dtype=bf16
blk.47.ffn_*_exps.143.weight cast dtype=bf16
blk.47.ffn_*_exps.399.weight cast dtype=bf16
blk.47.ffn_*_exps.445.weight cast dtype=bf16
blk.*.ffn_*_exps.*.weight gptq table=w4nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=calib/w4nl-calib
*_hc_down.weight rtn codes=fp8_e4m3 group=128 scale=f32
*_hc_up.weight rtn codes=fp8_e4m3 group=80 scale=f32
mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32
mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32
blk.*.attn_qg.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.attn_k.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.attn_v.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.attn_output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ffn_gate_up_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ffn_down_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16

License

The Qwen community license of the base model; see LICENSE.