| # qwen4exp-w4nl64-i8-hc8m.recipe -- Qwen3.8-Flash-Next from its bf16 checkpoint as it is served | |
| # (the w4nl64-i8lm.rad container): the routed experts as rotated u4 codes into the w4nl | |
| # codebook with one E4M3 scale a (row, 64 of K) under a fixed f32 of 2^-13 (libr4d's w4nl64a8h), | |
| # the codes chosen by GPTQ and three sweeps of coordinate descent; ten protected experts plain | |
| # bf16; the trunk's block linears and the lm_head int8, served W8A8; the connections' mixing | |
| # matrices and the MTP head's fc matrices E4M3; the n-gram table at one fixed E4M3 scale. | |
| # | |
| # Measured with the engine's KL mode against the bf16 model (docs/MOE-W4.md, "What is served"): | |
| # mean KL 0.0789 and top-1 agreement 92.85% over the whole corpus, where a second bf16 | |
| # implementation scores 0.0399 / 95.13%; 0.0239 / 94.57% on text and assistant turns. | |
| # | |
| # CALIB=DIR rad-convert <Qwen3.8-Flash-Next checkpoint> \ | |
| # --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe -o w4nl64-i8lm.rad | |
| # | |
| # DIR holds each layer's pooled Grams, recorded from ten million tokens of the bf16 model, and each | |
| # expert's own down Gram, from one million tokens with the per-expert tap, side by side; libquant | |
| # takes an expert's own wherever it holds at least 4096 rows and the layer's otherwise | |
| # (docs/MOE-W4.md, "Calibration"). A variant of this recipe converted with --reuse w4nl64-i8lm.rad | |
| # re-quantises only the weights whose rule changed. | |
| # | |
| # Everything not named here is served as the checkpoint holds it; the first rule that matches a | |
| # weight decides it. | |
| # The protected experts, from a 1M-token recording of the bf16 model on the calibration corpus: | |
| # an expert whose share of its layer's down-input energy is at least 3% and at least three times its | |
| # share of the routing, and on each rank's half of the layer an even count -- the four-bit GEMM takes | |
| # the rest as two equal tables by parity -- by adding that half's next most energetic expert | |
| # (407/496, 350/292, 290/392 and 445,143/399,122 below). Layer 47's expert 445 alone carries 64% of | |
| # its layer's down-input energy from 0.4% of the routing. | |
| blk.34.ffn_*_exps.407.weight cast dtype=bf16 | |
| blk.34.ffn_*_exps.496.weight cast dtype=bf16 | |
| blk.44.ffn_*_exps.292.weight cast dtype=bf16 | |
| blk.44.ffn_*_exps.350.weight cast dtype=bf16 | |
| blk.46.ffn_*_exps.290.weight cast dtype=bf16 | |
| blk.46.ffn_*_exps.392.weight cast dtype=bf16 | |
| blk.47.ffn_*_exps.122.weight cast dtype=bf16 | |
| blk.47.ffn_*_exps.143.weight cast dtype=bf16 | |
| blk.47.ffn_*_exps.399.weight cast dtype=bf16 | |
| blk.47.ffn_*_exps.445.weight cast dtype=bf16 | |
| # Every other routed expert, trunk and MTP layer alike: u4 codes into the w4nl table, rotated by | |
| # the 128-wide Hadamard, an E4M3 scale a group of 64 under the fixed 2^-13 searched for least error | |
| # as stored, the codes chosen by GPTQ against $CALIB and then swept three times by coordinate | |
| # descent. The experts' scales sit between 2^-15 and 2^-10.6, inside E4M3's normal range under | |
| # 2^-13 with four binades either side; one past its top would be held there and said on stderr. | |
| # The MTP layer has no recording and is rounded to nearest, which libquant says on stderr. | |
| blk.*.ffn_*_exps.*.weight gptq table=w4nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB | |
| # Every connection's two mixing matrices -- the 96 trunk connections, the final mixer and the MTP | |
| # head's three -- as E4M3 rows with an f32 scale a group of each row. A 128x128 block cannot cover | |
| # them: lowrank is 320, 2.5 blocks. Down is [320, 10240] and takes a scale every 128 columns; up is | |
| # [10240, 320] and takes four scales a row, 80 columns each. | |
| *_hc_down.weight rtn codes=fp8_e4m3 group=128 scale=f32 | |
| *_hc_up.weight rtn codes=fp8_e4m3 group=80 scale=f32 | |
| # The MTP head's fc_hidden and fc_embedding, the same rows. They only ever change the drafts. | |
| mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32 | |
| mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32 | |
| # The trunk's block linears as int8, a scale per 128 columns of a row searched for least error. | |
| # The QSA indexer's projection is not one of them: its top-k is a hard choice between blocks, and | |
| # keeping it as the checkpoint holds it costs 20 MB. | |
| blk.*.attn_qg.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.attn_k.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.attn_v.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.attn_output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.ffn_gate_up_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| blk.*.ffn_down_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| # The lm_head, int8 with a bf16 scale a row per 128 columns (logits_gemm_i8): at E4M3 it costs | |
| # ~0.6 points of top-1 on its own. | |
| output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search | |
| # The n-gram table at one fixed E4M3 scale; qwen4exp-w4.recipe says where the value comes from. | |
| blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4 | |
| # The MTP head's 2-bit draft head. | |
| mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16 | |