README.md

---
license: other
license_name: qwen-community-license-1.0
base_model: Qwen/Qwen3.8-Flash-Next-FP8
library: gguf
quantized_by: julianmb
pipeline_tag: text-generation
tags:
- qwen4exp
- strix-halo
- rocmfpx
- gguf
- ple-quantized
---

# Qwen3.8-Flash-Next GGUFs — provenance-verified quants for Strix Halo

Four files:

| file | size | what it is |
|---|---|---|
| `Qwen3.8-Flash-Next-IQ4_XS-PLE.gguf` | 91 GiB | **recommended daily driver** — iq4_xs trunk with the 27G PLE n-gram table at iq4_nl |
| `Qwen3.8-Flash-Next-IQ4_XS.gguf` | 116 GiB | static reference quant, PLE at q8_0 |
| `Qwen3.8-Flash-Next-IQ4_XS-M2.gguf` | 115 GiB | **imatrix-calibrated quant, PLE table at q8_0 — best measured perplexity** |
| `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` | 3.9 GiB | MTP draft sidecar for nathanw1014-lineage engines (fork-specific — will NOT load on apepojken/mainline) |

## M2 — the imatrix quant

second-generation quant: same trunk type (iq4_xs), but calibrated with a
926-entry imatrix (1,024 chunks) via the ROCmFPX banded quantizer, and the
51B PLE lookup table **left at q8_0** (no `--tensor-type` cut). 5.56 bpw,
115 giB — 24 giB bigger than the 91g PLE file.

perplexity (wiki.test.raw, ctx 2048, 145 chunks):

| quant | PPL |
|---|---|
| **M2 (imatrix, PLE q8_0)** | **4.2809 ±0.025** |
| PLE 91g | 4.2932 ±0.025 (statistically tied, <0.5σ) |
| static 116g | 4.5221 ±0.026 (~9σ worse) |

speed profile is honest-mixed (single runs, nathanw1014 vulkan engine,
q8_0 kv, temp 0):

| depth | plain tg | mtp tg |
|---|---|---|
| 8k | 23.8 (≈ PLE 23.8) | 25.1 (PLE 33.5 — M2 slower) |
| 32k | 19.0 (≈ PLE 19.0) | **29.1** (best of the three) |
| 128k | — | 13.3 (PLE 13.5 — tied) |

pick M2 when you want the **best measured quality** and don't mind the
24 giB: at ≤32k plain it matches PLE, and at 32k MTP it measured fastest.
for shallow-depth MTP speed take the 91g PLE; for deep 128k+ MTP the
static 116g had a small in-sweep edge (17.4 vs 13.3/13.5 — within the
same-config spread, n=1 caveat).

## provenance

every quant descends from an F16 that was byte-verified against the official
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint: hyper-connection norms folded to
(1 + w) (97/97 tensors — the converter bug that produces deterministic garbage
is fixed in our pipeline), PLE fp8 scale applied, expert stacking identity
probed 512x3, GDN v-head reorder checked. details:
https://github.com/julianmb/haloq38flash

## the PLE cut (what makes the 91G special)

the 51B-parameter n-gram lookup table was moved from q8_0 (54G) to iq4_nl
(27G) via --tensor-type. hash-gathered lookup rows tolerate the precision
drop — verified by smoke and full benchmark, no degradation observed:

| depth | static 116G plain/mtp t/s | **PLE 91G plain/mtp t/s** |
|---|---|---|
| 0 | 29.2 / 48.4 | 29.9 / **53.1** |
| 8k | 22.9 / 42.8 | 24.1 / **56.4** |
| 32k | 19.5 / 29.5 | 20.1 / **30.2** |

prefill at 32k: 384 → 397 t/s. no collapse at depth. engine: nathanw1014
strix-halo-vulkan (ad914eb), vulkan/radv, q8_0 KV, -ub 2048, temp 0.

## fork compatibility caveat (important)

the iq4_nl PLE rows assert in SOME forks: engines that feed gathered PLE
rows directly as mul_mat B operands without dequantizing (apepojken
qwen4exp-spec-mtp) abort at ggml-vulkan.cpp:7794 (b_type must be
F32/F16/Q8_1). verified working on nathanw1014 strix-halo-vulkan. if your
engine asserts on load or first token, use the 116G static file instead.
M2 keeps the PLE table at q8_0 and has no such assert exposure.

## provenance note

the same --tensor-type cut applied to unverified-source quants will NOT fix
a broken converter (hc norms, PLE scale) — garbage in, garbage out. ours is
built from a fixed, audited pipeline.

## 128k+ context caveat (measured)

the depth story is not monotonic. measured on the same engine (nathanw1014
vulkan, q8_0 kv, temp 0, single runs):

| depth | static 116g plain/mtp t/s | PLE 91g plain/mtp t/s |
|---|---|---|
| 0 | 29.2 / 48.4 | 29.9 / 53.1 |
| 8k | 22.9 / 42.8 | 24.1 / 56.4 |
| 32k | 19.5 / 29.5 | 20.1 / 30.2 |
| 128k | 10.8 / 26.9 | 11.0 / **18.6** |

- plain decode collapses with depth on both quants (~11 t/s at 128k) — the
  cost is context-mechanics (sparse-attention indexer), not the quant.
- with mtp at 128k the PLE quant measured SLOWER than static (18.6 vs 26.9,
  single runs): plausible draft-acceptance drop from ple quantization noise
  compounding over deep n-gram history. unverified mechanism, n=1 caveat.
- practical: <=32k work → PLE file. 128k+ contexts → static file (or M2 for
  the best quality at plain speed).

## run it (strix halo, 128 GB unified memory)

full methodology, receipts, and the benchmark record:
**https://github.com/julianmb/haloq38flash**

docker (one-liner; image default serves the 91g PLE on :8080):

```bash
git clone https://github.com/julianmb/haloq38flash && cd haloq38flash
docker compose up --build
```

point it at M2 with the MTP sidecar and the warm-turn cache:

```bash
docker compose run qwen38-flash-next /app/llama-server \
  -m /models/Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md /models/mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --cache-ram 8192 --ctx-checkpoints 32 \
  -c 32768 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4
```

or raw llama.cpp (nathanw1014 strix-halo-vulkan lineage engines):

```bash
llama-server -m Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4 -c 32768
```

perf notes: `-t 16` lifts prefill up to +43% at 128k (decode indifferent);
the `--cache-ram`/`--ctx-checkpoints` warm-turn flags make repeated
context nearly free (measured 438 s → 0.68 s at 128k, 994 s → 0.74 s at
256k — 640x/1351x). swap M2's filename for the other quants; same flags.

## license

qwen community license 1.0 (distribution permitted with notice; maas
restrictions apply). base model: Qwen/Qwen3.8-Flash-Next.