agentionai/Qwen3.8-Flash-Next-AP-GGUF

Qwen3.8-Flash-Next AP GGUF

Agention Precision (AP) quants of Qwen/Qwen3.8-Flash-Next: space-efficient, mainline llama.cpp compatible — standard quant types only, no fork needed.

Each tier uses a different quant type per tensor group rather than one type throughout: the dense backbone (attention, GDN, hyper-connections, shared experts) is held at Q6_K/Q8_0 because it is read on every token and is far more sensitive than its size suggests, while the MoE experts — where the parameters actually are — carry the compression. AP-IQ3_XXS beats AesSedai's IQ4_XS at 36 GiB smaller.

tier	folder	size	VRAM *	PPL	vs. ref	KLD **	top-1 **
AP-IQ3_XXS	AP-IQ3_XXS/	80.72 GiB	53.90 GiB	4.1566	+3.74%	0.0787	90.22%
AP-IQ4_XS	AP-IQ4_XS/	84.24 GiB	57.42 GiB	4.1572	+3.75%	—	—
AP-Q4_K_XL	AP-Q4_K_XL/	94.20 GiB	67.38 GiB	4.0536	+1.17%	—	—
AP-Q5_K_M	AP-Q5_K_M/	103.57 GiB	76.75 GiB	4.0468	+1.00%	0.0204	94.95%
AP-Q5_K_XL	AP-Q5_K_XL/	112.51 GiB	76.75 GiB	4.0339	+0.68%	ref	ref

Reference (unquantized): 4.0068 +/- 0.02271 (wikitext-2 raw, 145 chunks, -c 2048, as reported in PR 27742).

* VRAM with the n-gram table offloaded to CPU (-ot per_layer_token_embd=CPU), which is how these are normally run — see below. That table is a third of each file and is gathered, not multiplied, so keeping it in host RAM costs very little speed.

** KL-divergence and top-1 agreement are measured against AP-Q5_K_XL, not against the unquantized model (it does not fit on our hardware). They rank these tiers against each other correctly, but are not comparable to KLD figures published elsewhere.

Which one

• AP-Q5_K_XL unless you are short of disk. It has the same VRAM cost as AP-Q5_K_M — the extra 8.94 GiB is the higher-precision n-gram table, which lives in host RAM — and it is meaningfully better: dropping to AP-Q5_K_M changes the top token on 1 prediction in 20.
• AP-Q4_K_XL is the value pick, within ~1.2% of reference at 94 GiB.
• AP-IQ3_XXS is the smallest that still holds up, and supersedes AP-IQ4_XS (same perplexity within noise, 3.5 GiB smaller). AP-IQ4_XS is kept for anyone already depending on it.

A note on perplexity: three of these tiers sit inside one standard error of each other on PPL, so PPL alone cannot rank them — the KLD column can, and it is what the recommendation above is based on.

Run

Stock llama.cpp — no fork, no patches.

llama-server -m AP-Q5_K_XL/Qwen3.8-Flash-Next-AP-Q5_K_XL.gguf -ngl 99 -c 32768

Offloading the n-gram table (recommended)

per_layer_token_embd is a 51.2 B-parameter hash-embedding table — a third of the file. It is only ever gathered from, never multiplied, so putting it in host RAM costs little speed and takes it entirely out of your VRAM budget:

llama-server -m AP-Q5_K_XL/Qwen3.8-Flash-Next-AP-Q5_K_XL.gguf \
  -ngl 99 -c 32768 \
  --load-mode mmap --no-op-offload \
  --override-tensor "per_layer_token_embd=CPU"

That is what the VRAM column above reports. Measured cost on a Radeon 8060S: prompt processing 282 vs 285 t/s and generation 24.4 vs 25.4 t/s against a smaller tier held fully on the GPU — i.e. a few percent, in exchange for 36 GiB of VRAM.

--load-mode mmap is required if the file is larger than RAM + VRAM.

Faster at long context on our fork (optional)

These files are built for stock llama.cpp and that is how most people should run them. If you are on Vulkan and work at long context, our fork (LaurentZuijdwijk/llama.cpp, branch vulkan/qwen4exp-rocmfpx) runs the same file considerably faster as the context grows. The branch tracks upstream: it carries ggml-org/master merged at 0190529ec (30 Aug 2026), so you get upstream's qwen4exp work — including its graph-split reduction, worth 3-9% on prefill here — on top of the changes below.

AP-IQ4_XS, Radeon 8060S / Vulkan, llama-bench, q8_0 KV, identical weights on both sides:

depth	prefill (fork)	prefill (stock)	generation (fork)	generation (stock)
512	315.9	245.6	25.05	26.39
2048	302.8	219.7	24.75	22.70
8192	275.1	153.0	25.07	18.67
16384	232.4	107.4	22.91	16.22
32768	194.2	70.6	22.46	11.85

Stock decays 55% in generation from 512 to 32k; the fork decays 10%. Two causes, both fixed on the fork: the sparse-attention indexer asks for a top-k larger than a single Vulkan workgroup can serve, so past ~1024 cached tokens the op silently falls back to a single-threaded CPU path once per QSA layer per token; and the indexer recomputed its block summary keys over the whole context every step instead of caching them.

The fork also accepts --ngram-on-disk, which reads the table's rows straight from the file instead of holding them in host RAM at all: 70.5 GiB VRAM and ~0 host for AP-Q4_K_XL, versus ~23 GiB resident with the -ot approach above.

Sampling

Qwen's recommended settings:

parameter	thinking mode	instruct (non-thinking)
temperature	1.0	0.7
top_p	0.95	0.80
top_k	20	20
min_p	0.0	0.0
presence_penalty	0.0	1.5
repetition_penalty	1.0	1.0

Credits

qwen4exp support is the work of Daniel Han (@danielhanchen), from ggml-org/llama.cpp#27742, merged upstream. Calibration corpora from bartowski and Thireus. Base model by the Qwen team.

Quantized and published by Agention.

License

Qwen Community License 1.0, included as LICENSE.