Kindadodgy/qwen38-256k-on-16gb

Qwen3.8-27B at 256k context on a 16 GB GPU

A measured, quality-gated configuration of Qwen3.8-27B (27B params, 262k native context) that runs a full 256k allocation on a single 16 GB AMD RX 9070 XT (Windows, llama.cpp Vulkan), with a reproducible recipe and every number backed by a result file.

This is a research artifact, not an official Unsloth file. It is the "pareto" quant: 2.72 BPW, single-quantized from the verified Unsloth BF16 base, with q2_K applied only to the 99 FFN gate/up tensors (the integer-dot MMVQ fast path) and iq2_xxs elsewhere.

What	Number	How it was measured
Gate speed (256k allocated, 40k resident)	56.6 t/s (4th-best of 5)	GitHub repo: results/pareto-screen-c1..c5.json
Sustained 20-min floor	58.5 t/s	GitHub repo: results/sustained-pareto256.json
200k resident (selective-skip, MTP on)	~42-46 t/s (46.16 measured, 44.6 fresh boot, ~42 sustained)	GitHub repo: results/bench-r6l2-mh-K30.json, bench-r6op-mh.json
200k stock (no skip)	~33 t/s	GitHub repo: results/bench-r6l2-mh-STOCK.json
Quality at 200k (depth-GSM8K, selection active)	29/30 (96.7%)	GitHub repo: results/bench-depth-gsm8k.json
Long-context retrieval	PASS at 200k (multi-hop 3/3, smoke-scale)	GitHub repo: results/bench-r6l2-mh-K30.json
llama-bench raw curve (MTP off)	45.3@4k -> 28.3@200k	GitHub repo: results/bench-lb-depth-*.json
Model footprint	8.66 GiB, 2.72 BPW	configs/pareto-map.txt

Two number regimes (read this before quoting)

• Server path, MTP on (how the model is actually used): the headline numbers above. Speculative decoding (MTP, n=2) amplifies throughput ~2.4-2.6x; the selective-attention tile-skip (GGML_VK_FA_SELECT_KEEP=30) is measured only on this path.
• llama-bench, MTP off (community raw-decode curve): 45.3@4k to 28.3@200k, identical with and without the skip (45.2/43.5/41.5/37.8/ 32.4/28.3). The skip is neutral on the raw path; I do not claim a raw-decode win.

How to run

# 1. download the blob (8.66 GiB) and verify
curl -L -o pareto-bf16.gguf https://huggingface.co/Kindadodgy/qwen38-256k-on-16gb/resolve/main/pareto-bf16.gguf
echo "57DDA505ECD2B0731341C002FF03EDE1526740DC899E6F840AE1DFB7F1B3FA81  pareto-bf16.gguf" | sha256sum -c -

Path A: stock llama.cpp (works everywhere, no custom build)

llama-server --model pareto-bf16.gguf \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.60 \
  --n-gpu-layers 99 --host 127.0.0.1 --port 8080 --ctx-size 262144 \
  -fa on -ctkd q4_0 -ctvd q4_0

This runs on any recent llama.cpp (b10537-era or newer; MTP support required). It is a good daily driver: 18-24 t/s on short contexts with the MTP boost, full 256k allocation, quality gates intact. The selective-skip is NOT active here (it needs the custom build), so at deep context you get stock speeds (33 t/s at 200k).

Path B: the selective-skip build (the 42-46 t/s number)

export GGML_VK_FA_SELECT_KEEP=30
export GGML_VK_FA_SELECT_MINKV=32768
llama-server --model pareto-bf16.gguf \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.60 \
  --n-gpu-layers 99 --host 127.0.0.1 --port 8080 --ctx-size 262144 \
  -fa on -ctkd q4_0 -ctvd q4_0

Requires a llama.cpp Vulkan build from the b10537-era (commit bf0040e15) with the selective-attention selector. The selector is an environment variable, not a compile flag: unset it for dense attention. The exact mechanism is documented in the GitHub repo's patches/ (reconstructed from the build notes; the build-machine diff was not archived). This path gives ~42-46 t/s at 200k resident with the quality gates intact.

File integrity

• SHA256 of pareto-bf16.gguf: 57DDA505ECD2B0731341C002FF03EDE1526740DC899E6F840AE1DFB7F1B3FA81
• Rebuild recipe: unsloth BF16 base (LFS oids verified 3/3) + imatrix_unsloth.gguf + the tensor map (configs/pareto-map.txt).

Honest caveats

• One rig, single GPU, n=1 sessions; not leaderboard-comparable. RULER pending (needs Python tooling).
• 50+ t/s is at working depth (<=64k resident). At 200k the honest number is 42-46 with the skip (33 stock); at full 250k resident ~27-32, limited by an upstream Vulkan attention-bandwidth issue (#26663 class).
• Multi-hop N=3 is a smoke test, not a certification; larger-N pending.
• 2-bit is a real quality cliff; quality is held by localizing the damage (q2_K only on FFN gate/up), not by pretending it is lossless.
• The rig has a bistable fast/slow driver state (~1.36x spread) and ~20% sustained drift; all comparisons are same-day, interleaved, cache-hit.
• Base GSM8K (144/150 at K30) ran selection-inactive; depth-GSM8K 29/30 at 200k is the selection-active gate.

Full data and recipe

• GitHub repo (all receipts, quant map, launch flags, repro scripts, paper draft): https://github.com/7269827-rgb/qwen38-256k-on-16gb
• Every number in the table above has its raw result JSON in the GitHub repo's results/ directory. This HF repo hosts the model + card + configs; the receipts live on GitHub (they are the source of record).

A note from the author

This is my only GPU, and I am still paying it off. It is also my school and everything rig, so I could not go as deep into this as I wanted. I left all the data and receipts so people can improve on this and do not waste time on ground I already covered. I hope this helps people who cannot afford a big rig, and I welcome feedback and replication. If something here is wrong, show me the receipt and I will fix it.

Credits

Model: Qwen3.8-27B (Apache-2.0) and the Unsloth GGUF pipeline (BF16 base, imatrix, MTP heads). Tooling: llama.cpp. The selective-attention tile-skip builds on the Chinese open-source selective-attention research lineage (Quest/NSA-style). This work was produced with human oversight plus AI assistants by role: DeepSeek-flash (execution), Claude and Opus (analysis), and an anonymous advisor (pre-registered decision logic).

License

• Model weights: derived from Qwen3.8-27B, Apache-2.0.
• Quant recipe and config: MIT (see GitHub LICENSE).
• Paper text and figures: CC-BY-4.0 (see paper/LICENSE).
• This GGUF is a research artifact; it is not an official Unsloth file.