---
license: apache-2.0
tags:
- gguf
- mtp
- speculative-decoding
- vision
- v100
- llama.cpp
- qwen
base_model:
- ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
---

# Qwen3.8-Flash-Next — V100 recipe companion files

The two small files you need to run **Qwen3.8-Flash-Next** with **MTP
speculative decoding** *and* **vision** on 2× Tesla V100-PCIE-32GB.

This repository deliberately does **not** host the model weights. It hosts only
the two companion files that are easy to miss, plus a link to the full
reproduction recipe.

> **Full recipe:** [github.com/mw00/peasant-smith — `recipes/qwen38-flash-next-v100-32gb.md`](https://github.com/mw00/peasant-smith/blob/main/recipes/qwen38-flash-next-v100-32gb.md)
> — exact llama.cpp build, CUDA/sm_70 setup, all flags, measured results.

## What is here

| File | Size | Purpose |
|---|---|---|
| `MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` | 2.60 GB | **MTP draft head** — enables speculative decoding |
| `GSQ-RCO-IQ3_XXS/mmproj-Qwen3.8-Flash-Next-BF16.gguf` | 0.91 GB | **Vision projector** — enables multimodal input |

## What is NOT here (get the weights from upstream)

The 75.8 GB of quantized model weights live in the original repository. Download
them from there:

```bash
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
  --include "IQ3_XXS/*" --local-dir .
```

Source: [`ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF`](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)

That gives you two shards (`00001-of-00002`, 47.0 GB and `00002-of-00002`,
28.8 GB). Note that **shard 2 is not more weights** — it is the 51.2B-parameter
per-layer n-gram embedding table (`per_layer_token_embd`) at IQ4_NL. It is a
lookup table, not matmul weights, so it is byte-identical across every quant
variant.

The vision projector is *also* available upstream at the root of that repo. The
copy here exists only so you can grab the two small files from one place.

Where the MTP head comes from: [`unsloth/Qwen3.8-Flash-Next-GGUF`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)
(`MTP/`). The `shared-Q8_0` variant is the fastest of the set and is what is
mirrored here.

## Full download, all three sources

```bash
# 1. model weights (75.8 GB)
hf download ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
  --include "IQ3_XXS/*" --local-dir .

# 2. vision projector (0.91 GB)
hf download peasantsmith/qwen38-flash-next-v100-recipe \
  --include "GSQ-RCO-IQ3_XXS/*" --local-dir .

# 3. MTP draft head (2.60 GB)
hf download peasantsmith/qwen38-flash-next-v100-recipe \
  --include "MTP/*" --local-dir .
```

## Requirements — read this before anything else

**A stock `ggml-org/llama.cpp` build cannot use the MTP head.** Mainline has no
MTP graph for the `qwen4exp` architecture and no `--spec-type draft-mtp` option.
Passing a head to it silently does nothing.

You need one of:

- The Unsloth prebuilt binaries — tag `b10715-mix-86bd2d3` or newer, from
  [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
- The upstream MTP pull request ([ggml-org/llama.cpp#28243](https://github.com/ggml-org/llama.cpp/pull/28243))

**On a V100 (sm_70) specifically:** pick the **`cuda12-portable`** bundle. The
release ships several CUDA 12 assets and they do *not* all support Volta — the
`cuda12-newer` bundle targets `min sm 86` and will load then fail at device time.
Check before running:

```bash
grep -E '^(variant|min sm|toolkit version):' BUILD_INFO.txt
# min sm must be <= 70
```

The recipe pins **`b11030-mix-5ff778e`** / `cuda12-portable` / commit `6ba30d0`,
which is what all the measurements were taken on.

## Minimal working command

```bash
./llama-server \
  -m  IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf \
  --mmproj GSQ-RCO-IQ3_XXS/mmproj-Qwen3.8-Flash-Next-BF16.gguf \
  -md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 131072 -fa on \
  -ngl 99 -ncmoe 0 \
  --tensor-split 55,45 \
  -ctk bf16 -ctv bf16 -ctkd bf16 -ctvd bf16 \
  -t 12 -b 2048 -ub 512 \
  -lm mmap --lazy-mode on \
  --jinja --port 8080
```

## Measured results (2× V100-PCIE-32GB, 64 GB total)

| Metric | Value |
|---|---|
| Production decode @ 131k ctx | **80.9 tok/s** |
| Peak decode (tuned, short ctx) | **103.3 tok/s** |
| MTP speedup over no speculation | **2.4×** |
| Draft acceptance @ 131k, `n-max 3` | **87.8%** |
| Prefill | 176–311 tok/s |

> **Pricing note (for anyone budgeting):** used V100 prices fluctuate a lot and
> depend on seller, condition, and SXM2 vs PCIe. As of **23 September 2026**,
> refurbished V100-PCIE-32GB cards were listed around **£550 (~$735) each**.
> Dated reference point only — check live listings.

## The three things that cost the most time

1. **`min sm` must be ≤ 70.** Verify it in `BUILD_INFO.txt`, not from the
   version string. Wrong bundle = loads, then fails at device time.
2. **`-ncmoe 0` is mandatory.** Without it experts spill to CPU and decode
   collapses from ~54 to ~19 tok/s.
3. **The default tensor split is a trap.** It loads with ~500 MB of headroom and
   OOMs once the KV cache grows. Use `55,45`.

Everything else — CUDA 13 dropping Volta, the `nvcc` shadowing trap, KV
quantization, `n-max` having opposite optima at short vs long context, and the
full tuning matrix — is in the
[recipe](https://github.com/mw00/peasant-smith/blob/main/recipes/qwen38-flash-next-v100-32gb.md).

## Credit

- Model: [ISTA-DASLab](https://huggingface.co/ISTA-DASLab) (GSQ/RCO non-uniform
  quantization)
- MTP draft heads: [Unsloth](https://huggingface.co/unsloth)
- Prebuilt MTP-capable llama.cpp: [unslothai/llama.cpp](https://github.com/unslothai/llama.cpp)
- Base model: [Qwen](https://huggingface.co/Qwen)