Instructions to use originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0
# Run inference directly in the terminal:
llama cli -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0
# Run inference directly in the terminal:
llama cli -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0

Use Docker

docker model run hf.co/originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0

LM Studio
Jan
Ollama
How to use originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY with Ollama:
```
ollama run hf.co/originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0
```
Unsloth Desktop
Docker Model Runner
How to use originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY with Docker Model Runner:
```
docker model run hf.co/originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0
```

Lemonade

How to use originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY:Q8_0

Run and chat with the model

lemonade run user.Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY-Q8_0

List all available models

lemonade list

Atomic Chat

Qwen3.6-27B Unsloth MTP Head Only GGUF

This repo contains only the Qwen3.6-27B MTP / NextN draft head extracted from Unsloth's Q8_0 MTP GGUF.

It is not a standalone chat model. It is a compact graft source for putting the native MTP head back onto a compatible non-MTP Qwen3.6-27B GGUF.

File

File	Purpose
`Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY.gguf`	15-tensor GGUF containing only `blk.64.*` MTP tensors

Local metadata:

tensor_count: 15
qwen35.nextn_predict_layers: 1
qwen35.block_count: 65
mtp_shard.source_block: 64
sha256: F17F5FD3138F7081197E0AFAC75C509F4367099DBDBB2F350CFCBC8017539697

Tensor names:

blk.64.attn_k.weight
blk.64.attn_k_norm.weight
blk.64.attn_norm.weight
blk.64.attn_output.weight
blk.64.attn_q.weight
blk.64.attn_q_norm.weight
blk.64.attn_v.weight
blk.64.ffn_down.weight
blk.64.ffn_gate.weight
blk.64.ffn_up.weight
blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight
blk.64.post_attention_norm.weight

Provenance

Extracted from:

Model repo: https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
Exact donor file: https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-Q8_0.gguf
Upstream base config: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json

The upstream Qwen3.6-27B config declares mtp_num_hidden_layers: 1. The head in this file is the Q8_0 GGUF version of that MTP block, placed at blk.64.* for the 64-layer 27B trunk.

Why This Exists

MTP is a separate draft module, not just a flag. NVIDIA's MTP docs describe it as extra modules that predict future tokens beyond the normal next-token head. In Qwen3.6-27B, those weights live after the 64 trunk blocks.

Some fine-tune, merge, quantization, or export flows only keep the main model trunk and skip the extra MTP tensors. When that happens, the model can still run normally, but speculative decoding cannot use the trained native draft head. Depending on runtime, MTP may fail to load, silently draft from missing/random weights, or have near-zero acceptance.

Grafting the original MTP head back in can restore native MTP speculative decoding for compatible Qwen3.6-27B derivatives. It is not guaranteed to be optimal after heavy fine-tuning: the draft head was trained against the original trunk hidden states, so acceptance can fall if the tuned trunk diverged a lot.

Compatibility checklist:

Same architecture: qwen35
Same hidden size: 5120
Same FFN width: 17408
Same attention heads: 24
Same KV heads: 4
Same tokenizer/vocab and embedding/lm-head compatibility
Same 27B depth: 64 main layers, with the grafted MTP block becoming blk.64.*

Do not use this blindly on a model with changed tokenizer, changed dimensions, changed architecture, or missing matching Qwen3.6-27B metadata.

Grafting Onto A Non-MTP Qwen3.6-27B GGUF

Use a GGUF-aware graft script. Do not concatenate files.

The graft operation needs to:

Read the body GGUF header and tensor table.
Verify it is a compatible Qwen3.6-27B body.
Copy the body tensor data unchanged.
Append the 15 tensors from this head-only GGUF.
Rewrite tensor metadata and offsets.
Set or update:

qwen35.block_count = 65
qwen35.nextn_predict_layers = 1

Conceptual command shape:

python inject_mtp.py `
  --body Qwen3.6-27B-non-mtp.gguf `
  --mtp-head Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY.gguf `
  --out Qwen3.6-27B-with-MTP.gguf

After grafting, inspect the result:

$env:PYTHONPATH = "C:\path\to\llama.cpp\gguf-py"
python -m gguf.scripts.gguf_dump Qwen3.6-27B-with-MTP.gguf --no-tensors
python -m gguf.scripts.gguf_dump Qwen3.6-27B-with-MTP.gguf | Select-String "blk.64"

Expected signs:

qwen35.block_count = 65
qwen35.nextn_predict_layers = 1
blk.64.nextn.eh_proj.weight
blk.64.nextn.shared_head_norm.weight

Grafting Onto Depth-Expanded Qwen3.6-40B Variants

This head can also be grafted onto a compatible Qwen3.6-40B derivative when that model is a depth-expanded descendant of the same Qwen3.6-27B family, as with the local IQ4_XS graft. Treat this as an experimental compatibility path, not as a native 40B MTP head.

The important difference is layer count. The original Qwen3.6-27B body has 64 main layers, so this donor head is stored as blk.64.*. The expanded 40B body used locally has 96 main layers, so the grafted MTP block belongs after those layers as blk.96.*.

For a 96-layer 40B body, the graft script needs to:

Verify the body is still qwen35 and is dimensionally compatible.
Copy the body tensor data unchanged.
Rename the donor MTP tensors from blk.64.* to blk.96.* while appending them.
Set or update:

qwen35.block_count = 97
qwen35.nextn_predict_layers = 1

Conceptual command shape:

python inject_mtp.py `
  --body Qwen3.6-40B-IQ4_XS-non-mtp.gguf `
  --mtp-head Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY.gguf `
  --out Qwen3.6-40B-IQ4_XS-with-MTP.gguf `
  --target-mtp-block 96

After grafting, inspect for blk.96 rather than blk.64:

$env:PYTHONPATH = "C:\path\to\llama.cpp\gguf-py"
python -m gguf.scripts.gguf_dump Qwen3.6-40B-IQ4_XS-with-MTP.gguf --no-tensors
python -m gguf.scripts.gguf_dump Qwen3.6-40B-IQ4_XS-with-MTP.gguf | Select-String "blk.96"

Expected signs:

qwen35.block_count = 97
qwen35.nextn_predict_layers = 1
blk.96.nextn.eh_proj.weight
blk.96.nextn.shared_head_norm.weight

The reason this works structurally is that the expanded 40B variant keeps the same core tensor shapes as Qwen3.6-27B: hidden size 5120, attention heads 24, KV heads 4, and FFN width 17408. Those matching dimensions make the MTP head tensors loadable after remapping from block 64 to block 96.

That does not mean quality or acceptance is guaranteed. The donor MTP head was trained to predict from Qwen3.6-27B's layer-64 hidden state, while a 96-layer 40B graft feeds it the representation after layer 96. If draft acceptance is poor, lower --spec-draft-n-max to 3 or 1 and benchmark again. A truly optimal 40B MTP head would need to be trained against the 40B trunk.

llama.cpp Setup Notes

MTP support for Qwen3.6 GGUF has moved past the original PR #22673 merge notes. A useful build now needs all of these pieces:

Qwen3.6 MTP / NextN loading and --spec-type draft-mtp: https://github.com/ggml-org/llama.cpp/pull/22673
Draft MTP KV-cache type propagation, so --spec-draft-type-k and --spec-draft-type-v are honored: https://github.com/ggml-org/llama.cpp/pull/23646
NVFP4 MTP scale tensor handling for native-MTP NVFP4 GGUFs: https://github.com/ggml-org/llama.cpp/pull/23563
CUDA fast Walsh-Hadamard transform support and follow-up PDL/fallback fixes: https://github.com/ggml-org/llama.cpp/pull/23615 and https://github.com/ggml-org/llama.cpp/pull/23690
TurboQuant KV-cache types turbo2, turbo3, and turbo4, if you want the TurboQuant profiles documented below.

Still-current flag notes:

Use --spec-type draft-mtp, not the older --spec-type mtp.
Use --spec-draft-type-k and --spec-draft-type-v to mirror the target KV cache type you actually want for the MTP draft path.
--spec-draft-n-max 6 is the normal live depth for 27B Qwen3.6 native-MTP profiles here.
Local coding-agent profiles currently use --spec-draft-p-min 0.0 for throughput. 0.75 remains a useful stricter acceptance / regression fallback from the Unsloth guidance.
You can chain ngram speculative decoding with MTP for replay-heavy workloads, for example --spec-type ngram-mod,draft-mtp.

Related external notes:

Unsloth MTP guide: https://unsloth.ai/docs/models/qwen3.6#mtp-guide
Unsloth Reddit note: https://www.reddit.com/r/unsloth/comments/1tdw5jq/qwen36_mtp_unsloth_ggufs_now_18x_faster/

Fresh check on 2026-05-28: ggml-org/llama.cpp master was at b9371-1-g8ad8aef44. No newer MTP, NextN, NVFP4, FWHT, or TurboQuant change was better than the tested vs7 stack below. The most relevant newer CUDA change only gates PDL for old CUDA Toolkit versions; this build uses CUDA 13.1.

Current Tested Runtime

The old hand-merge recipe for "PR #22673 plus TurboQuant" has been superseded locally. The current tested Windows / RTX 5090 runtime is:

repo: llama-cpp-turboquant-MTP-NVFP4-13.1-ngram
branch: tq-mtp-refresh-20260526-vs7-fwht
commit: ea82a5bda
llama.cpp version: 9432
build: build-cuda131-sm120-vs7-fwht
binary: build-cuda131-sm120-vs7-fwht\bin\Release\llama-server.exe

That vs7 runtime combines TheTom's TurboQuant KV-cache branch with the current MTP cleanup stack, the MTP draft-KV type fix, NVFP4 MTP scale tensor support, and CUDA FWHT support. In practice, this means the local tree no longer needs the old conflict-resolution recipe in this README; use the checked-out vs7 branch or recreate that patch stack directly.

Feature sanity checks:

.\build-cuda131-sm120-vs7-fwht\bin\Release\llama-server.exe --help |
  findstr /C:"draft-mtp" /C:"ngram-simple" /C:"turbo4"

.\build-cuda131-sm120-vs7-fwht\bin\Release\test-backend-ops.exe test -b CUDA0 -o MUL_MAT_HADAMARD

CUDA 13.1 Note

For Blackwell / RTX 5090 we used CUDA 13.1, not CUDA 13.2. The Unsloth discussion and guide warn that CUDA 13.2 can produce garbled output in this area; the community note says to use CUDA 13.1 or lower until NVIDIA fixes it in a later toolkit.

The tested local build used:

nvcc: Cuda compilation tools, release 13.1, V13.1.80
MSVC: 19.44
CMake: Visual Studio 17 2022 generator
CUDA arch: 120, normalized by CMake/CUDA to 120a

The local build tree was:

llama-cpp-turboquant-MTP-NVFP4-13.1-ngram
branch: tq-mtp-refresh-20260526-vs7-fwht
commit: ea82a5bda
describe: feature-turboquant-kv-cache-b9410-2b61ea2-22-gea82a5bda
included fixes: llama.cpp PR #23646, #23563, #23615, #23690
latest upstream checked: ggml/master b9371-1-g8ad8aef44 on 2026-05-28

Build command shape:

$cuda = "C:\Users\Lui\Documents\GitHub\Local_llamacpp\cuda-13.1-toolkit"
$repo = "C:\Users\Lui\Documents\GitHub\Local_llamacpp\llama-cpp-turboquant-MTP-NVFP4-13.1-ngram"

$env:CUDA_PATH = $cuda
$env:CUDA_PATH_V13_1 = $cuda
$env:PATH = "$cuda\bin;$cuda\bin\x64;$env:PATH"

cmake -S $repo -B "$repo\build-cuda131-sm120-vs7-fwht" `
  -G "Visual Studio 17 2022" -A x64 `
  -T "cuda=$cuda" `
  -DGGML_CUDA=ON `
  -DGGML_CUDA_FA_ALL_QUANTS=ON `
  -DGGML_CUDA_GRAPHS=ON `
  -DCMAKE_CUDA_ARCHITECTURES=120 `
  -DCUDAToolkit_ROOT="$cuda" `
  -DLLAMA_CURL=OFF `
  -DLLAMA_BUILD_UI=OFF

cmake --build "$repo\build-cuda131-sm120-vs7-fwht" `
  --config Release `
  --target llama-server llama-bench test-backend-ops `
  --parallel 12

Copy CUDA 13.1 runtime DLLs beside the binaries if needed:

Copy-Item "$cuda\bin\x64\*.dll" "$repo\build-cuda131-sm120-vs7-fwht\bin\Release" -Force

Runtime Flags Used Locally

NVFP4 + true TurboQuant KV + MTP, live coding-agent default:

set TURBO_AUTO_ASYMMETRIC=0

llama-server.exe `
  --model Qwen3.6-27B-with-MTP-or-native-MTP-NVFP4.gguf `
  --ctx-size 262144 `
  --n-gpu-layers 999 `
  --main-gpu 0 `
  --parallel 1 `
  --threads 8 `
  --threads-batch 8 `
  --batch-size 2048 `
  --ubatch-size 512 `
  --cache-type-k turbo4 `
  --cache-type-v turbo4 `
  --spec-draft-type-k turbo4 `
  --spec-draft-type-v turbo4 `
  --flash-attn on `
  --poll 100 `
  --cache-ram 16384 `
  --spec-type draft-mtp `
  --spec-draft-n-max 6 `
  --spec-draft-p-min 0.0 `
  --host 127.0.0.1 `
  --port 8080 `
  --metrics

Replay/copy-heavy variants:

--spec-type ngram-mod,draft-mtp
# or targeted benchmark mode:
--spec-type ngram-simple,draft-mtp

Stricter MTP acceptance fallback:

--spec-draft-p-min 0.75

Local Benchmark Notes

On an RTX 5090 with the local CUDA 13.1 build and a native-MTP NVFP4 Qwen3.6-27B GGUF, historical targeted sweeps showed:

Repeated-context copy workload, 1024 generated tokens:

Spec type	TG	Note
`draft-mtp`	188.70 t/s	baseline MTP
`ngram-simple,draft-mtp`	804.12 t/s	best tested replay speed
`ngram-map-k4v,draft-mtp`	781.82 t/s	close second
`ngram-mod,draft-mtp`	689.76 t/s	good, but not best here
`ngram-map-k,draft-mtp`	502.60 t/s	unstable in this sample
`ngram-cache,draft-mtp`	152.96 t/s	not useful for this setup

Long-context coding-agent style workload, about 86k prompt tokens, 1024 generated tokens:

Spec type	TG
`draft-mtp`	66.78 t/s
`ngram-simple,draft-mtp`	70.70 t/s
`ngram-map-k4v,draft-mtp`	58.84 t/s
`ngram-mod,draft-mtp`	55.75 t/s

For this machine/build, the live 27B coding-agent profiles currently stay on plain draft-mtp. Ngram + MTP remains useful for replay/copy-heavy prompts and should be selected per workload, not treated as a universal default. The reduced vs7 correctness smoke on the Unsloth 27B MTP profile initialized draft-MTP with draft KV q8_0/turbo3 and accepted 9 of 12 draft tokens.

Caveats

This is a head-only GGUF. It will not run by itself.
Grafting should be done with a GGUF-aware script.
Acceptance depends on how close the target trunk is to the original Qwen3.6-27B trunk.
Heavy fine-tunes may need MTP-head retraining for best acceptance.
Vision + MTP has been fragile in local llama.cpp testing; use text-only first.
CUDA 13.2 was intentionally avoided for this build.

Downloads last month: 40

GGUF

Model size

0.4B params

Architecture

qwen35

Hardware compatibility

8-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for originalGeek/Qwen3.6-27B-unsloth-MTP-Q8_0-HEAD-ONLY

Base model

Qwen/Qwen3.6-27B

Quantized

(732)

this model