Instructions to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("feature-extraction", model="Anbeeld/Qwen3.5-9B-DFlash-GGUF", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Anbeeld/Qwen3.5-9B-DFlash-GGUF", trust_remote_code=True)
model = AutoModel.from_pretrained("Anbeeld/Qwen3.5-9B-DFlash-GGUF", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with Ollama:
```
ollama run hf.co/Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with Docker Model Runner:
```
docker model run hf.co/Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M
```

Lemonade

How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-9B-DFlash-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Anbeeld/Qwen3.5-9B-DFlash-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Anbeeld/Qwen3.5-9B-DFlash-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen 3.5 9B DFlash GGUF

GGUF quantizations of z-lab DFlash draft model for Qwen 3.5 9B.

Use with BeeLlama.cpp, a llama.cpp fork with advanced quantization features.

Qwen3.5-9B-DFlash

Paper | Github | Blog

This DFlash draft model is a joint retrain from Z-Lab and Modal, trained with 40k sequence length and sliding-window attention for improved long-context performance. It is mirrored across the following Hugging Face repositories:

This repository contains a DFlash draft model for Qwen/Qwen3.5-9B. It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server.

DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model's output distribution.

Quick Start

Installation

SGLang

Install a recent SGLang build with DFlash support:

uv pip install --upgrade "sglang[all]"

For best performance on Blackwell GPUs, use an SGLang build that includes DFlash, FA4/TRT-LLM attention, and FlashInfer support.

vLLM

For vLLM support, please refer to vllm-project/vllm#40898. We will update the PR to make it merge-ready soon.

Launch Server

This model should be used with an inference server that supports DFlash speculative decoding. An example SGLang deployment is:

export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-9B \
  --trust-remote-code \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.5-9B-DFlash \
  --speculative-dflash-block-size 8 \
  --speculative-draft-attention-backend fa4 \
  --attention-backend trtllm_mha \
  --linear-attn-prefill-backend flashinfer \
  --linear-attn-decode-backend flashinfer \
  --mamba-scheduler-strategy extra_buffer \
  --tp-size 1 \
  --max-running-requests 32 \
  --cuda-graph-max-bs-decode 32 \
  --cuda-graph-backend-prefill tc_piecewise \
  --enable-flashinfer-allreduce-fusion \
  --mem-fraction-static 0.8 \
  --host 0.0.0.0 \
  --port 30000

Block size 8 is the recommended default for higher-concurrency serving. Block size 16 gives longer accept lengths and strong concurrency-1 throughput in most workloads.

Benchmark Results

We benchmarked DFlash against the autoregressive baseline and Qwen's built-in MTP draft path. DFlash reaches up to 5.01x speedup at concurrency 1 and 2.58x at concurrency 32. Across the benchmark suite, DFlash delivers higher throughput than MTP at every matched setting where both completed.

Setup

Runtime: SGLang on 1x NVIDIA B200 GPU, tensor parallel size 1, bfloat16
Backends: trtllm_mha target attention, fa4 DFlash draft attention, flashinfer linear-attention prefill and decode
Workloads: GSM8K, MATH500, HumanEval, MBPP, and MT-Bench with the Qwen chat template
Decoding: greedy, thinking enabled, max output length 4096 tokens
Measurement: 5 independent runs per configuration at concurrency 1 and 32 with continuous batching
Throughput: generated output tokens / wall-clock benchmark time, including prefill and scheduling
Accept length: completion_tokens / spec_verify_ct per generation turn, averaged across generation turns

Throughput and Speedup

Each cell is output tok/s (speedup). Bold marks the fastest speculative configuration in each row.

Concurrency 1

Workload	Baseline	MTP steps=3	DFlash block=4	MTP steps=7	DFlash block=8	MTP steps=15	DFlash block=16
gsm8k	245.2 (1.00x)	537.7 (2.19x)	609.8 (2.49x)	573.4 (2.34x)	890.7 (3.63x)	435.5 (1.78x)	1027.3 (4.19x)
math500	244.6 (1.00x)	558.3 (2.28x)	636.3 (2.60x)	617.3 (2.52x)	987.6 (4.04x)	485.3 (1.98x)	1225.7 (5.01x)
humaneval	243.4 (1.00x)	537.6 (2.21x)	633.2 (2.60x)	586.9 (2.41x)	959.5 (3.94x)	447.6 (1.84x)	1195.8 (4.91x)
mbpp	244.7 (1.00x)	526.4 (2.15x)	624.4 (2.55x)	543.6 (2.22x)	935.7 (3.82x)	403.6 (1.65x)	1092.6 (4.46x)
mt-bench	243.7 (1.00x)	501.7 (2.06x)	560.2 (2.30x)	494.6 (2.03x)	757.5 (3.11x)	368.0 (1.51x)	834.3 (3.42x)

Concurrency 32

Workload	Baseline	MTP steps=3	DFlash block=4	MTP steps=7	DFlash block=8	MTP steps=15	DFlash block=16
gsm8k	5837.6 (1.00x)	10421.0 (1.79x)	11882.9 (2.04x)	10132.5 (1.74x)	13718.0 (2.35x)	8332.8 (1.43x)	11019.7 (1.89x)
math500	5885.7 (1.00x)	11124.9 (1.89x)	12534.0 (2.13x)	11213.3 (1.91x)	15198.3 (2.58x)	7785.0 (1.32x)	13227.4 (2.25x)
humaneval	5513.0 (1.00x)	9645.9 (1.75x)	11882.7 (2.16x)	9701.0 (1.76x)	14229.9 (2.58x)	6901.7 (1.25x)	12406.1 (2.25x)
mbpp	5538.8 (1.00x)	9116.4 (1.65x)	11561.2 (2.09x)	8701.6 (1.57x)	13460.3 (2.43x)	6220.5 (1.12x)	11338.9 (2.05x)
mt-bench	5491.7 (1.00x)	9135.0 (1.66x)	10072.2 (1.83x)	8436.9 (1.54x)	10718.2 (1.95x)	5917.8 (1.08x)	8495.7 (1.55x)

Accept Length

Mean accept length at concurrency 1. Bold marks the higher value in each matched MTP/DFlash pair.

Workload	MTP steps=3	DFlash block=4	MTP steps=7	DFlash block=8	MTP steps=15	DFlash block=16
gsm8k	3.464	3.452	5.276	5.400	6.388	6.949
math500	3.541	3.555	5.466	5.757	6.728	7.721
humaneval	3.493	3.571	5.326	5.798	6.399	7.927
mbpp	3.338	3.454	4.790	5.376	5.508	6.820
mt-bench	3.229	3.193	4.551	4.606	5.438	5.716

Citation

If you find DFlash useful, please cite the original paper:

@article{chen2026dflash,
  title   = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author  = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  journal = {arXiv preprint arXiv:2602.06036},
  year    = {2026}
}