Instructions to use z-lab/Qwen3.8-27B-DFlash2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use z-lab/Qwen3.8-27B-DFlash2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="z-lab/Qwen3.8-27B-DFlash2")

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("z-lab/Qwen3.8-27B-DFlash2")
model = AutoModel.from_pretrained("z-lab/Qwen3.8-27B-DFlash2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use z-lab/Qwen3.8-27B-DFlash2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "z-lab/Qwen3.8-27B-DFlash2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.8-27B-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/z-lab/Qwen3.8-27B-DFlash2

SGLang

How to use z-lab/Qwen3.8-27B-DFlash2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "z-lab/Qwen3.8-27B-DFlash2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.8-27B-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "z-lab/Qwen3.8-27B-DFlash2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.8-27B-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use z-lab/Qwen3.8-27B-DFlash2 with Docker Model Runner:
```
docker model run hf.co/z-lab/Qwen3.8-27B-DFlash2
```

for 3090 MTP and gguf is best for performance 250k context and accuracy.

by akierum - opened 10 days ago

Discussion

akierum

10 days ago

Because Qwen3.8-27B uses a 3:1 hybrid attention design (offloading 75% of layers to linear attention), running Q8 model weights alongside Q8 KV cache requires ~39.3 GB of VRAM, leaving ~8.7 GB of free headroom on 2x RTX 3090 (48 GB total).

Additionally, because the RTX 3090 (Ampere SM86) features native hardware INT8 Tensor Cores, Q8 weights and Q8 KV cache execute at full hardware speeds without the software-emulation penalties seen with FP8.

Component VRAM Consumption
Q8_0 Model Weights 28.60 GB
250k Q8_0 KV Cache 8.19 GB
CUDA Runtime & Engine Overhead ~2.50 GB
Total Required VRAM 39.29 GB
Total Available VRAM (2x RTX 3090) 48.00 GB

Switch to DFlash 2 if:
Your prompts are short to medium (< 1K–2K tokens) and your primary goal is the fastest possible interactive output streaming (~220+ tok/s).

akierum

10 days ago

•

edited 10 days ago

For raw inference speed on an RTX 3090, INT8 is the clear winner, while Q8 (GGUF) offers the best output quality. FP8 provides memory savings but delivers poor execution performance on Ampere GPUs like the RTX 3090.

Format Native Tensor Core Support Speed (it/s or tokens/s) Visual / Text Quality Primary Advantage

INT8 Yes (Ampere INT8 cores) Fastest (1.8x–2x over FP8) Very Good Maximum speed and throughput
Q8 (GGUF) Partial (Engine dependent) Moderate Highest (99% of FP16/BF16) Minimal degradation from original model
FP8 No (Emulated in software) Slowest on 3090 Moderate Fits large models into VRAM (Ada/Hopper optimized)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment