Instructions to use Qwen/Qwen3.8-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-27B-FP8

SGLang

How to use Qwen/Qwen3.8-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-27B-FP8
```

Quantized FP8 MTP Layers hurt acceptance rates

#13

by andrew-stanton - opened 23 days ago

Discussion

andrew-stanton

23 days ago

In the FP8 variant, MTP layers are quantized to FP8. This turns out to have a real impact on draft acceptance rates, with an average drop in my experiments of around 3-4% across each position. I'd recommend updating this repo with the full BF16 (it costs ~0.4GB of VRAM) as the speed up is noticeable - I've validated this locally as well with the exact patch (Unsloth's NVFP4 also preserves the original BF16 MTP layers).

marszhuo

22 days ago

FP8 版本在grounding 定位方面极速下降，分析原因可能是 VIT 模块被量化，导致坐标计算误差扩大

danielcherubini

21 days ago

Does anyone have the BF16 mtp layers ready so we can test it?

amartinOnepoint

21 days ago

•

edited 21 days ago

https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4/blob/main/model_mtp.safetensors
but 0 acceptance with it...

alex81k

20 days ago

•

edited 20 days ago

I vibecoded the script that made things work bf16 MTP but I used original model to extract MTP head not NVFP4. Probably it doesn't matter, sizes are the same.
So

Download bf16 mtp hf download hf://unsloth/Qwen3.8-27B-NVFP4/model_mtp.safetensors --local-dir .
Rename model_mtp.safetensors to mtp.safetensors
Replace model.safetensors.index.json with https://gist.github.com/kryoz/7e31dcf2724e679b50cc36f488051513

alex81k

20 days ago

•

edited 20 days ago

I've tested bf16 mtp and haven't found any significant improvement in acceptance rate over fp8 mtp built-in head. More on that bf16 is slower so overall speed is better with default fp8 mtp.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment