Instructions to use hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6")
model = AutoModelForMultimodalLM.from_pretrained("hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6

SGLang

How to use hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 with Docker Model Runner:
```
docker model run hf.co/hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6
```

Swift-Qwen3.8-27B-PARO-MXFP6

ukisai/Swift-Qwen3.8-27b with ParoQuant rotations and MXFP6 (E2M3) weights, for AMD Radeon AI PRO R9700 (RDNA4) on a patched vLLM.

Size: 25.1 GB (bf16: 55.6 GB).

This is a 27B model; the Hub's params badge counts packed tensor elements.

Quality

	this model	bf16
WikiText-2 perplexity	7.083	7.098
KL divergence vs bf16 (top-256, nats)	0.0117	0
Top-1 token match with bf16	95.06%	100%
MMLU 5-shot	82.6%	—
GSM8K flexible / strict	96.7% / 95.9%	— / —
HumanEval / HumanEval+	97.0% / 93.9%	— / —

bf16 = the unquantized source model served on the same image; — = not measured. Perplexity is cross-stack — this model at tensor parallel 1, bf16 at 2 — and wikitext-2's train split is in the stage-2 calibration mix, so a reading at or below bf16 is not evidence of a general gain. GSM8K and HumanEval are chat-templated with reasoning on, greedy, on a Qwen chat template at reasoning_effort=xhigh rather than the default template the command below uses; both scores move with that choice.

Performance

2× Radeon AI PRO R9700, TP=2, DFlash2-FP8 drafter (7 tokens), KV pinned with KV_MEM=13500000000, tokens/s.


Single-stream decode	138.0
Concurrent decode c1 / c2 / c4 / c8	124.7 / 209.6 / 321.6 / 394.0
Prefill 2k / 8k / 16k / 32k / 60k	3327 / 3628 / 3570 / 3479 / 3348
KV cache tokens	685,554
Weights per GPU	12.85 GiB

Each row is a single measured run — the speed rows at a 12.5 GB pin — on a newer radiance build than the 0.9.3 image the setup below installs, so expect these figures to move. KV capacity follows the pin, not the weight format.

How to run

Requirements

Two AMD RDNA4 (gfx1201) GPUs. The kernels are built for that architecture only; measured on 2 x Radeon AI PRO R9700 at tensor parallel 2.
Linux with the amdgpu kernel driver loaded, so /dev/kfd and /dev/dri exist. ROCm userspace ships inside the image, so there is no host ROCm install.
docker or podman (auto-detected), and a host git and python3 (standard library only).
Disk for the checkpoint, plus about 12 GiB for the DFlash2-FP8 drafter and the stilldeadcode/vllm-radiance:0.9.3 image, both of which setup fetches.

Two ways to get the code. MXFP6 support is open as a pull request to ggz14/radiance-vllm-mxfp4 (#46). Clone the fork below to run it today, or wait for the pull request to merge and clone ggz14's repository instead — the commands are the same either way.

Install and serve

git clone https://codeberg.org/hugypufy/radiance-vllm-mxfp4 && cd radiance-vllm-mxfp4

SRC_REPO=hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 SNAP=~/models/Swift-Qwen3.8-27B-PARO-MXFP6 \
  ./setup-paroquant.sh --mxfp6 --yes

MODEL_DIR=Swift-Qwen3.8-27B-PARO-MXFP6 MODE=prod SPEC=7 ./paroquant/run_paroquant.sh

Setup is one-time and safe to re-run. The launcher reads the quantization format from config.json and sets the kernel knobs it needs, so that is the whole serve command; the server answers as Qwen3.8-PARO on http://localhost:8080/v1. Every start patches vLLM inside the container and compiles the ParoQuant kernels for gfx1201, about two minutes, before the model loads; the first start also compiles Triton and inductor graphs, which adds several minutes and is cached for later starts.

Test

curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.8-PARO","messages":[{"role":"user","content":"Hello!"}]}'

Options — set as environment variables on the serve command.

Variable	Default	Notes
`KV_MEM`	unset; the pool is sized from `GPU_UTIL` (0.92)	Pin the KV cache in bytes per GPU; this checkpoint is served at `KV_MEM=13500000000`
`SPEC`	`5`	DFlash2 draft tokens per step. The numbers above used `7`
`CHAT_TEMPLATE`	the repository's `qwen-fixed-v22.3.jinja`	Any `.jinja` on the host, for example this checkpoint's own `chat_template.jinja`
`RADIANCE_VERIFY_HEAD`	`1`	Set to `0` to use `prompt_logprobs`, which hangs at the default

Format

Weights: OCP MXFP6 E2M3, one e8m0 scale per 32 weights — 6.25 bits/weight.
Rotations: from z-lab/Qwen3.8-27B-PARO, frozen.
Recipe: round-to-nearest, then a stage-2 fine-tune (FT) of weights and block scales, rotations frozen.
fp16: visual tower, linear_attn.in_proj_a/b, linear_attn.conv1d / A_log / dt_bias, lm_head, embeddings, norms, and the rotation theta / channel_scales (pairs are int16). MTP head not included.

Licence

Governed by the base model's Swift Open License v1.0 (terms); included as LICENSE, with Qwen's Apache-2.0 as LICENSE-APACHE-2.0. Modified derivative, not made or endorsed by UkisAI; NOTICE lists the changes. Rotations from z-lab/Qwen3.8-27B-PARO (Apache-2.0); original base Qwen/Qwen3.8-27B.

Downloads last month: 178

Safetensors

Model size

21B params

Tensor type

F16

I16

Model tree for hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6

Base model

Qwen/Qwen3.8-27B

Finetuned

ukisai/Swift-Qwen3.8-27b

Quantized

(54)

this model