Instructions to use Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4")
model = AutoModelForMultimodalLM.from_pretrained("Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4

SGLang

How to use Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4 with Docker Model Runner:
```
docker model run hf.co/Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4
```

Capicua25x commited on Aug 25

Commit

dd8bb26

verified ·

1 Parent(s): 532de84

remove single-card section pending further evaluation

Browse files

Files changed (1) hide show

README.md +0 -47

README.md CHANGED Viewed

@@ -163,53 +163,6 @@ not drafting. What C buys instead: the full 262k native window (FP8+MTP tops out
 Pick FP8 for short-query burst capacity; pick this model for max context on the same silicon —
 at real context sizes you give up nothing measurable.
-## Single card (1× R9700 / 32GB-class)
-**Single-card path: MXFP4 @ fp8 KV** — the only quant that fits (weights ≈21 GB); fp8 KV stretches
-the window from 32k to 104k. It serves — with real limits: ≈4 concurrent users and a window still
-well short of the 262k the multi-card port gets on the same silicon. llama.cpp/GGUF can also run
-this model on one card for plain chat, but skips the batching, prefix caching, and window this
-config gets from the same card — not the recommendation unless that's genuinely all you need.
-Measured 2026-08-25 on 1× R9700, rc10 image, bench v4, KV pool 113,642 tokens (127,348 at 4 slots)
-→ 104k-token window:
-| workload (v4, TP1 + fp8 KV, 104k window, 4 slots) | c1 | c2 | c4 |
-|---|---|---|---|
-| trivial | 43.8 / 44 | 40.7 / 81 | 39.9 / 143 |
-| short essay | 29.0 / 29 | 27.6 / 51 | 23.5 / 85 |
-| 6k-prefix essay | 32.4 / 32 | 27.0 / 51 | 23.7 / 90 |
-Stays ≥20 tok/s/user through c4 (untested beyond). Serve command — one device pair, TP1, 104k
-window, 4 slots:
-```bash
-docker run --rm --name vllm-qwen --network=host \
-  --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/card1 \
-  --group-add=video --group-add=render --ipc=host \
-  -v /path/to/quants:/quant:ro -e HF_HUB_OFFLINE=1 \
-  --entrypoint /usr/local/bin/vllm capicua25x/vllm-rocm-rdna4:latest \
-  serve /quant/Qwen3.8-27B-MXFP4-Quark-RDNA4 \
-  --served-model-name qwen --port 8011 --trust-remote-code \
-  --tensor-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 106496 \
-  --attention-backend TRITON_ATTN --enable-prefix-caching \
-  --max-num-seqs 4 --max-num-batched-tokens 8000 \
-  --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 --max-cudagraph-capture-size 32 \
-  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'
-```
-**16GB cards (RX 9070 XT): this 27B does not fit — in ANY runtime.** Our quants need 21 GB of
-weights; even a Q4 GGUF (≈15.5 GB) technically loads and then leaves no room for KV, i.e. no
-usable context window. llama.cpp's 16GB options are CPU offload (at a large speed cost) or a
-smaller model — the latter is the honest recommendation. The port's kernels run fine on gfx1200
-silicon; pair the card with a model whose weights + context leave real headroom.
-Quality note: fp8 KV over this MXFP4 quant carries one accuracy smoke (gsm8k n=50 thinking-on:
-0.94 flexible / 0.88 strict — within the ±0.04 sampling noise of the bf16-KV baseline's 0.91–0.93,
-at its lower edge). Validate on your own workload before committing; a KV-calibrated variant with
-scale side-files exists if deeper validation matters to you.
 ## Accuracy (AA class-A, paired items, seed 1234, on-spec sampling)
 **ref** = the same checkpoint served in bf16 by a cloud provider. Same judge for all judged rows.

 Pick FP8 for short-query burst capacity; pick this model for max context on the same silicon —
 at real context sizes you give up nothing measurable.
 ## Accuracy (AA class-A, paired items, seed 1234, on-spec sampling)
 **ref** = the same checkpoint served in bf16 by a cloud provider. Same judge for all judged rows.