Instructions to use ukisai/Swift-Qwen3.8-27b-int4-AMD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ukisai/Swift-Qwen3.8-27b-int4-AMD with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ukisai/Swift-Qwen3.8-27b-int4-AMD")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ukisai/Swift-Qwen3.8-27b-int4-AMD")
model = AutoModelForMultimodalLM.from_pretrained("ukisai/Swift-Qwen3.8-27b-int4-AMD", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ukisai/Swift-Qwen3.8-27b-int4-AMD with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ukisai/Swift-Qwen3.8-27b-int4-AMD"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ukisai/Swift-Qwen3.8-27b-int4-AMD",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ukisai/Swift-Qwen3.8-27b-int4-AMD

SGLang

How to use ukisai/Swift-Qwen3.8-27b-int4-AMD with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ukisai/Swift-Qwen3.8-27b-int4-AMD" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ukisai/Swift-Qwen3.8-27b-int4-AMD",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ukisai/Swift-Qwen3.8-27b-int4-AMD" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ukisai/Swift-Qwen3.8-27b-int4-AMD",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ukisai/Swift-Qwen3.8-27b-int4-AMD with Docker Model Runner:
```
docker model run hf.co/ukisai/Swift-Qwen3.8-27b-int4-AMD
```

Swift-Qwen3.8-27b-int4-AMD / quantization_report.json

ukisai

initial release

37e7e5e 3 days ago

Raw

History Blame Contribute Delete

4.15 kB

	{
	"source_model": "ukisai/Swift-Qwen3.8-27b",
	"source_revision": "1b30aaaf753fe5c1cb51ada2ea0367a53445359c",
	"checkpoint": "ukisai/Swift-Qwen3.8-27b-int4-AMD",
	"framework": "PyTorch",
	"validation_hardware": "NVIDIA H100",
	"amd_hardware_tested": false,
	"versions": {
	"torch": "2.11.0+cu128",
	"transformers": "5.2.0",
	"amd-quark": "0.12.post1+cu128.torch2.11",
	"accelerate": "1.15.0",
	"safetensors": "0.8.0"
	},
	"evaluation_data": {
	"dataset": "Salesforce/wikitext",
	"config": "wikitext-2-raw-v1",
	"split": "test",
	"fingerprint": "a46124b21ac53738",
	"windows": 8,
	"window_length": 512,
	"selection": "first 4096 tokens; small sanity check, not full benchmark",
	"input_sha256": "2ebdb31fd4394419aabae16a635375c8e70d306c14462051d849056893cd2b27"
	},
	"evaluation": {
	"label": "int4-final-export",
	"nlls": [
	1.820142388343811,
	2.3994803428649902,
	2.3420217037200928,
	2.2320547103881836,
	2.1242759227752686,
	2.34279727935791,
	2.3792335987091064,
	2.4060728549957275
	],
	"mean_nll": 2.2557598501443863,
	"perplexity": 9.542541457549426,
	"samples": [
	{
	"prompt": "What is 17 multiplied by 23? Answer with only the number.",
	"answer": "391",
	"generated_tokens": 4
	},
	{
	"prompt": "Return only valid JSON with keys \"status\" set to \"ok\" and \"count\" set to 3.",
	"answer": "{\"status\": \"ok\", \"count\": 3}",
	"generated_tokens": 13
	}
	],
	"evaluation_seconds": 49.92316593322903,
	"scope": "8 non-overlapping 512-token Wikitext test windows; sanity check only"
	},
	"quantization": "Quark AWQ signed INT4, symmetric groups of 128, BF16 activations",
	"calibration": {
	"dataset": "mit-han-lab/pile-val-backup",
	"split": "validation",
	"fingerprint": "fa7b9ce01fad1b22",
	"samples": 128,
	"sequence_length": 512,
	"selection": "first 128 rows, matching AMD's example default",
	"padding": "left, EOS token, matching Quark get_tokenizer",
	"input_sha256": "9bbae467c3c633d882fdfad4f398aea0515c211c26eafea482585ddb73995b15"
	},
	"audit": {
	"tensor_bytes": 19512618464,
	"weight_scale_tensors": 496,
	"preserved_mtp_tensors": 15,
	"exact_preserved_tensors": 349,
	"mtp_tensors_already_exported": 15,
	"mtp_tensors_added": 0,
	"copied_file_sha256": {
	"chat_template.jinja": "c3cf9e34abf4f9e36c2d72165aa9c132d3e2a725b6c2586aaa3a8af9d7a81041",
	"generation_config.json": "e70c136c1b78ddc1fb0905bac8e733a4dc448d4f852a5dd75143fffc70be550e",
	"merges.txt": "a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d",
	"preprocessor_config.json": "27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516",
	"tokenizer_config.json": "b11349aafa7cdc6a320767cf7ceb29ed82f7eda5d65e8e0819e76f0ce947bf27",
	"video_preprocessor_config.json": "7768af27c1fafa9cc9011c1dc20067e03f8915e03b63504550e11d5066986d13",
	"vocab.json": "ce99b4cb2983d118806ce0a8b777a35b093e2000a503ebde25853284c9dfa003",
	"tokenizer.json": "0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3"
	}
	},
	"round_trip_max_nll_difference": 0.0,
	"bf16_reference": {
	"label": "bf16",
	"nlls": [
	1.7963014841079712,
	2.3732309341430664,
	2.2346465587615967,
	2.221674919128418,
	2.0747108459472656,
	2.3078770637512207,
	2.3398988246917725,
	2.372152090072632
	],
	"mean_nll": 2.215061590075493,
	"perplexity": 9.161973380864124,
	"samples": [
	{
	"prompt": "What is 17 multiplied by 23? Answer with only the number.",
	"answer": "391",
	"generated_tokens": 4
	},
	{
	"prompt": "Return only valid JSON with keys \"status\" set to \"ok\" and \"count\" set to 3.",
	"answer": "{\"status\": \"ok\", \"count\": 3}",
	"generated_tokens": 13
	}
	],
	"evaluation_seconds": 6.864971877075732,
	"scope": "8 non-overlapping 512-token Wikitext test windows; sanity check only"
	},
	"perplexity_ratio_int4_over_bf16": 1.041537784586906
	}