Instructions to use d0xin/Swift-Qwen3.8-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use d0xin/Swift-Qwen3.8-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="d0xin/Swift-Qwen3.8-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("d0xin/Swift-Qwen3.8-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("d0xin/Swift-Qwen3.8-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use d0xin/Swift-Qwen3.8-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "d0xin/Swift-Qwen3.8-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d0xin/Swift-Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/d0xin/Swift-Qwen3.8-27B-FP8

SGLang

How to use d0xin/Swift-Qwen3.8-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "d0xin/Swift-Qwen3.8-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d0xin/Swift-Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "d0xin/Swift-Qwen3.8-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d0xin/Swift-Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use d0xin/Swift-Qwen3.8-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/d0xin/Swift-Qwen3.8-27B-FP8
```

Swift-Qwen3.8-27B-FP8 / README.md

d0xin

Rename related-models navigation header

04636a8 verified 2 days ago

preview code

Raw History Blame Contribute Delete

9.05 kB

	---
	license: other
	license_name: swift-open-license-1.0
	library_name: transformers
	pipeline_tag: image-text-to-text
	base_model: ukisai/Swift-Qwen3.8-27b
	base_model_relation: quantized
	tags:
	- qwen3_8
	- qwen3_5
	- fp8
	- compressed-tensors
	- sglang
	- speculative-decoding
	- dflash
	- reasoning
	- efficient-thinking
	- conversational
	---

	<!-- D0XIN_RELEASE_NAV_START -->
	> Related models: [all models](https://huggingface.co/d0xin)
	>
	> [Swift-1.5-Qwen3.8-27B-Uncensored-BF16](https://huggingface.co/d0xin/Swift-1.5-Qwen3.8-27B-Uncensored-BF16) · [Swift-1.5-Qwen3.8-27B-Uncensored-FP8](https://huggingface.co/d0xin/Swift-1.5-Qwen3.8-27B-Uncensored-FP8) · [Swift-1.5-Qwen3.8-27B-Uncensored-FP8-NInfer](https://huggingface.co/d0xin/Swift-1.5-Qwen3.8-27B-Uncensored-FP8-NInfer) · [Swift-1.5-Qwen3.8-Flash-Next-NVFP4-FP8PLE](https://huggingface.co/d0xin/Swift-1.5-Qwen3.8-Flash-Next-NVFP4-FP8PLE) · [Swift-1.5-Qwen3.8-Flash-Next-Rank2-Abliteration-Patch](https://huggingface.co/d0xin/Swift-1.5-Qwen3.8-Flash-Next-Rank2-Abliteration-Patch) · [Swift-Qwen3.8-27B-FP8](https://huggingface.co/d0xin/Swift-Qwen3.8-27B-FP8) · [Swift-Qwen3.8-27B-Uncensored-BF16](https://huggingface.co/d0xin/Swift-Qwen3.8-27B-Uncensored-BF16) · [Swift-Qwen3.8-27B-Uncensored-FP8](https://huggingface.co/d0xin/Swift-Qwen3.8-27B-Uncensored-FP8) · [Swift-Qwen3.8-27B-Uncensored-NVFP4-LocalHessian-ActivationHeadroom-NInfer](https://huggingface.co/d0xin/Swift-Qwen3.8-27B-Uncensored-NVFP4-LocalHessian-ActivationHeadroom-NInfer)
	<!-- D0XIN_RELEASE_NAV_END -->

	# Swift-Qwen3.8-27B-FP8

	FP8 quantization of
	[`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b).

	This is an independent community quantization and is **not an official
	UkisAI release**.

	The goal of this checkpoint is to preserve the behavior of Swift-Qwen3.8-27B
	while reducing VRAM requirements and enabling high-throughput inference with
	SGLang, including speculative decoding with the model's native MTP head or an
	external DFlash2 draft model.

	## Model summary

	- Upstream model: `ukisai/Swift-Qwen3.8-27b`
	- Architecture: `Qwen3_5ForConditionalGeneration`
	- Quantization format: `compressed-tensors`
	- Quantization scheme: `FP8_BLOCK`
	- Weight block size: `128 x 128`
	- Activations: dynamic FP8
	- Activation group size: `128`
	- Quantizer: `llmcompressor 0.13.0`
	- Declared context length: `262,144`
	- Checkpoint size: approximately `29 GB`
	- Native MTP components: retained

	The checkpoint was produced from the BF16 Swift-Qwen3.8-27B model rather than
	requantizing an already quantized derivative.

	## Quantization details

	The quantization process used the official Qwen3.8 FP8 configuration as a
	reference for the block-FP8 layout.

	Checkpoint audit:

	\| Item \| Count \|
	\|---\|---:\|
	\| Total checkpoint tensors \| 1,199 \|
	\| 2D weight tensors \| 617 \|
	\| FP8 quantization candidates \| 407 \|
	\| Effectively excluded / preserved modules \| 626 \|
	\| Incompatible FP8 candidates after validation \| 0 \|

	Matrices that are not compatible with the required `128 x 128` block
	structure were preserved instead of being forcibly quantized.

	The following classes of tensors were intentionally preserved where
	appropriate:

	- embeddings
	- `lm_head`
	- normalization parameters
	- non-2D weights
	- matrices whose dimensions are incompatible with the FP8 block layout

	The native MTP layers are retained. Compatible MTP projection matrices are
	quantized to FP8, while incompatible components remain unquantized.

	## Validation

	Validated locally on:

	- NVIDIA RTX PRO 6000 Blackwell 96 GB
	- SGLang `0.5.19.dev135+ga4ffb996d`
	- `compressed-tensors 0.18.0`
	- CUDA-capable Linux deployment
	- single-GPU tensor parallelism (`TP=1`)

	SGLang successfully loads the checkpoint as:

	```text
	type=Qwen3_5ForConditionalGeneration
	quant=compressed-tensors
	```

	Observed target-model weight memory during loading:

	```text
	28.47 GB
	```

	OpenAI-compatible `/v1/chat/completions` inference was validated successfully.

	Multimodal inference has not yet been separately benchmarked for this
	quantized checkpoint.

	## Performance

	All measurements below are local measurements from a single
	NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.

	They are intended to document this deployment, not to serve as standardized
	cross-model benchmarks.

	### Fixed 4,096-token generation

	Same prompt and generation setup for all configurations:

	\| Configuration \| Median throughput \|
	\|---\|---:\|
	\| Swift FP8, target model only \| 50.45 tok/s \|
	\| Swift FP8 + native NEXTN/MTP \| 103.38 tok/s \|
	\| Swift FP8 + DFlash2, 8 draft tokens \| 132.62 tok/s \|

	Measured DFlash2 runs:

	```text
	131.24 tok/s
	132.65 tok/s
	132.62 tok/s
	median: 132.62 tok/s
	```

	Compared with target-only generation, DFlash2 produced approximately
	2.63x higher output throughput in this test.

	Compared with native NEXTN/MTP, DFlash2 was approximately 28% faster.

	The DFlash2 draft model is external and is not included in this repository.

	## Reasoning-heavy agent test

	A separate local A/B test used the same diagnostic prompt, sampling
	parameters, seed, and reasoning setting for both systems.

	The prompt asked the model to diagnose an intermittently slow
	OpenAI-compatible inference deployment with high GPU utilization,
	large KV cache, speculative decoding, variable context sizes and
	concurrency-sensitive latency.

	\| Metric \| Previous Qwen FP8 production \| Swift FP8 + DFlash2 \|
	\|---\|---:\|---:\|
	\| Wall time \| 193.48 s \| 139.63 s \|
	\| Prompt tokens \| 229 \| 229 \|
	\| Reasoning tokens \| 14,838 \| 10,716 \|
	\| Completion tokens \| 22,848 \| 16,170 \|
	\| Finish reason \| stop \| stop \|
	\| Effective completion throughput \| 118.09 tok/s \| 115.80 tok/s \|

	Observed change:

	- wall-clock time: approximately -27.8%
	- reasoning tokens: approximately -27.8%
	- completion tokens: approximately -29.2%

	The main benefit in this test was not higher raw per-token throughput.
	Swift reached a similarly useful diagnostic answer with substantially fewer
	reasoning and completion tokens.

	This is a local workload test and should not be interpreted as a standardized
	quality benchmark.

	## SGLang usage

	### Basic serving

	```bash
	python -m sglang.launch_server \
	--model-path /path/to/Swift-Qwen3.8-27B-FP8 \
	--served-model-name Swift-Qwen3.8-27B-FP8 \
	--host 0.0.0.0 \
	--port 30000 \
	--attention-backend flashinfer \
	--reasoning-parser qwen3 \
	--tool-call-parser qwen3_coder
	```

	### Native NEXTN / MTP speculative decoding

	The retained native MTP head can be used with SGLang:

	```bash
	python -m sglang.launch_server \
	--model-path /path/to/Swift-Qwen3.8-27B-FP8 \
	--served-model-name Swift-Qwen3.8-27B-FP8 \
	--host 0.0.0.0 \
	--port 30000 \
	--attention-backend flashinfer \
	--reasoning-parser qwen3 \
	--tool-call-parser qwen3_coder \
	--speculative-algorithm NEXTN \
	--speculative-num-steps 3 \
	--speculative-eagle-topk 1 \
	--speculative-num-draft-tokens 4
	```

	In the validated SGLang build, NEXTN is internally represented through the
	EAGLE speculative-decoding path.

	### DFlash2 speculative decoding

	Best local throughput was obtained with a compatible external DFlash2 draft
	checkpoint:

	```bash
	python -m sglang.launch_server \
	--model-path /path/to/Swift-Qwen3.8-27B-FP8 \
	--served-model-name Swift-Qwen3.8-27B-FP8 \
	--host 0.0.0.0 \
	--port 30000 \
	--attention-backend flashinfer \
	--reasoning-parser qwen3 \
	--tool-call-parser qwen3_coder \
	--speculative-algorithm DFLASH \
	--speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
	--speculative-num-draft-tokens 8
	```

	The DFlash2 weights are not redistributed in this repository.

	Because speculative decoding verifies proposed tokens against the target
	model, the external draft model affects acceptance rate and speed rather than
	replacing the target model's token distribution.

	## Notes

	This repository contains the quantized target checkpoint only.

	It does not include:

	- a DFlash2 draft checkpoint
	- the original BF16 Swift checkpoint
	- SGLang runtime binaries or containers

	Performance depends heavily on GPU architecture, SGLang version, attention
	backend, context length, concurrency, KV-cache configuration and speculative
	decoding parameters.

	## License

	This checkpoint is derived from:

	[`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)

	and follows the Swift Open License v1.0 applicable to the upstream model.

	Please refer to the upstream repository and its license text for the
	authoritative licensing terms.

	No additional rights to the upstream model are granted by this repository.

	## Attribution

	Original model:

	- UkisAI
	- `ukisai/Swift-Qwen3.8-27b`

	FP8 conversion, validation and local performance measurements for this
	repository were performed independently by the repository maintainer.

	## Citation

	For the underlying Swift model, please cite or reference the upstream project:

	```bibtex
	@misc{swift-qwen3.8-27b,
	title = {Swift-Qwen3.8-27B},
	author = {UkisAI},
	year = {2026},
	url = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
	}
	```