Instructions to use leoncca/Qwen3.8-27B-Huihui-Mixed-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use leoncca/Qwen3.8-27B-Huihui-Mixed-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="leoncca/Qwen3.8-27B-Huihui-Mixed-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("leoncca/Qwen3.8-27B-Huihui-Mixed-FP8")
model = AutoModelForMultimodalLM.from_pretrained("leoncca/Qwen3.8-27B-Huihui-Mixed-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use leoncca/Qwen3.8-27B-Huihui-Mixed-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leoncca/Qwen3.8-27B-Huihui-Mixed-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-Huihui-Mixed-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/leoncca/Qwen3.8-27B-Huihui-Mixed-FP8

SGLang

How to use leoncca/Qwen3.8-27B-Huihui-Mixed-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "leoncca/Qwen3.8-27B-Huihui-Mixed-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-Huihui-Mixed-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "leoncca/Qwen3.8-27B-Huihui-Mixed-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-Huihui-Mixed-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use leoncca/Qwen3.8-27B-Huihui-Mixed-FP8 with Docker Model Runner:
```
docker model run hf.co/leoncca/Qwen3.8-27B-Huihui-Mixed-FP8
```

Qwen3.8-27B Huihui Mixed FP8 — 32.92 GB / 30.66 GiB

High-quality mixed E4M3 block-128 FP8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated. This is a community quantization, not an official Qwen or Huihui checkpoint.

Versions

main / v2-739e3c5b: current build from upstream revision 739e3c5b89849f6c238ce1e5b70008612ae42cdd.
v1-d42ca897: preserved original release from upstream revision d42ca8978c5a66e92c3446d46e8adfe03ef692ff, with the full 4x V100 validation record retained in that revision's model card.

Download a stable version explicitly:

hf download leoncca/Qwen3.8-27B-Huihui-Mixed-FP8 \
  --revision v2-739e3c5b \
  --local-dir Qwen3.8-27B-Huihui-Mixed-FP8-v2

Use --revision v1-d42ca897 for the original V100-validated checkpoint.

Precision policy

336 large language-model GEMM weights use dynamic-activation E4M3 FP8 with 128x128 weight blocks and 336 BF16 inverse scales.
863 tensors remain bit-identical to the pinned BF16 source, totaling 10,264,372,704 bytes.
The complete vision tower (333 tensors) and native MTP branch (15 tensors) remain BF16.
Q/K/V/O in all 16 full-attention layers (64 tensors) remain BF16.
Embeddings, LM head, norms, biases, and state-sensitive GDN/SSM tensors remain BF16.
KV-cache precision is a runtime choice and is not encoded in this checkpoint.

What changed in v2

The updated Huihui source changed 28 tensors across 6 of 18 source shards. Under this precision policy, that produces exactly 52 changed checkpoint tensors relative to v1:

24 E4M3 weights: 14 MLP down_proj and 10 linear-attention out_proj;
24 corresponding BF16 weight_scale_inv tensors;
4 protected BF16 full-attention o_proj tensors.

The other 12 output shards are byte-identical to v1. All six affected shards were regenerated from the pinned v2 BF16 source; unchanged tensors within those shards were also recomputed and compared to prevent a partial-overlay artifact.

Current validation status

The v2 checkpoint passed the release artifact gates:

18 shards, 1,535 indexed tensors, and 32,916,379,104 tensor bytes;
336 E4M3 weights, 336 BF16 scales, and 863 protected BF16 tensors;
863/863 protected tensors bit-identical to the v2 BF16 source;
Vision 333/333, native MTP 15/15, and full-attention QKVO 64/64;
reference FP8 conversion reproduced the official Qwen algorithm with zero byte mismatch on the frozen reference tensors;
exactly 52 candidate tensors differ from v1, matching the 28-tensor upstream change after FP8 scale expansion.

Exact 1Cat-vLLM / SM70 runtime validation for v2 is pending an available 4x V100 window. The historical quality, speed, long-context, and production results in v1-d42ca897 apply only to v1 and are not transferred to v2.

Serving notes

The accepted v1 V100 contract used TP4, FP16 compute, FP16 KV, Flash-V100, prefix caching, and Mamba cache alignment. Treat that as a compatibility starting point for v2, not as a completed v2 runtime acceptance claim. Verify the selected FP8 kernel, attention path, KV dtype, and numerical output on your own runtime before production use.

Integrity and limitations

SHA256SUMS covers every published file other than itself. The release also includes the pinned source manifest, precision manifest, per-tensor FP8 error metrics, static audit, and upstream-refresh summary.

This release inherits the capabilities and risks of the Huihui source. It may produce inaccurate, unsafe, illegal, or otherwise harmful content. Users are responsible for prompts, outputs, downstream actions, access control, legal compliance, and deployment-appropriate safeguards. The weights are provided as-is without warranty.