Instructions to use prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8")
model = AutoModelForMultimodalLM.from_pretrained("prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8

SGLang

How to use prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8 with Docker Model Runner:
```
docker model run hf.co/prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8
```

Qwen-Image-2.1-PE-T2I-FP8 / README.md

prithivMLmods

Update README.md

6f49d7d verified 6 days ago

preview code

Raw

History Blame Contribute Delete

7.12 kB

	---
	license: other
	license_name: qwen-research
	license_link: https://huggingface.co/Qwen/Qwen-Image-2.1-PE-T2I/blob/main/LICENSE
	base_model:
	- Qwen/Qwen-Image-2.1-PE-T2I
	library_name: transformers
	tags:
	- text-generation-inference
	- vllm
	- 8-bit
	- qwen
	- prompt-rewriting
	- image-editing
	language:
	- en
	pipeline_tag: text-generation
	---

	# Qwen-Image-2.1-PE-T2I-FP8

	> Qwen-Image-2.1-PE-T2I-FP8 is an FP8 dynamic-quantized build of [Qwen/Qwen-Image-2.1-PE-T2I](https://huggingface.co/Qwen/Qwen-Image-2.1-PE-T2I), the text-to-image prompt rewriting model for [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1). The base model is a fine-tuned Qwen3.5-VL 9B that turns a brief image request in any language into a detailed English prompt plus a recommended aspect ratio. This checkpoint was compressed with [llm-compressor](https://github.com/vllm-project/llm-compressor) using the FP8_DYNAMIC scheme, stored in the compressed-tensors format, and is intended to be served with [vLLM](https://github.com/vllm-project/vllm). The Linear layers are quantized to FP8 weights with dynamic per-token FP8 activations, which lowers weight memory and improves serving throughput relative to the BF16 original. The lm_head, embedding layers, vision modules, and linear attention layers are excluded and kept in their original precision. No calibration data is required, because activation scales are computed at runtime.

	> [!NOTE]
	System Prompt — https://huggingface.co/Qwen/Qwen-Image-2.1-PE-T2I/blob/main/system_prompt.txt

	## Quantization Details

	\| Property \| Value \|
	\|---\|---\|
	\| Base model \| `Qwen/Qwen-Image-2.1-PE-T2I` \|
	\| Architecture \| Qwen3.5-VL 9B (fine-tuned) \|
	\| Quantization tool \| `llm-compressor` \|
	\| Modifier \| `QuantizationModifier` \|
	\| Scheme \| `FP8_DYNAMIC` \|
	\| Checkpoint format \| `compressed-tensors` \|
	\| Quantized modules \| `Linear` layers \|
	\| Excluded modules \| `lm_head`, `embed_tokens`, `visual`, `linear_attn` \|
	\| Calibration data \| Not required \|
	\| Inference engine \| vLLM \|

	### Recipe

	```yaml
	default_stage:
	default_modifiers:
	QuantizationModifier:
	targets: [Linear]
	ignore: ['re:.lm_head', 're:.embed_tokens$', 're:.visual.', 're:.model.visual.',
	're:.linear_attn.']
	scheme: FP8_DYNAMIC
	bypass_divisibility_checks: false
	requires_calibration_data: false
	```

	### Reproduction

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	from llmcompressor import oneshot

	model_id = "Qwen/Qwen-Image-2.1-PE-T2I"
	save_dir = "Qwen-Image-2.1-PE-T2I-FP8"

	model = AutoModelForCausalLM.from_pretrained(
	model_id, dtype=torch.bfloat16, device_map="auto"
	)
	tokenizer = AutoTokenizer.from_pretrained(model_id)

	# recipe.yaml is the recipe shown above
	oneshot(model=model, recipe="recipe.yaml")

	model.save_pretrained(save_dir, save_compressed=True)
	tokenizer.save_pretrained(save_dir)
	```

	## Quick Start

	### Installation

	```bash
	pip install vllm openai huggingface_hub
	```

	Use a recent vLLM release with support for Qwen3.5-VL and compressed-tensors FP8 checkpoints. Native FP8 compute requires a GPU with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell). On older GPUs vLLM falls back to weight-only FP8 kernels.

	### Serve with vLLM

	```bash
	vllm serve prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8 \
	--max-model-len 32768
	```

	Do not enable a reasoning parser, so the thinking block is returned in the response content and can be split from the JSON answer as shown below.

	### Query the Server

	```python
	import json

	import huggingface_hub
	from openai import OpenAI

	model_id = "prithivMLmods/Qwen-Image-2.1-PE-T2I-FP8"
	client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

	# System prompt shipped with the base model
	sys_prompt_path = huggingface_hub.hf_hub_download(
	"Qwen/Qwen-Image-2.1-PE-T2I", "system_prompt.txt"
	)
	system_prompt = open(sys_prompt_path).read().strip()

	user_prompt = "一只在雨中弹吉他的柯基"

	response = client.chat.completions.create(
	model=model_id,
	messages=[
	{"role": "system", "content": system_prompt},
	{"role": "user", "content": user_prompt},
	],
	max_tokens=16256,
	temperature=1.0,
	top_p=0.95,
	seed=42,
	extra_body={"top_k": 20},
	)

	gen = response.choices[0].message.content

	# Split thinking from the answer
	thinking, _, answer = gen.partition("</think>")
	result = json.loads(answer.strip())
	print(result)
	# {"rewritten_prompt": "<long detailed English prompt>", "wh_ratio": "16:9"}
	```

	### Integration with Diffusers

	```python
	import torch
	from diffusers import QwenImage21Pipeline

	WH_RATIO_TO_SIZE = {
	"1:1": (2048, 2048), "4:3": (2400, 1792), "3:4": (1792, 2400),
	"3:2": (2528, 1696), "2:3": (1696, 2528), "16:9": (2752, 1536),
	"9:16": (1536, 2752),
	}

	# `result` from the vLLM call above
	prompt = result["rewritten_prompt"]
	width, height = WH_RATIO_TO_SIZE.get(result["wh_ratio"], (2048, 2048))

	pipe = QwenImage21Pipeline.from_pretrained(
	"Qwen/Qwen-Image-2.1", torch_dtype=torch.bfloat16
	).to("cuda")

	image = pipe(
	prompt=prompt,
	width=width, height=height,
	num_inference_steps=40,
	generator=torch.Generator("cuda").manual_seed(42),
	).images[0]

	image.save("rewritten_t2i.png")
	```

	## Output Format

	The model emits a JSON object after a `<think>` reasoning block:

	```json
	{
	"rewritten_prompt": "<long detailed English prompt describing the finished image>",
	"wh_ratio": "16:9"
	}
	```

	\| Field \| Description \|
	\|---\|---\|
	\| `rewritten_prompt` \| Expanded English prompt to pass to the image generation model \|
	\| `wh_ratio` \| Recommended aspect ratio for rendering \|

	The aspect ratios used in the Diffusers example map to the following output sizes:

	\| `wh_ratio` \| Width x Height \|
	\|---\|---\|
	\| `1:1` \| 2048 x 2048 \|
	\| `4:3` \| 2400 x 1792 \|
	\| `3:4` \| 1792 x 2400 \|
	\| `3:2` \| 2528 x 1696 \|
	\| `2:3` \| 1696 x 2528 \|
	\| `16:9` \| 2752 x 1536 \|
	\| `9:16` \| 1536 x 2752 \|

	## Notes

	- FP8 quantization introduces small numerical differences from the BF16 original, so rewritten prompts may differ slightly in wording. Validate output quality on your own generation workloads before replacing the base model.
	- Always parse the answer with a JSON loader and handle malformed output, since sampling at `temperature=1.0` can occasionally produce invalid JSON.
	- Input requests can be written in any language, and the rewritten prompt is always English.

	## License

	This model inherits the license of the base model and is distributed under the [Qwen Research License Agreement](https://huggingface.co/Qwen/Qwen-Image-2.1-PE-T2I/blob/main/LICENSE).

	## Acknowledgements

	Base model by the Qwen team: [Qwen/Qwen-Image-2.1-PE-T2I](https://huggingface.co/Qwen/Qwen-Image-2.1-PE-T2I). Quantization with [llm-compressor](https://github.com/vllm-project/llm-compressor). Serving with [vLLM](https://github.com/vllm-project/vllm). See the [Qwen-Image-2.1 GitHub repo](https://github.com/QwenLM/Qwen-Image-2.1) and [blog](https://qwen.ai/blog?id=qwen-image-2.1) for details on the full pipeline.