Instructions to use QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ")
model = AutoModelForMultimodalLM.from_pretrained("QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ

SGLang

How to use QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ with Docker Model Runner:
```
docker model run hf.co/QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ
```

Doesn't work with vllm

by mobicham - opened Oct 6, 2025

Discussion

mobicham

Oct 6, 2025

Hey, thanks for the upload!
It doesn't seem to work with vllm following the readme. Does it require building vllm from master?

tclf90

QuantTrio org Oct 7, 2025

•

edited Oct 7, 2025

This repo can be served with the default build. If your machine can't run, please consider
(1) make sure you really created a new python venv
(2) try vllm serve the original qwen3 vl a3b, see if it works, if not, your py vllm environment is not correctly installed

mobicham

Oct 8, 2025

Yeah the issue was uv pip, switching to pip fixes the issue (+ installing other dependencies).
Unfortunately it runs out of vram on the L40s (48GB), normally the 4-bit version should fit in a 24GB GPU.

tclf90

QuantTrio org Oct 8, 2025

Glad to hear that

"Unfortunately it runs out of vram on the L40s (48GB)"
Have you tried setting "--limit-mm-per-prompt.video 0" as suggested from vLLM Qwen3-VL Usage Guide

tclf90

QuantTrio org Oct 8, 2025

•

edited Oct 8, 2025

I happen to have a sm89 48GB device next to me, so I just ran a quick check,

install using:

uv venv
source .venv/bin/activate

# Install vLLM >=0.11.0
uv pip install -U vllm

# Install Qwen-VL utility library (recommended for offline inference)
uv pip install qwen-vl-utils==0.0.14

serve using

export TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
export OMP_NUM_THREADS=4
vllm serve \
    PATH/QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ \
    --served-model-name MY_MODEL \
    --swap-space 4 \
    --max-num-seqs 8 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.95 \
    --tensor-parallel-size 1 \
    --distributed-executor-backend mp \
    --trust-remote-code \
    --disable-log-requests \
    --host 0.0.0.0 \
    --port 8000

Having no issue whatsoever. Just for your reference.

mobicham changed discussion status to closed Oct 14, 2025

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment