Instructions to use jaehyeono/Qwen3.8-27B-W3A16-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jaehyeono/Qwen3.8-27B-W3A16-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="jaehyeono/Qwen3.8-27B-W3A16-GPTQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("jaehyeono/Qwen3.8-27B-W3A16-GPTQ")
model = AutoModelForMultimodalLM.from_pretrained("jaehyeono/Qwen3.8-27B-W3A16-GPTQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jaehyeono/Qwen3.8-27B-W3A16-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jaehyeono/Qwen3.8-27B-W3A16-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jaehyeono/Qwen3.8-27B-W3A16-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/jaehyeono/Qwen3.8-27B-W3A16-GPTQ

SGLang

How to use jaehyeono/Qwen3.8-27B-W3A16-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jaehyeono/Qwen3.8-27B-W3A16-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jaehyeono/Qwen3.8-27B-W3A16-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jaehyeono/Qwen3.8-27B-W3A16-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jaehyeono/Qwen3.8-27B-W3A16-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use jaehyeono/Qwen3.8-27B-W3A16-GPTQ with Docker Model Runner:
```
docker model run hf.co/jaehyeono/Qwen3.8-27B-W3A16-GPTQ
```

Qwen3.8-27B-W3A16-GPTQ

3-bit weight-quantized version of Qwen/Qwen3.8-27B, produced with GPTQ via llm-compressor.

The vision tower is left in bf16 — only the language decoder is quantized. The model remains a fully functional vision-language model; image+text inference works as with the base model.


Weights	INT3, group size 128, symmetric
Activations	bf16 (unquantized)
Format	`compressed-tensors` / `pack-quantized`
Quantized	language decoder (256 Linear modules)
Kept in bf16	vision tower, `lm_head`, linear-attention projections
Size	55.6 GB → 22.8 GB (~2.4x smaller)

Requirements

This model uses the qwen3_5 architecture, which requires transformers >= 5.14:

pip install "transformers>=5.14" compressed-tensors

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model_id = "jaehyeono/Qwen3.8-27B-W3A16-GPTQ"

model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype="auto", device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image in one sentence."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[Image.open("example.jpg")], return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=64)
print(processor.decode(out[0], skip_special_tokens=True))

Quantization details

Quantized with GPTQModifier from llm-compressor 0.13.0 (compressed-tensors 0.18.0).

Parameter	Value
Bits	3
Group size	128
Symmetric	true
Strategy	group
Activation ordering	static
Dampening fraction	0.01
Observer	`memoryless_minmax`
Calibration	128 image–caption pairs from Flickr30k
Seed	42

The full recipe is included as recipe.yaml.

Why the vision tower is not quantized

GPTQ needs activation statistics for every module it touches. The vision tower contributes a small fraction of total parameters (depth 27, hidden 1152) compared to the language decoder (64 layers, hidden 5120), so excluding it costs little compression while avoiding degradation in the visual pathway. Linear-attention projections and lm_head are likewise excluded, per common practice for these layers.

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B.

Downloads last month: 173

Safetensors

Model size

10B params

Tensor type

I32

BF16

Model tree for jaehyeono/Qwen3.8-27B-W3A16-GPTQ

Base model

Qwen/Qwen3.8-27B

Quantized

(1067)

this model