Instructions to use nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8")
model = AutoModelForMultimodalLM.from_pretrained("nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8

SGLang

How to use nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 with Docker Model Runner:
```
docker model run hf.co/nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8
```

SM120 support

by InformaticsSolutions - opened Jul 13

Discussion

InformaticsSolutions

Jul 13

Hi,
thank you so much for creating and sharing this quant. I am trying to load it on 2xRTX6000 Pro (SM120) and i'm getting this error:

RuntimeError: dispatch_scaled_mm, /home/user1/vllm/csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_helper.hpp:34, Int8 not supported on SM120. Use FP8 quantization instead, or run on older arch (SM < 100)

I am confused about this Int8 not being supported on SM120: this quant https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/W8A16-GS128 works great. Is there something about the quark quant making it incompatible? Or am i doing something wrong?
My setup:

vllm version 0.23.0
cuda  13.2, V13.2.78
NVIDIA driver 595.58.03

thanks a lot!

nameistoken

Owner Jul 14

SM120 does have INT8 capabilities in general, but vLLM currently does not support the W8A8 INT8 scaled_mm kernel path on SM120.
In vLLM's SM120 scaled_mm implementation, the FP8 kernel is registered, but the INT8 dispatch is explicitly set to nullptr, so W8A8 INT8 checkpoints fail with that error.
Weight-only INT8 models such as W8A16/AutoRound use a different kernel path, which is why they can still work on the same GPU.

InformaticsSolutions

Jul 14

thank you.

InformaticsSolutions changed discussion status to closed Jul 14

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment