Instructions to use inferRouter/Qwen3.6-35B-A3B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inferRouter/Qwen3.6-35B-A3B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inferRouter/Qwen3.6-35B-A3B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inferRouter/Qwen3.6-35B-A3B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("inferRouter/Qwen3.6-35B-A3B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inferRouter/Qwen3.6-35B-A3B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inferRouter/Qwen3.6-35B-A3B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inferRouter/Qwen3.6-35B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inferRouter/Qwen3.6-35B-A3B-NVFP4

SGLang

How to use inferRouter/Qwen3.6-35B-A3B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inferRouter/Qwen3.6-35B-A3B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inferRouter/Qwen3.6-35B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inferRouter/Qwen3.6-35B-A3B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inferRouter/Qwen3.6-35B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inferRouter/Qwen3.6-35B-A3B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/inferRouter/Qwen3.6-35B-A3B-NVFP4
```

slavikz commited on May 12

Commit

8a59276

verified ·

1 Parent(s): d484e9d

vllm_patches: rename PATCH_README.md → README.md (delete step)

Browse files

Files changed (1) hide show

vllm_patches/PATCH_README.md +0 -59

vllm_patches/PATCH_README.md DELETED Viewed

@@ -1,59 +0,0 @@
-# vLLM patches for the `fp8-head` branch
-This folder contains the local vLLM overlay used to serve the
-`fp8-head` branch of
-[`inferRouter/Qwen3.6-35B-A3B-NVFP4`](https://huggingface.co/inferRouter/Qwen3.6-35B-A3B-NVFP4)
-when the installed vLLM build does not yet include compressed-tensors FP8
-`lm_head` dispatch.
-The `main` branch does not need this overlay because its `lm_head` and
-`embed_tokens` are BF16. This 35 B repository intentionally does not ship
-an FP8-embedding branch.
-| File | Purpose |
-|---|---|
-| `apply_ct_fp8_lmhead_patch.py` | Idempotent patcher for a vLLM source tree or installed package. Adds the compressed-tensors dispatch needed for FP8 `lm_head`, keeps the scale companion loader working, and skips patch points that are already present upstream. |
-| `compressed_tensors_embedding.py` | Companion module shared with the broader CT-FP8 patch bundle. It implements the FP8 embedding experiment, but no FP8-embedding artifact is published in this 35 B repo. |
-| `Dockerfile.turboquant` | Example overlay Dockerfile on top of a base image that already includes TurboQuant hybrid support (vLLM PR #39931) and upstream FP8 lm_head work. |
-## Quick Start
-Patch an installed vLLM package:
-```bash
-curl -O https://huggingface.co/inferRouter/Qwen3.6-35B-A3B-NVFP4/resolve/main/vllm_patches/apply_ct_fp8_lmhead_patch.py
-curl -O https://huggingface.co/inferRouter/Qwen3.6-35B-A3B-NVFP4/resolve/main/vllm_patches/compressed_tensors_embedding.py
-python3 apply_ct_fp8_lmhead_patch.py /usr/local/lib/python3.12/site-packages
-```
-Or patch a source checkout:
-```bash
-git clone --depth 1 --branch v0.20.2 https://github.com/vllm-project/vllm.git vllm-src
-python3 apply_ct_fp8_lmhead_patch.py ./vllm-src
-```
-For `--kv-cache-dtype turboquant_4bit_nc`, the base vLLM build must also
-include TurboQuant hybrid attention support, merged in vLLM PR #39931.
-Without that PR, serve with `--kv-cache-dtype fp8_e4m3`.
-The patcher is idempotent. It detects PR #41000-style `lm_head` wire-up
-and skips already-applied edits, so it works against both stock v0.20.x
-trees and newer images that already include part of the upstream FP8 work.
-## Expected Startup Signal
-When the patch is active and the `fp8-head` branch is loaded, the vLLM log
-should contain a compressed-tensors FP8 kernel selection similar to:
-```text
-Selected CutlassFp8BlockScaledMMKernel for CompressedTensorsW8A8Fp8
-```
-For the `main` branch, this line is not expected because `lm_head` is BF16.
-## License
-The patcher and companion module are released under Apache 2.0, the same
-license as this repository.