Instructions to use leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ")
model = AutoModelForMultimodalLM.from_pretrained("leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ

SGLang

How to use leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ with Docker Model Runner:
```
docker model run hf.co/leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ
```

Qwen3.5-122B-A10B Heretic Selective AWQ

Checkpoint size: 82.08 GB (76.45 GiB) of safetensors weights.

This is an activation-aware, selectively scoped AWQ checkpoint of trohrbaugh/Qwen3.5-122B-A10B-heretic, built for and runtime-validated on 4x NVIDIA Tesla V100 PCIe 32 GB (SM70).

The release keeps the model's vision path, attention paths, shared expert, router, layer 0, embeddings, norms, LM head, and native MTP tensors in their source BF16 representation. Only the routed language experts in layers 1-47 are packed as AWQ W4A16. The checkpoint is approximately 82.1 GB and was validated with TP4 without CPU weight offload.

V100 validation used a local 1Cat-vLLM SM70 AWQ runtime. This release does not claim drop-in compatibility with stock vLLM, Transformers, or every AWQ loader. The checkpoint metadata and tensors are standard AWQ GEMM where possible, but Qwen3.5 MoE, multimodal, native-MTP, and SM70 support remains runtime-dependent.

This is a community quantization. It is not an official Qwen release and is not published by the author of the Heretic base model.

Frozen source revisions

Component	Repository	Revision
Heretic BF16 mother	`trohrbaugh/Qwen3.5-122B-A10B-heretic`	`356e5abe1fbc83bb3613b46e470f6e2a5565ca47`
Native MTP tensors	`Qwen/Qwen3.5-122B-A10B`	`dc4d348443bc740c68e2d77492492c11606384d5`
Quantizer	`modelcloud/GPTQModel` 7.3.2	`f6f1ab12fe8115f84c52253871ca039ac6c425ee` plus the included build patch

Exactly 785 official mtp.* tensors were grafted from the pinned Qwen source and retained in BF16. See mtp-graft-manifest.json for tensor-level provenance.

Quantization recipe

Item	Value
Method	Activation-aware AWQ, GEMM W4A16
Weight bits	4
Group size	128
Zero point	Asymmetric, enabled
Activation order	Disabled (`desc_act=false`)
Packed dtype	INT32 `qweight` and `qzeros`
Scale dtype	BF16
Quantized scope	Routed expert `gate_proj`, `up_proj`, `down_proj`, language layers 1-47
Quantized modules	47 x 256 x 3 = 36,096
Protected scope	Vision, full/linear attention, router, shared expert, layer 0, embeddings, norms, LM head, native MTP

The calibration run used a frozen 320-conversation corpus with 190,590 active tokens: 96 English reference samples, 64 Chinese reference samples, 64 code samples, 64 structured/tool/math samples, and 32 image-conditioned samples. Its pinned inputs were derived from WikiText-103, Chinese Wikipedia, CodeParrot-clean, and held-out COCO assets. The corpus itself is not included; the source manifests and corpus digest are included for provenance.

Natural router coverage was supplemented only for 46 expert/layer pairs that were not reached by the frozen calibration pass. This selective routing bypass was used to collect activation statistics; it did not exempt those expert weights from quantization. All 36,096 target modules were packed.

Precision and artifact audit

The final artifact audit reports:

36,096/36,096 target modules packed;
2,621 protected tensors unchanged byte-for-byte;
333 vision tensors and 785 MTP tensors retained;
10 weight shards fully SHA-256 hashed;
qweight, qzeros, and scales stored as INT32, INT32, and BF16;
simulated V100 BF16-to-FP16 loading of every AWQ scale found zero non-finite values, zero FP16 overflows, and zero nonzero values underflowing to zero.

The last point is a safety audit, not a claim that BF16-to-FP16 casting is mathematically lossless. The checkpoint itself was not globally converted to FP16.

See artifacts/final-output-audit.json, artifacts/quantization-report.json, quantization-provenance.json, and SHA256SUMS.

V100 TP4 performance

Validated profile:

4x Tesla V100 PCIe 32 GB, tensor parallel size 4;
no CPU weight offload;
FP16 KV cache, one active sequence;
greedy streaming generation, 256 output tokens, MTP disabled for the table;
local 1Cat-vLLM 1.2.2 SM70/TurboMind AWQ path.

Actual prompt tokens	Prefill/TTFT-derived tok/s	Decode tok/s
8,051	3,482.74	58.01
64,052	1,817.16	49.00
128,053	1,166.45	41.77

These are single-request local engineering measurements, not standardized cross-model benchmark scores. The raw result JSON files are included under benchmarks/.

A separate 262,144-token engine profile with native MTP1 also loaded and ran successfully. The longest completed request in that acceptance pass contained 211,042 prompt tokens and completed without OOM or worker restart. This is not a claim that a near-full 262,144-token prompt was tested.

Local quality checks and known limits

The following local deterministic checks passed:

7/7 short agent/tool behavior cases without repetition or length-cap hits;
8/8 exact records plus the correct aggregate at 33,137 prompt tokens;
33/33 completed frozen NeedleBench/RULER retrieval cases around 127K-131K;
32/32 authoritative-vs-foil records at 123,457 prompt tokens;
structured tool calls and basic single-image inference.

Important failures and limits are retained rather than hidden:

at 120,369 prompt tokens, all 8 exact records were retrieved but the derived sum was wrong (173 instead of 137);
the strict multimodal fixture passed 5/6 cases; one arrow-route case failed;
multi-hop long-context reasoning was inconsistent and some thinking-enabled reruns reached their output cap;
no standardized paired coding/agent evaluation established quality non-inferiority to the BF16 mother or to a retained 27B full-precision model.

Accordingly, this checkpoint was retained locally as a secondary cold-standby, not promoted as the preferred general-purpose model. Parameter count alone must not be treated as proof that this quantization is better than a smaller full-precision model.

Download

hf download leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ \
  --local-dir Qwen3.5-122B-A10B-Heretic-Selective-AWQ

Verify the download from inside the model directory:

sha256sum -c SHA256SUMS

Runtime flags depend on the inference engine. Do not assume that a loader which supports dense AWQ also supports this selectively quantized Qwen3.5 MoE multimodal checkpoint on SM70.

License and use

The included model files are released under Apache-2.0, following the source repositories. Review the upstream model cards and license before use.

The Heretic base intentionally changes refusal behavior. This checkpoint is not safety-tuned, and its outputs may be inaccurate, offensive, or unsafe. Downstream users are responsible for evaluation, safeguards, and compliance in their own deployment context.

Acknowledgements

Downloads last month: 109

Safetensors

Model size

125B params

Tensor type

BF16

I32

Model tree for leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ

Base model

trohrbaugh/Qwen3.5-122B-A10B-heretic

Quantized

(7)

this model

Collection including leoncca/Qwen3.5-122B-A10B-Heretic-Selective-AWQ

V100-Validated Quantizations

Collection

Models built for and runtime-validated on NVIDIA Tesla V100/SM70, with frozen recipes, checksums, benchmarks, and explicit quality limits. • 7 items • Updated 1 day ago • 3