Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,163 @@
+---
+base_model: Qwen/Qwen3.5-35B-A3B
+library_name: transformers
+license: apache-2.0
+tags:
+- qwen3.5
+- moe
+- gptq
+- quantized
+- 4-bit
+- vllm
+- vision
+- multimodal
+- mtp
+---
+
+# Qwen3.5-35B-A3B GPTQ 4-bit
+
+GPTQ 4-bit quantization of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), a 35B-parameter Mixture-of-Experts (MoE) multimodal model with 3B activated parameters per token.
+
+Includes full vision encoder and MTP (Multi-Token Prediction) module for image understanding and speculative decoding support.
+
+## Model Overview
+
+- **Architecture**: Qwen3_5MoeForConditionalGeneration (multimodal: text + vision)
+- **Total parameters**: ~35B
+- **Activated parameters**: ~3B per token (8 of 256 experts selected per token)
+- **Layers**: 40 (30 linear attention + 10 full attention, repeating 3:1 pattern)
+- **Experts**: 256 per layer + 1 shared expert per layer
+- **Context length**: 262,144 tokens
+- **Vision encoder**: 27-block ViT (1152 hidden, 16x16 patches), BF16
+- **MTP module**: 1-layer speculative decoding head, BF16
+
+## Quantization Details
+
+All 30,720 MoE expert modules (256 experts x 3 projections x 40 layers) are quantized to INT4 using GPTQ. Non-expert modules (including the full vision encoder and MTP module) remain at BF16/FP16 for quality preservation.
+
+| Component | Precision | Notes |
+|-----------|-----------|-------|
+| MoE experts (`gate_proj`, `up_proj`, `down_proj`) | INT4 (GPTQ) | 30,720 modules quantized |
+| Full attention (`q_proj`, `k_proj`, `v_proj`, `o_proj`) | FP16 | Every 4th layer |
+| Linear attention (`in_proj_qkv`, `in_proj_z`, `out_proj`) | FP16 | Full precision |
+| Shared experts | FP16 | Full precision |
+| Vision encoder (`model.visual.*`) | BF16 | 333 tensors, full precision |
+| MTP module (`mtp.*`) | BF16 | 785 tensors, full precision |
+| Embeddings, LM head, norms | FP16 | Full precision |
+
+**GPTQ configuration:**
+- **Bits**: 4
+- **Group size**: 32
+- **Symmetric**: Yes
+- **desc_act**: No
+- **true_sequential**: Yes
+- **act_group_aware**: Yes
+- **Failsafe**: RTN for poorly-calibrated rare experts (1,350 of 30,720 modules, ~4.4%)
+
+### Calibration
+
+- **Dataset**: Mixed - evol-codealpaca-v1 (code) + C4 (general text)
+- **Samples**: 2,048
+- **Quantizer**: [GPTQModel](https://github.com/modelcloud/gptqmodel) v5.7.1
+
+## Model Size
+
+| Version | Size | Compression |
+|---------|------|-------------|
+| BF16 (original) | 67 GB | - |
+| GPTQ 8-bit | 40 GB | 1.7x |
+| GPTQ 4-bit | 25 GB | 2.7x |
+
+## Perplexity
+
+Evaluated on wikitext-2-raw-v1 (test set), seq_len=2048, stride=512:
+
+| Model | Perplexity | Degradation |
+|-------|------------|-------------|
+| BF16 (original) | 6.0695 | - |
+| GPTQ 8-bit | 6.0748 | +0.09% |
+| GPTQ 4-bit | 6.1260 | +0.93% |
+
+## Usage
+
+### vLLM (Recommended for Serving)
+
+```bash
+vllm serve btbtyler09/Qwen3.5-35B-A3B-GPTQ-4bit \
+  --gpu-memory-utilization 0.95 \
+  --max-model-len 256000 \
+  --tensor-parallel-size 4 \
+  --reasoning-parser qwen3 \
+  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
+  --dtype float16 \
+  --skip-mm-profiling \
+  --limit-mm-per-prompt '{"image": 2}'
+```
+
+| Parameter | Description |
+|-----------|-------------|
+| `--gpu-memory-utilization 0.95` | Use 95% of GPU VRAM for KV cache + weights |
+| `--max-model-len 256000` | Full 256K context window support |
+| `--tensor-parallel-size 4` | Shard across 4 GPUs (adjust to your setup) |
+| `--reasoning-parser qwen3` | Enable thinking/reasoning token parsing |
+| `--enable-auto-tool-choice --tool-call-parser qwen3_coder` | Enable tool/function calling |
+| `--dtype float16` | Run in FP16 (required for ROCm GPTQ kernels) |
+| `--skip-mm-profiling` | Skip multimodal memory profiling at startup |
+| `--limit-mm-per-prompt '{"image": 2}'` | Allow up to 2 images per request |
+
+> **vLLM bug workaround**: vLLM versions up to at least 0.15.2 have a bug in `Qwen3_5MoeTextConfig` where `ignore_keys_at_rope_validation` is defined as a `list` instead of a `set`, causing a `TypeError` during config parsing. Apply this fix before serving:
+>
+> ```python
+> python3 -c "
+> for f in [
+>     '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5_moe.py',
+>     '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5.py',
+> ]:
+>     t = open(f).read()
+>     t = t.replace(
+>         'ignore_keys_at_rope_validation\"] = [\n            \"mrope_section\",\n            \"mrope_interleaved\",\n        ]',
+>         'ignore_keys_at_rope_validation\"] = {\n            \"mrope_section\",\n            \"mrope_interleaved\",\n        }')
+>     open(f,'w').write(t)
+>     print('Patched', f)
+> "
+> ```
+
+### Vision Example (via OpenAI API)
+
+```python
+import base64, requests
+
+with open("image.png", "rb") as f:
+    b64 = base64.b64encode(f.read()).decode()
+
+response = requests.post("http://localhost:8000/v1/chat/completions", json={
+    "model": "btbtyler09/Qwen3.5-35B-A3B-GPTQ-4bit",
+    "messages": [{"role": "user", "content": [
+        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
+        {"type": "text", "text": "Describe what you see in this image."},
+    ]}],
+    "max_tokens": 1024,
+})
+print(response.json()["choices"][0]["message"]["content"])
+```
+
+### GPTQModel / transformers
+
+> **Note**: Neither GPTQModel nor transformers can currently load this model directly. GPTQModel's `Qwen3_5MoeGPTQ` class expects the text-only weight prefix (`model.layers.*`) and does not support the multimodal architecture (`model.language_model.layers.*`). The transformers GPTQ path delegates to `optimum`, which does not handle the fused-expert architecture. Use **vLLM** for inference.
+
+## Technical Notes
+
+Qwen3.5-35B-A3B stores MoE expert weights as fused 3D `nn.Parameter` tensors rather than individual `nn.Linear` modules. During quantization, GPTQModel's `MODULE_CONVERTER_MAP` converts these to individual quantizable `nn.Linear` layers. This same conversion must also run during model loading for the quantized kernels to be applied correctly.
+
+The vision encoder (27-block ViT) and MTP speculative decoding module are preserved at full BF16 precision from the original model. Only the text model's MoE expert weights are quantized.
+
+## Credits
+
+- **Base Model**: [Qwen](https://huggingface.co/Qwen) - Qwen3.5-35B-A3B
+- **Quantization**: GPTQ via [GPTQModel](https://github.com/modelcloud/gptqmodel) v5.7.1
+- **Expert Converter**: `convert_qwen3_5_moe_expert_converter` for fused 3D expert weights
+- **Quantized by**: [btbtyler09](https://huggingface.co/btbtyler09)
+
+## License
+
+This model inherits the [Apache 2.0 license](https://huggingface.co/Qwen/Qwen3.5-35B-A3B/blob/main/LICENSE) from the base model.

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,169 @@
+{
+  "architectures": [
+    "Qwen3_5MoeForConditionalGeneration"
+  ],
+  "image_token_id": 248056,
+  "model_type": "qwen3_5_moe",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 32,
+    "linear_value_head_dim": 128,
+    "max_position_embeddings": 262144,
+    "mlp_only_layers": [],
+    "model_type": "qwen3_5_moe_text",
+    "moe_intermediate_size": 512,
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 16,
+    "num_experts": 256,
+    "num_experts_per_tok": 8,
+    "num_hidden_layers": 40,
+    "num_key_value_heads": 2,
+    "rms_norm_eps": 1e-06,
+    "router_aux_loss_coef": 0.001,
+    "shared_expert_intermediate_size": 512,
+    "use_cache": true,
+    "vocab_size": 248320,
+    "mamba_ssm_dtype": "float32",
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "rope_type": "default",
+      "rope_theta": 10000000,
+      "partial_rotary_factor": 0.25
+    }
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.0.dev0",
+  "video_token_id": 248057,
+  "vision_config": {
+    "deepstack_visual_indexes": [],
+    "depth": 27,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen3_5_moe",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 2048,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 248054,
+  "vision_start_token_id": 248053,
+  "quantization_config": {
+    "bits": 4,
+    "checkpoint_format": "gptq",
+    "desc_act": false,
+    "dynamic": {
+      "-:.*linear_attn\\.in_proj_qkv": {},
+      "-:.*linear_attn\\.in_proj_z": {},
+      "-:.*linear_attn\\.out_proj": {},
+      "-:.*shared_expert\\.down_proj": {},
+      "-:.*shared_expert\\.gate_proj": {},
+      "-:.*shared_expert\\.up_proj": {}
+    },
+    "format": "gptq",
+    "group_size": 32,
+    "lm_head": false,
+    "meta": {
+      "act_group_aware": true,
+      "auto_forward_data_parallel": false,
+      "damp_auto_increment": 0.01,
+      "damp_percent": 0.05,
+      "failsafe": {
+        "smooth": {
+          "group_size_threshold": 128,
+          "k": 2.75,
+          "type": "mad"
+        },
+        "strategy": "rtn",
+        "threshold": "0.5%"
+      },
+      "gc_mode": "interval",
+      "gptaq": null,
+      "hessian": {
+        "chunk_bytes": null,
+        "chunk_size": null,
+        "staging_dtype": "float32"
+      },
+      "mock_quantization": false,
+      "mse": 0.0,
+      "offload_to_disk": false,
+      "offload_to_disk_path": null,
+      "pack_impl": "cpu",
+      "quantizer": [
+        "gptqmodel:5.7.1"
+      ],
+      "static_groups": false,
+      "true_sequential": true,
+      "uri": "https://github.com/modelcloud/gptqmodel",
+      "vram_strategy": "exclusive",
+      "wait_for_submodule_finalizers": false
+    },
+    "pack_dtype": "int32",
+    "quant_method": "gptq",
+    "sym": true
+  }
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 248044,
+  "do_sample": true,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
+  "pad_token_id": 248044,
+  "temperature": 1.0,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.2.0"
+}

model-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b4358552b46e1a784378cf44b96af2dfacfc8660d97b413518ce9ef6a6edfde
+size 3879674832

model-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3621df26f6e63eeaf11420b8e002634e86130bccbaaa8b8bd1a55cff3c1a783c
+size 4297389000

model-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5777d1a7a3116a6b95b0c82df8546589ecdc1950de8bdf930a67a7de766c27e6
+size 4298770456

model-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:093429512178971d603455090cccf4ccb19899065a1e2051b84260e587fe9749
+size 4298312840

model-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7637b2fd3be33fe9160d660498494dda9b58446b46efa402e532c39bcdfb5162
+size 4298769424

model-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd11ea2d7a2d6bb9b09e28d2c9eae432fd66a144bb10e76c0d1f7f84c0b5066a
+size 2652010424

model-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1249c64ad3a44c3e63d97a666a0e5fa47f023c15357069ab23c916601cc19d7f
+size 2582556456

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f32519b5a8a0e2fa8242ec8271823b102b1327214a20bb21271b8ed20ead4faa
+size 13140672

perplexity_results.txt

@@ -0,0 +1,6 @@
+Model: ./quant/Qwen3.5-35B-A3B-GPTQ-4bit/
+Dataset: wikitext-2-raw-v1 (test)
+Samples: 2891
+Sequence length: 2048
+Stride: 512
+Perplexity: 6.1260

preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 65536
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "image_processor_type": "Qwen2VLImageProcessorFast"
+}

quantize.py

@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+"""
+Quantize Qwen3.5-35B-A3B (35B MoE multimodal) to GPTQ 4-bit using GPTQModel v5.7.1.
+
+Requires:
+  - GPTQModel >= 5.7.1 (with Qwen3.5 MoE expert converter support)
+  - ~160GB system RAM (model loaded to CPU with offload_to_disk=False)
+  - 1+ GPUs with >= 32GB VRAM
+
+Quantization strategy:
+  - MoE experts (gate_proj, up_proj, down_proj): INT4 GPTQ, group_size=32
+  - Everything else (attention, linear_attn, shared_expert, norms, embeddings): FP16
+  - Vision encoder and MTP module: untouched (BF16)
+  - Mixed calibration: code (evol-codealpaca) + general text (C4)
+  - 2048 samples with context length binning for uniform expert coverage
+  - RTN failsafe for rare experts with insufficient calibration data
+
+Note: GPTQModel currently quantizes only the text model (Qwen3_5MoeForCausalLM).
+  The vision encoder and MTP module must be restored separately after quantization.
+  See the model card for details.
+"""
+
+import os
+import logging
+import random
+
+import torch
+from gptqmodel import GPTQModel
+from gptqmodel.quantization import QuantizeConfig
+from datasets import load_dataset
+from transformers import AutoTokenizer
+
+# Configuration
+MODEL_ID = "Qwen/Qwen3.5-35B-A3B"
+OUTPUT_DIR = "./Qwen3.5-35B-A3B-GPTQ-4bit"
+NUM_CALIBRATION_SAMPLES = 2048
+MAX_SEQ_LENGTH = 2048
+BITS = 4
+GROUP_SIZE = 32
+
+# Context length bins for uniform distribution
+TOKEN_BINS = [(256, 512), (512, 1024), (1024, 1536), (1536, 2048)]
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
+logger = logging.getLogger(__name__)
+
+
+def prepare_calibration_dataset(tokenizer, num_samples, token_bins):
+    """
+    Prepare mixed calibration dataset with uniform context length distribution.
+    Loads code (evol-codealpaca) and general text (C4), bins by token count,
+    and returns a uniform distribution across context length bins.
+    """
+    logger.info(f"Target: {num_samples} samples across {len(token_bins)} bins")
+
+    binned_samples = {i: [] for i in range(len(token_bins))}
+
+    # Load datasets
+    logger.info("Loading code dataset: theblackcat102/evol-codealpaca-v1")
+    code_dataset = load_dataset("theblackcat102/evol-codealpaca-v1", split="train")
+
+    logger.info("Loading general text dataset: allenai/c4")
+    c4_dataset = load_dataset(
+        "allenai/c4",
+        data_files="en/c4-train.00001-of-01024.json.gz",
+        split="train"
+    )
+
+    def format_code_sample(sample):
+        instruction = sample.get("instruction", "")
+        output = sample.get("output", "")
+        return f"### Instruction:\n{instruction}\n\n### Response:\n{output}"
+
+    # Pre-tokenize all samples
+    all_samples = []
+
+    for sample in code_dataset:
+        text = format_code_sample(sample)
+        if len(text) < 50:
+            continue
+        token_count = len(tokenizer.encode(text, add_special_tokens=False))
+        all_samples.append((text, token_count))
+
+    for idx, sample in enumerate(c4_dataset):
+        if idx >= 50000:
+            break
+        text = sample.get("text", "")
+        if len(text) < 50:
+            continue
+        token_count = len(tokenizer.encode(text, add_special_tokens=False))
+        all_samples.append((text, token_count))
+
+    logger.info(f"Total samples pool: {len(all_samples)}")
+
+    # Bin samples by token length
+    for text, token_count in all_samples:
+        for bin_idx, (min_tok, max_tok) in enumerate(token_bins):
+            if min_tok <= token_count < max_tok:
+                binned_samples[bin_idx].append(text)
+                break
+
+    # Chain short samples for sparse long-context bins
+    random.shuffle(all_samples)
+
+    def create_chained_sample(min_tokens, max_tokens):
+        chained_texts, current_tokens, attempts = [], 0, 0
+        while current_tokens < min_tokens and attempts < 50:
+            text, tok_count = random.choice(all_samples)
+            if current_tokens + tok_count > max_tokens:
+                attempts += 1
+                continue
+            chained_texts.append(text)
+            current_tokens += tok_count
+            attempts = 0
+        if min_tokens <= current_tokens < max_tokens:
+            return "\n\n---\n\n".join(chained_texts)
+        return None
+
+    samples_per_bin = num_samples // len(token_bins)
+    for bin_idx, (min_tok, max_tok) in enumerate(token_bins):
+        needed = samples_per_bin + 1 - len(binned_samples[bin_idx])
+        if needed > 0:
+            for _ in range(needed * 20):
+                chained = create_chained_sample(min_tok, max_tok)
+                if chained:
+                    binned_samples[bin_idx].append(chained)
+                    needed -= 1
+                if needed <= 0:
+                    break
+
+    for bin_idx, (min_tok, max_tok) in enumerate(token_bins):
+        logger.info(f"  Bin {bin_idx} ({min_tok}-{max_tok} tokens): {len(binned_samples[bin_idx])} samples")
+
+    # Sample uniformly from each bin
+    remainder = num_samples % len(token_bins)
+    final_samples = []
+    for bin_idx in range(len(token_bins)):
+        target = samples_per_bin + (1 if bin_idx < remainder else 0)
+        bin_data = binned_samples[bin_idx]
+        if len(bin_data) < target:
+            final_samples.extend(bin_data)
+            final_samples.extend(random.choices(bin_data, k=target - len(bin_data)))
+        else:
+            final_samples.extend(random.sample(bin_data, target))
+
+    random.shuffle(final_samples)
+    logger.info(f"Final calibration dataset: {len(final_samples)} samples")
+    return final_samples
+
+
+# Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True, trust_remote_code=True)
+
+# Prepare calibration data
+calibration_data = prepare_calibration_dataset(tokenizer, NUM_CALIBRATION_SAMPLES, TOKEN_BINS)
+
+# Dynamic exclusions: keep these at full precision (FP16)
+dynamic_exclusions = {
+    "-:.*linear_attn\\.in_proj_qkv": {},
+    "-:.*linear_attn\\.in_proj_z": {},
+    "-:.*linear_attn\\.out_proj": {},
+    "-:.*shared_expert\\.gate_proj": {},
+    "-:.*shared_expert\\.up_proj": {},
+    "-:.*shared_expert\\.down_proj": {},
+}
+
+# Create quantization config
+quant_config = QuantizeConfig(
+    bits=BITS,
+    group_size=GROUP_SIZE,
+    sym=True,
+    desc_act=False,
+    true_sequential=True,
+    offload_to_disk=False,
+    lm_head=False,
+    dynamic=dynamic_exclusions,
+)
+
+# Load and quantize
+logger.info(f"Loading {MODEL_ID}...")
+model = GPTQModel.load(MODEL_ID, quant_config, trust_remote_code=True)
+
+logger.info("Starting quantization...")
+model.quantize(calibration_data, batch_size=1)
+
+logger.info(f"Saving to {OUTPUT_DIR}...")
+model.save(OUTPUT_DIR)
+tokenizer.save_pretrained(OUTPUT_DIR)
+
+logger.info("Done!")

quantize_config.json

@@ -0,0 +1,54 @@
+{
+  "bits": 4,
+  "dynamic": {
+    "-:.*linear_attn\\.in_proj_qkv": {},
+    "-:.*linear_attn\\.in_proj_z": {},
+    "-:.*linear_attn\\.out_proj": {},
+    "-:.*shared_expert\\.gate_proj": {},
+    "-:.*shared_expert\\.up_proj": {},
+    "-:.*shared_expert\\.down_proj": {}
+  },
+  "group_size": 32,
+  "desc_act": false,
+  "lm_head": false,
+  "quant_method": "gptq",
+  "checkpoint_format": "gptq",
+  "pack_dtype": "int32",
+  "meta": {
+    "quantizer": [
+      "gptqmodel:5.7.1"
+    ],
+    "uri": "https://github.com/modelcloud/gptqmodel",
+    "damp_percent": 0.05,
+    "damp_auto_increment": 0.01,
+    "static_groups": false,
+    "true_sequential": true,
+    "mse": 0.0,
+    "gptaq": null,
+    "act_group_aware": true,
+    "failsafe": {
+      "strategy": "rtn",
+      "threshold": "0.5%",
+      "smooth": {
+        "type": "mad",
+        "group_size_threshold": 128,
+        "k": 2.75
+      }
+    },
+    "offload_to_disk": false,
+    "offload_to_disk_path": null,
+    "pack_impl": "cpu",
+    "mock_quantization": false,
+    "gc_mode": "interval",
+    "wait_for_submodule_finalizers": false,
+    "auto_forward_data_parallel": false,
+    "hessian": {
+      "chunk_size": null,
+      "chunk_bytes": null,
+      "staging_dtype": "float32"
+    },
+    "vram_strategy": "exclusive"
+  },
+  "sym": true,
+  "format": "gptq"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4
+size 19989343

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

video_preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 25165824,
+        "shortest_edge": 4096
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "video_processor_type": "Qwen3VLVideoProcessor"
+}