Add files using upload-large-folder tool

README.md

@@ -0,0 +1,86 @@
+---
+license: mit
+base_model:
+  - zai-org/GLM-5.2
+base_model_relation: quantized
+language:
+  - ja
+  - en
+  - zh
+tags:
+  - quantization
+  - vector-quantization
+  - aqlm
+  - mixture-of-experts
+  - glm
+  - vllm
+pipeline_tag: text-generation
+---
+
+# GLM-5.2 — mixed-bit VQ (AQLM) ~1.86-bit
+
+A **~180 GiB** quantization of **[GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)**
+(744B Chinese-native reasoning MoE, MIT) that keeps Japanese/English/Chinese
+**thinking-mode** quality at **~1.86 bit/weight**, using **vector quantization with
+GPTQ error compensation (AQLM-style)** instead of scalar rounding.
+
+Runs on **2× RTX PRO 6000 (sm_120 Blackwell, ~95 GiB each)** via vLLM.
+
+## Why VQ
+
+At the same size, scalar mixed-bit rounding loses too much at 1–2 bit. Replacing the
+scalar codes with a **shared vector codebook + per-row error compensation** recovers most
+of it (the two together are *super-additive* — neither alone is enough):
+
+| Metric | Scalar mixed-bit (same ~180 GiB) | **This (VQ + compensation)** |
+|---|---|---|
+| Calibration KL (fake-quant, iso-size) | baseline | **−47 %** |
+| Greedy arithmetic eval (JA/EN/ZH, terminate + correct) | 21/22 | **22/22** |
+| 1-bit experts | collapse (KL ≈ 13) | **survive (KL ≈ 0.42)** |
+
+> The −47 % KL is a fake-quant, iso-size comparison; the deployable end-to-end signal is
+> the greedy eval (multi-digit multiplication and word problems in all three languages,
+> including held-out items not in calibration).
+
+## Serving
+
+This is **not** a plug-and-play GGUF — it needs a matching sm_120 stack:
+
+- **vLLM** with `GlmMoeDsaForCausalLM` + sm_120 kernels (reference: `jasl/vllm` PR-41834 sm12x preview).
+- **transformers 5.12**.
+- The **VQ serving plugin** from **[mmzz164/OneCompression @ `glm-serving-v1`](https://github.com/mmzz164/OneCompression)** — see [`example/glm-5.2/`](https://github.com/mmzz164/OneCompression/tree/glm-serving-v1/example/glm-5.2) for the launcher and full instructions.
+- **2× ~95 GiB sm_120 GPUs**, EP=1 / TP=2 (VQ codes can't be tensor-parallel-sharded).
+
+```bash
+GLM_CKPT=/path/to/this/model bash start_glm_api_vq.sh   # OpenAI API :8001, served as "glm-5.2"
+```
+
+`MixedVQMoEMethod` is auto-selected from the `format:"vq"` markers in `quantization_config`.
+
+## Performance
+
+- **~16 tok/s** steady-state decode (single stream), **38×** over the eager dequant baseline
+  (grouped Triton VQ-GEMM + CUDA graphs; the key win was fixing a shared-memory bank conflict
+  in the codebook gather).
+- **Context ≤ 4096** (dense MLA — sm_120 has no sparse-DSA forward kernel; dense is exact at
+  ctx ≤ 2048 and validated functional, incl. >2048 needle retrieval, up to 4096).
+
+## Allocation
+
+Mixed **1/2/3-bit per expert** (≈ 15.8k @1-bit / 34.0k @2-bit / 7.8k @3-bit projections),
+allocated by activation-frequency-aware AutoBit (arithmetic-routing tuned), then re-encoded
+to VQ codes + a shared codebook per bit-width. Non-expert "spine" stays scalar 4-bit.
+
+## Limitations
+
+- sm_120-specific serving stack (not portable to plain wheels).
+- Context capped at 4096 on this hardware (sm_120 sparse-attention kernel gap + VRAM); longer
+  context needs smaller weights.
+- Default serving template is **thinking-on at max reasoning effort** — thorough but verbose
+  for casual chat (pass `chat_template_kwargs={"enable_thinking": false}` for direct answers).
+
+## License & attribution
+
+- This quantized model: **MIT**.
+- Base **GLM-5.2**: **MIT**, © Zhipu AI — this is a derivative; all rights/attribution to upstream.
+- Quantization/serving built on **OneCompression** (MIT, © Fujitsu Ltd.) and vLLM / transformers (Apache-2.0).

chat_template.jinja

@@ -0,0 +1,119 @@
+[gMASK]<sop>
+{%- set effective_reasoning_effort = 'high' if reasoning_effort is defined and reasoning_effort == 'high' else 'max' -%}
+{%- if (enable_thinking is not defined or enable_thinking) and effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
+{%- if tools -%}
+{%- macro tool_to_json(tool) -%}
+    {%- set ns_tool = namespace(first=true) -%}
+    {{ '{' -}}
+    {%- for k, v in tool.items() -%}
+        {%- if k != 'defer_loading' and k != 'strict' -%}
+            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
+            {%- set ns_tool.first = false -%}
+            "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- '}' -}}
+{%- endmacro -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{%- if 'function' in tool -%}
+    {%- set tool = tool['function'] -%}
+{%- endif -%}
+{% if tool.defer_loading is not defined or not tool.defer_loading %}
+{{ tool_to_json(tool) }}
+{% endif %}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
+                {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
+                {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) -%}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{%- for m in messages -%}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- elif '</think>' in content %}
+    {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
+    {%- set content = content.split('</think>')[-1] %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
+{{ '<think>' + reasoning_content +  '</think>'}}
+{%- else -%}
+{{ '<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' -}}
+{%- endif %}
+{%- if m.content is string -%}
+    {{- '<tool_response>' + m.content + '</tool_response>' -}}
+{%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0.type == "tool_reference" -%}
+    {{- '<tool_response><tools>\n' -}}
+    {% for tr in m.content %}
+        {%- for tool in tools -%}
+            {%- if 'function' in tool -%}
+                {%- set tool = tool['function'] -%}
+            {%- endif -%}
+            {%- if tool.name == tr.name -%}
+                {{- tool_to_json(tool) + '\n' -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endfor -%}
+    {{- '</tools></tool_response>' -}}
+{%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0 is mapping and m.content.0.output is defined -%}
+    {%- for tr in m.content -%}
+        {{- '<tool_response>' + tr.output + '</tool_response>' -}}
+    {%- endfor -%}
+{%- else -%}
+    {{- '<tool_response>' + visible_text(m.content) + '</tool_response>' -}}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '<think></think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 154820,
+  "transformers_version": "5.12.0",
+  "use_cache": true
+}

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "TokenizersBackend"
+}

vq-base-model-00002-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dcb15f56f9e3cfc9732385f222f5c2fcf203d5bfed5fe238f0aa0978ed4b035e
+size 236159704

vq-base-model-00006-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f22b0263ee5247e797757a9b4e52be1d140e4a82077f761e1bb9d35d86907b3
+size 217415376

vq-base-model-00008-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be85ec345e5b5f0fef78c1d9020d752da64f9895437a4ae53f2282f672d60807
+size 168700640

vq-base-model-00013-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ce5b2a9b460da4fd7af3a20b6a0d1c7f521a28919082f5ee6671bdde7999b5c
+size 217415376

vq-base-model-00019-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dff8f1f927d8238653d8dbb44ae9ab799a52352bce8426b7e5e7091115c477a
+size 236159784

vq-base-model-00020-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b6abdd7c5172988fd887aee68902a13f9a6541c5c79c4e881b528a51482aef6
+size 165051656

vq-base-model-00023-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42b417d77f107dff8e9256480d609f2241777b059daaab6ab1610ee79ae68709
+size 236159784

vq-base-model-00025-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e1ab6cd1eed5a14c010ce920cc3a189707dde4e66716a51ee0f677527c74675
+size 217415376

vq-base-model-00026-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58a31c0fc22c4175d30057a27fb29f913691fc271c1b6c805abf192e1d7d6a9c
+size 236159784

vq-base-model-00030-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27d1a5f6b897e2c6a413b8300216b88e4ee35e56a4bc3273c5f298bee8aeba47
+size 108707664

vq-base-model-00033-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e164d3df9c94bebc0c406d0cd7b7b318449b36c3ddbb4f5d6735bb4bc78d544b
+size 156860776

vq-base-model-00034-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e64116c2002f0eb50835b5986d81bd8fd98d4b3f4297fc4eace2c9d293c89d9f
+size 188006672

vq-base-model-00035-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:614f4bee1dcccb464756f3c1d4874aab270ce29528e517cd07c9aec9eeba3daf
+size 236159784

vq-base-model-00036-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92e79454c6ec16f06497519dfcbc3c4209f66b43660845f19742d630de5d7b99
+size 217415376

vq-base-model-00038-of-00046.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09d78f05581770a4bb4cd5fd3cb10bcca0f4c6bf2679e5d0a25d271722454185
+size 236159784

vq-experts-L003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10df3b804bc25f8698e2c8efe75d6578c694cf0b62d251dfb1aff065fa02fefe
+size 2007162208

vq-experts-L004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f618d7a52d572e155648eb2c994b87450c30bd64123e9dc456c388afa57f1a0a
+size 2403524176

vq-experts-L005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cd6e8c183029f5d59c7ec8272735e6b2e9ba116646b7dd5c9e2892a010a6100
+size 2513624680

vq-experts-L006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86c14d7307227118e6f521b6cc73ae08669c844baa8937c5a7c74a68faf9059c
+size 2653609672

vq-experts-L009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4cbcb339232b4e0168ee92a6ee542cad17cc4f20f4eb99dd6fe1ed8152f8bf1
+size 2615860920

vq-experts-L011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7d835446417f018c763dea46ee6bc1a50ff5e3e0e5f1d495c59b28e07cfb99e
+size 2601706656

vq-experts-L014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:621989f591ad7ca4ef4bdc7d2a8d7e3623c05e4bfbdd9171844333cb657eeab2
+size 2548229264

vq-experts-L021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40517e7dfd06fcaf16ee1a8ead23935d4d85359b4831da51599b2a694d1aeb84
+size 2335892504

vq-experts-L023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f27939b4e0bbf1f2d282c29002499e3d0af6309a3f9fdae71acf3edbd81c7bb6
+size 2513626200

vq-experts-L026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af55f00e37e897808b78e192af54c7d962209a5c1fb1307559fbffe8f5bf08a8
+size 2434982992

vq-experts-L027.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b3dbf56931dce2b39b0d181d4b2c62364a959b9c2b79549bf124f45c386a9c7
+size 2409817176

vq-experts-L028.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f7ce97f0ecee5b0b13f80cd87033277d4a4f9ef68c1efe6190451a6da8370f0
+size 2460148808

vq-experts-L029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb9bf1a15883bce8fced9a115ff13b968533671a5581adb769dd21a9fa303eb4
+size 2340611080

vq-experts-L030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2dd35fc1ec3e40ebd92b79d2fae084c3a058b80a6d92e88c24b665c07aa2659
+size 2434982968

vq-experts-L036.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55f95b546debd8e9827e5b961256af09242b8efc004b60f9c0013ca612d06601
+size 2324882424

vq-experts-L037.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:183f0e805002ed917b5b1847644fea568e04222daa54d11dfc30d16c2e15ef8c
+size 2390942736

vq-experts-L039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5f7815b03ade2cd2a21a3d6608f9be286eb4ecb8770b219b56aba08d7d51c22
+size 2332746816

vq-experts-L040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57f096fd1f01656f6c4a0a02724a53b0eafd3e6c0ecfb1cacf02adc1c65d81bd
+size 2307580928

vq-experts-L041.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f5efbd5333fba5260c11ac8ebd0ea89def811360d91854d61c92e51e62e7070
+size 2326455312

vq-experts-L042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f415d16f03aa5cc041ab42ba7749acc0c8e209bfa153616648118a4b478bd43
+size 2370495488

vq-experts-L048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46254f9d2ed47f9a716dbe0d953c4456001f7d05aac0445f09bd8939f49d2e48
+size 2350048296

vq-experts-L053.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:718ae4fb55747c5238cdcde9b945394378da189c38698055a3cc7e19d4575e76
+size 2444420128

vq-experts-L054.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:726e302fe061254b19a3afd285cfd908e98bb63021ae6e81b58ea7ccd950c559
+size 2356339704

vq-experts-L059.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:703827b1d938dab6ce354d7408e3891ac7a2e8d6088ad100bb235aaa8a8998be
+size 2350048280

vq-experts-L060.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:138463d0ac04140ebbc31224f87cd8b00f789457376130430450078d7361b9f2
+size 2339038216

vq-experts-L067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa677dd8e780ea4e307ca79b28f16dcfb19a0785ae8b13567a955693594c190e
+size 2389369904

vq-experts-L069.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ac6b4dc999e16faeb4074bcf469369d0b2b2fa9474f7107f86fa7f1358f7b53
+size 2368922664

vq-experts-L077.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c95b13572114cf46d22cc856b6e69de373a5a590764031b3bdbcd90016008d27
+size 2466440272