Add GLM-5.2-W4A16 (INT4 weights, BF16 activations)

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,127 @@
+---
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
+base_model: zai-org/GLM-5.2
+tags:
+  - glm
+  - moe
+  - quantized
+  - w4a16
+  - int4
+  - compressed-tensors
+  - vllm
+language:
+  - en
+  - zh
+---
+
+# GLM-5.2-W4A16
+
+4-bit (W4A16) weight-quantized version of [**zai-org/GLM-5.2**](https://huggingface.co/zai-org/GLM-5.2).
+
+Weights are quantized to **INT4** (group size 128); activations run in **BF16**. The
+result is a **388 GB** checkpoint — about **3.9× smaller** than the 1.5 TB BF16
+original — that fits on a single 8×A100 (80 GB) node while preserving full-precision
+quality on reasoning and knowledge benchmarks.
+
+Because compute stays in BF16 and only the weights are INT4, this model runs on
+**NVIDIA Ampere (A100) and newer** — it does **not** require Hopper/Blackwell FP8 support.
+
+## Highlights
+
+- **Quality on par with the FP8 release** — no measurable degradation (see below).
+- **~3.9× smaller** than BF16: 388 GB vs. 1.5 TB.
+- **Runs on A100** (Ampere) — no FP8 hardware needed.
+- Serves out of the box with **vLLM** (`compressed-tensors` format).
+
+## Evaluation
+
+Evaluated against the reference **FP8** deployment of GLM-5.2. Greedy decoding
+(temperature 0); reasoning traces stripped and the final answer graded.
+
+| Benchmark | This model (W4A16) | Reference (FP8) |
+|---|---|---|
+| GSM8K (n=200), exact-match | **96.5%** | 94.5% |
+| MMLU (n=200), accuracy | **86.5%** | 80.0% |
+
+W4A16 matches the FP8 reference within evaluation noise (n=200, standard error
+≈ 2 pts). The takeaway is **parity** — 4-bit quantization retains GLM-5.2's
+reasoning and knowledge capability.
+
+## Model details
+
+| | |
+|---|---|
+| Base model | `zai-org/GLM-5.2` |
+| Architecture | `GlmMoeDsaForCausalLM` (MoE, 78 layers, 256 routed + 1 shared expert, top-8) |
+| Weight precision | INT4, group size 128, symmetric |
+| Activation precision | BF16 |
+| Format | `compressed-tensors` (`pack-quantized`) |
+| Checkpoint size | 388 GB (8 shards) |
+| Context length | up to 1,048,576 tokens |
+
+The sparse-attention (DSA) indexer, the MoE router, and the LM head are kept in
+BF16; the large linear and expert weights carry the INT4 quantization.
+
+## Serving on A100 (8× A100 80 GB, vLLM)
+
+The full INT4 checkpoint fits on one 8×A100-80GB node with room for KV cache.
+
+```bash
+pip install "vllm>=0.24.0"
+
+# A100 (Ampere) note: use BF16 compute paths and skip Hopper-only kernels.
+export VLLM_USE_FLASHINFER_SAMPLER=0   # avoid FlashInfer sampler JIT on some CUDA toolkits
+export VLLM_USE_DEEP_GEMM=0            # DeepGEMM (FP8 block-scale) is not needed on A100
+
+vllm serve lowbitcoffee/GLM-5.2-W4A16 \
+  --tensor-parallel-size 8 \
+  --dtype bfloat16 \
+  --max-model-len 32768 \
+  --gpu-memory-utilization 0.92 \
+  --served-model-name glm-5.2-w4a16 \
+  --trust-remote-code
+```
+
+vLLM auto-detects the quantization from the checkpoint — no `--quantization`
+flag is required. Increase `--max-model-len` toward the model's 1M limit only if
+you have KV-cache headroom; lower it to raise concurrency.
+
+> On 8× **A100 40 GB**, the weights alone (388 GB) exceed the 320 GB of aggregate
+> VRAM — use two nodes (`--tensor-parallel-size 16`) or the 80 GB SKU.
+
+### Query it (OpenAI-compatible)
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+        "model": "glm-5.2-w4a16",
+        "messages": [{"role": "user", "content": "What is 84 * 3 / 2?"}],
+        "max_tokens": 1024,
+        "temperature": 0
+      }'
+```
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
+resp = client.chat.completions.create(
+    model="glm-5.2-w4a16",
+    messages=[{"role": "user", "content": "Explain MoE routing in two sentences."}],
+    max_tokens=1024,
+    temperature=0.6,
+)
+print(resp.choices[0].message.content)
+```
+
+GLM-5.2 is a reasoning model: responses may include a `<think>…</think>` block
+before the final answer. Strip it client-side, or configure a reasoning parser
+in your serving stack if you want the fields separated.
+
+## License
+
+Released under the **MIT** license, inheriting the license of the base model
+`zai-org/GLM-5.2`.

chat_template.jinja

@@ -0,0 +1,119 @@
+[gMASK]<sop>
+{%- set effective_reasoning_effort = 'high' if reasoning_effort is defined and reasoning_effort == 'high' else 'max' -%}
+{%- if (enable_thinking is not defined or enable_thinking) and effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
+{%- if tools -%}
+{%- macro tool_to_json(tool) -%}
+    {%- set ns_tool = namespace(first=true) -%}
+    {{ '{' -}}
+    {%- for k, v in tool.items() -%}
+        {%- if k != 'defer_loading' and k != 'strict' -%}
+            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
+            {%- set ns_tool.first = false -%}
+            "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- '}' -}}
+{%- endmacro -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{%- if 'function' in tool -%}
+    {%- set tool = tool['function'] -%}
+{%- endif -%}
+{% if tool.defer_loading is not defined or not tool.defer_loading %}
+{{ tool_to_json(tool) }}
+{% endif %}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%}
+                {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%}
+                {{- "<reminder>You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods.</reminder>" }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) -%}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{%- for m in messages -%}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- elif '</think>' in content %}
+    {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
+    {%- set content = content.split('</think>')[-1] %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
+{{ '<think>' + reasoning_content +  '</think>'}}
+{%- else -%}
+{{ '<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' -}}
+{%- endif %}
+{%- if m.content is string -%}
+    {{- '<tool_response>' + m.content + '</tool_response>' -}}
+{%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0.type == "tool_reference" -%}
+    {{- '<tool_response><tools>\n' -}}
+    {% for tr in m.content %}
+        {%- for tool in tools -%}
+            {%- if 'function' in tool -%}
+                {%- set tool = tool['function'] -%}
+            {%- endif -%}
+            {%- if tool.name == tr.name -%}
+                {{- tool_to_json(tool) + '\n' -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endfor -%}
+    {{- '</tools></tool_response>' -}}
+{%- elif m.content is iterable and m.content is not mapping and m.content and m.content.0 is mapping and m.content.0.output is defined -%}
+    {%- for tr in m.content -%}
+        {{- '<tool_response>' + tr.output + '</tool_response>' -}}
+    {%- endfor -%}
+{%- else -%}
+    {{- '<tool_response>' + visible_text(m.content) + '</tool_response>' -}}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '<think></think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "do_sample": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "top_p": 0.95,
+  "transformers_version": "5.10.1"
+}

model-00001-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c34baff8bc0746006fc73880f1ffc47a6405bcd0f0819b2c4210baf22ad5d19
+size 49997402336

model-00002-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d5ed084a714e0d89ffa067ca12a8bc3c3a17d0dc0f35b4a581aa894fdfb581b
+size 49999575032

model-00003-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d34cc57c7d24b5bd339fa0083e20d4c91c61ab794d7e366853fb7d537f403445
+size 49999575008

model-00004-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d106fec8a9a59f1f680951393eed0a91c9f173d302d8e75ad20edeaf96c7c9e
+size 49999575280

model-00005-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65a0ea903313997d647ec5711f72d105a764dc5522d2176df3bbd6bfc49bb93d
+size 49999575520

model-00006-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72caa80144360ec26705388f8af993c582926ed362140ad5c91e265c6f1e556e
+size 50002867272

model-00007-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd76639c4eff7c58c8e13e9a68aee04d0396698d48fbadb896c0d157b9b1630d
+size 49999575032

model-00008-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61f3ecffa2e554661d1ebe5075715d9128a0cf0d5701267c1dc621db3f172a0a
+size 37691064128

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74d73bfaa26425beaf618342f4a0851b21d9198138b76bfb678f88164d987beb
+size 17041081

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d
+size 20217442

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>"
+  ],
+  "is_local": true,
+  "local_files_only": true,
+  "model_max_length": 1048576,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "TokenizersBackend"
+}