Add model card

README.md

@@ -0,0 +1,137 @@
+---
+language:
+- en
+library_name: mlx
+license: mit
+pipeline_tag: image-text-to-text
+base_model: zai-org/GLM-5.3-Flash
+tags:
+- mlx
+- jang
+- jangtq
+- quantized
+- apple-silicon
+- vision
+- video
+- reasoning
+- thinking
+- agent
+- tool-use
+- glm5_next
+- moe
+- gptq
+- imatrix
+---
+
+<p align="center">
+  <img src="./jangq-logo.png" alt="JANGQ" width="220">
+  &nbsp;&nbsp;&nbsp;
+  <img src="./vmlx-logo.png" alt="vMLX" width="90">
+</p>
+
+> ⚠️ **Runtime not released yet.** This bundle uses **JANGTQ v2**, a new routed-expert format, on a new
+> architecture (`glm5_next`: KDA linear attention + MLA/DSA hybrid + mHC). No released vMLX or Osaurus build can load
+> it today. They refuse it at load time rather than producing wrong output. Access is gated until runtime support
+> ships. The numbers below were measured with the internal vMLX build this bundle was made for.
+
+# JANGQ-AI/GLM-5.3-Flash-JANGTQ2
+
+**GLM-5.3-Flash for 128 GB Macs.** Same size as our previous affine release, with **2.7x lower median KL**, **+5.1
+points top-1**, and faster decode.
+
+A JANGTQ v2 bundle of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash): a 300B-class MoE (288
+routed experts, top-8 + shared expert) with KDA linear attention, sparse attention, and vision + video towers.
+- **Routed experts**: JANGTQ v2 at 2-3 bits. Codebook-quantized with per-row scales and a blockwise Hadamard rotation,
+  calibrated with GPTQ + a per-expert importance matrix. Bits are placed per layer by measurement.
+- **Everything else**: 8-bit (MXFP8 where the source weights sit on an FP8-MX grid, affine 8-bit elsewhere) or full
+  precision.
+- **Vision tower**: kept in bf16.
+
+This replaces `JANGQ-AI/GLM-5.3-Flash-JANG` and `-JANG-MTP`.
+
+## Quality vs the official FP8 release
+15,830 teacher-forced positions on held-out prompts, top-128 renormalized KL.
+
+| Bundle | Size | median KL ↓ | mean KL ↓ | p90 / p95 / p99 ↓ | top-1 ↑ | top-5 ↑ | top-10 ↑ |
+|---|---|---|---|---|---|---|---|
+| **GLM-5.3-Flash-JANGTQ2** | **95.89 GiB** | **0.0323** | **0.351** | **0.89 / 1.76 / 4.53** | **83.7%** | **97.0%** | **98.5%** |
+| GLM-5.3-Flash-JANG (affine, previous release) | 95.35 GiB | 0.0882 | 0.528 | 1.50 / 2.55 / 5.67 | 78.6% | 94.6% | 96.8% |
+| orcarouter GLM-5.3-Flash-MLX `2bit-lite` ¹ | 95.4 GiB | 0.2122 | 0.83 | — | 71.4% | 90.8% | 94.3% |
+
+¹ Measured earlier on the same 20 prompts (15,850 positions), loading its shipped quantized weights natively.
+
+## Agentic fidelity vs the bf16 model itself
+Scored against the full bf16 model, run layer by layer from disk. 72 held-out tool-use conversations (25,867
+positions; tools and phrasings disjoint from calibration), including 114 "call a tool or answer?" decision points.
+
+| | **JANGTQ2** | affine JANG (previous) |
+|---|---|---|
+| median KL vs bf16 | **0.145** | 0.595 |
+| top-1 agreement with bf16 | **68.7%** | 54.9% |
+| tool-call decisions: same choice as bf16 | **50 / 50** | 50 / 50 |
+| median P(`<tool_call>`) at call points (bf16: 0.998) | **0.999** | 0.981 |
+| lowest P(`<tool_call>`) at a call point | **0.987** | 0.784 |
+| answer decisions: same next token as bf16 | **89.1%** | 50.0% |
+| decisions flipped call ↔ answer vs bf16 | **0** | 3 |
+
+These are synthetic agent transcripts, so absolute KL is high for every bundle; read the columns against each other.
+The calibration set includes agent-style conversations from the same generator (different tools and wording), so
+part of this gain is in-distribution. The FP8 table above is independent of that.
+
+## Live behavior (served, temperature 0)
+
+| suite | **JANGTQ2** | affine JANG (previous) |
+|---|---|---|
+| 48-case tool-use eval (required / auto × thinking on / off) | **48 / 48** | 48 / 48 |
+| 16 behavior probes: math, follow-up, tool round trip, efforts low/high/max, image, video | **16 / 16** | 16 / 16 |
+| same 16 probes after a server restart (SSD prefix-cache restore) | **16 / 16** | — |
+
+Both bundles saturate these suites, so they check behavior rather than rank the two.
+
+## Speed (M5 Max 128 GB, served, interleaved A/B/A/B, fresh prompts, median of 3)
+
+| | **JANGTQ2** | affine JANG (previous) |
+|---|---|---|
+| decode (tok/s) | **27.87 / 27.88** | 26.81 / 26.85 |
+| prefill, ~5.2k-token prompt (tok/s) | 348 / 390 | 421 / 416 |
+| peak memory while serving | 96.7 GiB | 96.1 GiB |
+
+Two interleaved runs per bundle, same runtime build, each run the median of 3 probes on a never-seen prompt.
+Decode is **~4% faster** than the affine bundle; long-prompt prefill is currently ~12% slower.
+
+## What's in the bundle
+- **Vision + video**: full bf16 vision tower + the consolidated image/video processor config.
+- **No MTP**: layer 45 is omitted; its bytes went into expert precision.
+- **Thinking + agentic**:
+  - Thinking is ON by default (the template opens `<think>`).
+  - Reasoning efforts are **`low` / `high` / `max`** (default `max`). There is no `medium`: the template renders any
+    other value as Max.
+  - `clear_thinking=false` preserves thinking in history.
+- **Tool calls**: GLM's XML dialect (`<tool_call>name<arg_key>…</arg_key><arg_value>…</arg_value></tool_call>`),
+  declared as `tool_parser: glm_xml_args`; tool results render as `<|observation|>`. Hermes-style JSON parsers will
+  not work.
+- **Self-describing**:
+  - `config.json` carries the JANGTQ v2 format block (codebook, packing, rotation, method) and a per-module
+    `quantization` map (126 expert projections, 147 MXFP8, 214 affine 8-bit).
+  - `jang_config.json` records calibration and per-layer expert bits.
+  - Raw evaluation results are in `evaluation/`.
+- **Memory**: 95.89 GiB weights, 96.7 GiB peak while serving. Fixed-size linear-attention state +
+  compressed-latent KV (~6 KB/token), so long contexts do not balloon memory.
+- Every shard is alignment-safe (zero-copy memory mapping).
+
+## Serving contract
+- Sampling: `temperature=1.0, top_p=0.95` (vendor defaults), no repetition penalty
+- EOS: `[154820, 154827, 154829]` · context: 1M native
+- Reasoning: `reasoning_effort` chat-template kwarg (`low` / `high` / `max`), default `max`
+- Thinking off: the template always opens `<think>`. Runtimes must close it in GLM's native form, `<think></think>`
+  with no whitespace; an R1-style `\n</think>\n\n` measurably degrades thinking-off tool decisions.
+
+## Build details
+- Source: `zai-org/GLM-5.3-Flash-BF16` @ `a5b45eb`
+- Calibration:
+  - 600k tokens (web / code / multi-turn chat incl. tool transcripts / math), referenced to the official FP8 release
+  - plus a bf16 agentic capture (448 GLM-template tool conversations), with evaluation prompts held out
+- Experts: JANGTQ v2 (odd-cubic codebook, fp16 per-row scale, Hadamard-32 rotation), GPTQ on all 42 MoE layers with a
+  per-expert importance matrix, bit allocation measured per layer (gate/up 2-3 bit, down 2-3 bit)
+
+Quantized and validated by **Jinho Jang** — eric@jangq.ai