Nairod785
/

Confucius4-R2T2-Q4_K_M-GGUF

@@ -52,80 +52,45 @@ tags:
   - transcribe.cpp
 ---
-# Confucius4-R2T2 — Q4_K_M GGUF (Arm M)
 Optimized **Q4_K_M GGUF** quantization of **NetEase Youdao Confucius4-R2T2**, a low-latency, append-only streaming ASR model (fine-tuned from Qwen3-ASR-1.7B with Longest Stable Prefix decoding), ready for native execution in [transcribe.cpp](https://github.com/NairoDorian/transcribe.cpp) and [audio.cpp](https://github.com/0xShug0/audio.cpp).
 * **Original model:** [netease-youdao/Confucius4-R2T2](https://huggingface.co/netease-youdao/Confucius4-R2T2) · [source code](https://github.com/netease-youdao/Confucius4-R2T2) · [demo](https://r2t2.youdao.com/demo)
 * **Base Architecture:** Qwen3-ASR audio encoder (24 layers, 1024 width, 2048 projected width) + Causal LM decoder (28 layers, 16 query heads, 8 KV heads, 128 head width).
-* **Reference GGUF repository & inspiration:** [davidxifeng/Confucius4-R2T2-gguf](https://huggingface.co/davidxifeng/Confucius4-R2T2-gguf) by David Xi Feng
-* **Quantization & Benchmarking:** Researched, validated, and tuned in [transcribe.cpp](https://github.com/NairoDorian/transcribe.cpp).
 ---
-## The Breakthrough: Solving the Low-Bit Quantization Barrier
-In previous GGUF releases (e.g. `davidxifeng/Confucius4-R2T2-gguf`), precision was restricted:
-> *"Precision policy: Q8_0 and higher are supported. 4-bit and 5-bit quantizations (legacy and k-quant) are rejected at load time with an actionable error, because this graph's kernels are not validated below Q8_0 and otherwise decode to empty text."*
-Uniform 4-bit and 5-bit quantization suffered from two severe failure modes:
-1. **The Empty Output Cliff:** Low-bit matmuls below Q4_K (e.g., tower or gate/up at Q3_K) produce abrupt total collapse—emitting empty output rather than degraded text.
-2. **Multilingual Language Drift:** Standard Q4_K and Q5_K quantizations passed simple English (`jfk`) and Chinese (`zh`) tests, but silently flipped to English when fed German or Russian audio.
-### The Arm M Recipe
-Through exhaustive per-tensor ablation and cross-lingual probing across 20+ experimental arms, the optimal configuration (**Arm M**) was isolated:
-- **Filename:** `r2t2-q4_k_m.gguf`
-- **File size:** **1.187 GB** (1,186,939,968 bytes)
-- **Footprint reduction:** **52% smaller** than the 2.478 GB Q8_0 reference.
-- **Multilingual stability:** 100% passes on English, Chinese, German, and French.
-| Component | Tensors | Target Dtype | Architectural Mechanism |
-|---|---|---|---|
-| **Audio Tower** | 147 | `BF16 / Q4_K` | Audio encoder layers quantized to Q4_K while keeping converter-selected projection weights at BF16. |
-| **Attention Projections** | 112 (`q`, `k`, `v`, `o_proj`) | `Q4_K` | Quantized within the LM error budget when paired with Q6_K down projections. |
-| **MLP Gate / Up** | 56 (`gate_proj`, `up_proj`) | `Q4_K` | Errors pass through SwiGLU non-linearities, bounding error propagation. |
-| **MLP Down** | 28 (`down_proj`) | **`Q6_K` (Floor)** | **Non-negotiable hard floor.** `down_proj` writes straight into the residual stream, compounding errors across all subsequent layers. Quantizing below Q6_K triggers severe English language drift on non-English audio. |
-| **Embeddings** | 1 (`embed_tokens`) | **`Q2_K`** | Lookup table error enters once and does not compound. Slices 311M parameters down with no measurable fidelity penalty. |
----
-## Why Arm M Works
-### 1. Cumulative Error Budget
-A transformer block cannot be quantized in isolation; its tolerance depends on what the rest of the model has spent:
-- **Da4** (Attention @ Q4_K, `down_proj` @ Q6_K) passes.
-- **E** (Attention @ Q6_K, `down_proj` @ Q4_K) passes simple probes.
-- **G** (Both Attention & `down_proj` @ Q4_K) **fails completely**.
-The LM decoder can afford **exactly one** of {attention, `down_proj`} at Q4_K. Arm M allocates that budget to attention while pinning `down_proj` strictly to Q6_K.
-### 2. `down_proj` @ Q6_K is the Hard Floor
-Every arm evaluated that placed `down_proj` at Q5_K or Q4_K (D5, E, L, N, K, G, G3) emitted English when given German audio. No arm with `down_proj` below Q6_K passes multilingual validation.
-### 3. Embeddings are the Cheapest Lever
-`embed_tokens` has 311M parameters. Quantizing it to **Q2_K** drops the total model size from 1.26 GB (Da4) to **1.187 GB** without triggering degradation.
 ---
-## Arm Evaluation & Benchmarks
-Measured on CPU (pooled ratio against Q8_0 baseline on JFK and ZH test audio):
-| Arm | Attention | Gate/Up | `down_proj` | Embed | Tower | Size (GB) | vs Q8_0 Speed | Status |
-|---|---|---|---|---|---|---|---|---|
-| `r2t2-q8_0` | Q8_0 | Q8_0 | Q8_0 | F16 | BF16/Q8_0 | 2.478 | 0% (baseline) | Reference |
-| `r2t2-q6_k` | Q6_K | Q6_K | Q6_K | F16 | BF16/Q6_K | 2.060 | +5% | Passes |
-| `r2t2-q5_k` | Q5_K | Q5_K | Q5_K | F16 | BF16/Q5_K | 1.832 | **-20% (slower)** | Fails Russian (flips to English) |
-| `r2t2-Da4` | Q4_K | Q4_K | Q6_K | Q4_K | BF16/Q4_K | 1.260 | — | Passes |
-| **`r2t2-q4_k_m` (M)** | **Q4_K** | **Q4_K** | **Q6_K** | **Q2_K** | **BF16/Q4_K** | **1.187** | **+20%+** | **Floor — Passes All** |
-| `r2t2-D5` | Q6_K | Q4_K | **Q5_K** | Q4_K | BF16/Q4_K | 1.304 | +18% | Fails German |
-| `r2t2-E` | Q6_K | Q4_K | **Q4_K** | Q4_K | BF16/Q4_K | 1.260 | +22% | Fails German |
-| `r2t2-G` | Q4_K | Q4_K | **Q4_K** | Q4_K | BF16/Q4_K | 1.169 | — | Fails German |
-| `r2t2-P` | Q4_K | **Q3_K** | Q6_K | Q2_K | BF16/Q4_K | 1.093 | — | Empty output cliff |
-*Note: Q5_K is strictly dominated — it runs ~20% slower than Q8_0 due to non-vectorized kernel routines and fails multilingual validation. Arm M provides both higher throughput and minimal size.*
 ---
@@ -133,7 +98,10 @@ Measured on CPU (pooled ratio against Q8_0 baseline on JFK and ZH test audio):
 | File | Quantization | Size | Description |
 |---|---|---|---|
-| `r2t2-q4_k_m.gguf` | Q4_K_M (Arm M) | 1.187 GB | Highly optimized floor quant; passes multilingual validation |
 ---
@@ -156,11 +124,10 @@ audiocpp_cli --task asr --mode streaming --family confucius4_r2t2   --model r2t2
 ### Server (OpenAI-compatible)
 ```bash
 curl http://127.0.0.1:8488/v1/audio/transcriptions -F model=r2t2-asr -F file=@speech.wav
-curl -N http://127.0.0.1:8488/v1/audio/transcriptions   -F model=r2t2-asr -F stream=true -F file=@speech.wav     # SSE transcript deltas
 ```
-### Streaming Parameters
-Chunk sizes from **80 ms to 2000 ms** are supported (320 ms recommended default). Pass `--language Chinese` to skip language detection and `--text "hotword, term"` for context/hotword prompting.
 ---

   - transcribe.cpp
 ---
+# Confucius4-R2T2 — Q4_K_M GGUF
 Optimized **Q4_K_M GGUF** quantization of **NetEase Youdao Confucius4-R2T2**, a low-latency, append-only streaming ASR model (fine-tuned from Qwen3-ASR-1.7B with Longest Stable Prefix decoding), ready for native execution in [transcribe.cpp](https://github.com/NairoDorian/transcribe.cpp) and [audio.cpp](https://github.com/0xShug0/audio.cpp).
 * **Original model:** [netease-youdao/Confucius4-R2T2](https://huggingface.co/netease-youdao/Confucius4-R2T2) · [source code](https://github.com/netease-youdao/Confucius4-R2T2) · [demo](https://r2t2.youdao.com/demo)
 * **Base Architecture:** Qwen3-ASR audio encoder (24 layers, 1024 width, 2048 projected width) + Causal LM decoder (28 layers, 16 query heads, 8 KV heads, 128 head width).
+* **Reference GGUF repo & inspiration:** [davidxifeng/Confucius4-R2T2-gguf](https://huggingface.co/davidxifeng/Confucius4-R2T2-gguf) by David Xi Feng
+* **Quantization & Benchmarking:** Researched and benchmarked in [transcribe.cpp](https://github.com/NairoDorian/transcribe.cpp).
 ---
+## Technical Overview
+Previous GGUF releases were constrained to Q8_0 or higher because uniform 4-bit and 5-bit quantizations either suffered from catastrophic empty-output cliffs or exhibited silent language drift (e.g. flipping to English when transcribing German or Russian).
+This release introduces an **asymmetric per-tensor quantization recipe** that breaks through that barrier:
+- **File:** `r2t2-q4_k_m.gguf`
+- **File size:** **1.187 GB** (1,186,939,968 bytes) — **52% smaller** than the 2.478 GB Q8_0 reference.
+- **Accuracy & Parity:** Preserves multilingual stability (verified across English, Chinese, German, and French).
+- **Speed:** Faster than Q8_0 on CPU, avoiding the slow non-vectorized paths found in Q5_K.
+### Quantization Composition
+| Block | Target Dtype | Architectural Role |
+|---|---|---|
+| **Audio Tower** | `BF16 / Q4_K` | Audio encoder layers quantized to Q4_K with sensitive projection weights preserved at BF16. |
+| **Attention Projections** | `Q4_K` | Fits within the LM error budget when down projections are held at higher precision. |
+| **MLP Gate / Up** | `Q4_K` | Errors are bounded through SwiGLU activation. |
+| **MLP Down** | **`Q6_K`** | **Hard Floor.** Writes into residual stream; lower precision triggers multilingual degradation. |
+| **Embeddings** | **`Q2_K`** | Lookup table error does not compound; cuts 311M parameters with negligible loss. |
 ---
+## Comprehensive Quantization Report
+For the full ablation study, complete empirical test tables across 20+ quantization arms, failure mode analyses, speed benchmarks, and conversion reproduction instructions:
+👉 **Read the full report in [QUANTIZATION.md](QUANTIZATION.md)**
 ---
 | File | Quantization | Size | Description |
 |---|---|---|---|
+| `r2t2-q4_k_m.gguf` | Q4_K_M | 1.187 GB | High-performance floor quantization; passes multilingual validation |
+| `QUANTIZATION.md` | — | — | Detailed technical report, ablation study, and benchmarks |
+| `NOTICE` | — | — | Attribution and derivative work disclaimers |
+| `LICENSE` / `LICENSE_zh` | — | — | NetEase Youdao Model Use License Agreement |
 ---
 ### Server (OpenAI-compatible)
 ```bash
 curl http://127.0.0.1:8488/v1/audio/transcriptions -F model=r2t2-asr -F file=@speech.wav
+curl -N http://127.0.0.1:8488/v1/audio/transcriptions   -F model=r2t2-asr -F stream=true -F file=@speech.wav
 ```
+Streaming chunk sizes from **80 ms to 2000 ms** are supported (320 ms recommended default).
 ---