| --- |
| base_model: netease-youdao/Confucius4-R2T2 |
| library_name: transcribe.cpp |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - audio |
| - speech-recognition |
| - asr |
| - gguf |
| - quantized |
| - q4_k_m |
| license: other |
| license_name: netease-model-use-license |
| license_link: https://huggingface.co/netease-youdao/Confucius4-R2T2 |
| language: |
| - zh |
| - en |
| - de |
| - fr |
| - ru |
| --- |
| |
| # Confucius4-R2T2 — Q4_K_M GGUF (Arm M) |
|
|
| Optimized **Q4_K_M GGUF** quantization of [netease-youdao/Confucius4-R2T2](https://huggingface.co/netease-youdao/Confucius4-R2T2), designed for native execution in [transcribe.cpp](https://github.com/NairoDorian/transcribe.cpp). |
|
|
| - **Filename:** `r2t2-q4_k_m.gguf` |
| - **File size:** **1.187 GB** (1,186,939,968 bytes) |
| - **RAM/VRAM footprint:** **52% smaller** than the 2.478 GB Q8_0 reference. |
| - **Base Architecture:** Qwen3-ASR audio encoder (24 layers, 1024 width, 2048 projected width) + Causal LM decoder (28 layers, 16 query heads, 8 KV heads, 128 head width). |
| - **Base Checkpoint:** `185ce639118ad1362d049ca0d8ed04b6ec5cd6c9` |
| |
| --- |
| |
| ## Quantization Summary: Arm M |
| |
| Uniform quantization breaks multilingual stability on R2T2. Arm M uses an **asymmetric per-tensor quantization recipe** that reaches the minimum empirical memory floor while preventing language drift and degradation: |
| |
| | Component | Tensors | Target Dtype | Architectural Rationale | |
| |---|---|---|---| |
| | **Audio Tower** | 147 | `BF16 / Q4_K` | Audio encoder layers quantized to Q4_K while keeping converter-selected projection weights at BF16. | |
| | **Attention Projections** | 112 (`q`, `k`, `v`, `o_proj`) | `Q4_K` | Quantized within the LM error budget when paired with Q6_K down projections. | |
| | **MLP Gate / Up** | 56 (`gate_proj`, `up_proj`) | `Q4_K` | Errors pass through SwiGLU non-linearities, bounding error propagation. | |
| | **MLP Down** | 28 (`down_proj`) | **`Q6_K` (Floor)** | **Non-negotiable hard floor.** `down_proj` writes straight into the residual stream, compounding errors across all subsequent layers. Quantizing below Q6_K triggers severe English language drift on non-English audio. | |
| | **Embeddings** | 1 (`embed_tokens`) | **`Q2_K`** | Lookup table error enters once and does not compound. Slices 311M parameters down with no measurable fidelity penalty. | |
|
|
| --- |
|
|
| ## Why Arm M Works |
|
|
| ### 1. The Error Budget is Cumulative |
| A transformer block cannot be quantized in isolation; its tolerance depends on what the rest of the model has spent: |
| - **Da4** (Attention @ Q4_K, `down_proj` @ Q6_K) passes. |
| - **E** (Attention @ Q6_K, `down_proj` @ Q4_K) passes simple probes. |
| - **G** (Both Attention & `down_proj` @ Q4_K) **fails completely**. |
| |
| The LM decoder can afford **exactly one** of {attention, `down_proj`} at Q4_K. Arm M allocates that budget to attention while pinning `down_proj` strictly to Q6_K. |
|
|
| ### 2. `down_proj` @ Q6_K is the Hard Floor |
| Every arm evaluated that placed `down_proj` at Q5_K or Q4_K (D5, E, L, N, K, G, G3) emitted English when given German audio. No arm with `down_proj` below Q6_K passes multilingual validation. |
| |
| ### 3. Embeddings are the Cheapest Lever |
| `embed_tokens` is 311M parameters. Quantizing it to **Q2_K** drops the total model size from 1.26 GB (Da4) to **1.187 GB** without triggering degradation. |
| |
| ### 4. Low-Bit Matmul Cliff |
| Matmuls below Q4_K exhibit a sharp failure cliff rather than gradual degradation: |
| - Tower @ Q3_K (`r2t2-Dt3`) -> **empty output**. |
| - Gate/Up @ Q3_K (`r2t2-P`) -> **empty output**. |
| |
| --- |
| |
| ## Arm Evaluation & Benchmarks |
| |
| Measured on CPU (pooled ratio against Q8_0 baseline on JFK and ZH test audio): |
| |
| | Arm | Attention | Gate/Up | `down_proj` | Embed | Tower | Size (GB) | vs Q8_0 Speed | Status | |
| |---|---|---|---|---|---|---|---|---| |
| | `r2t2-q8_0` | Q8_0 | Q8_0 | Q8_0 | F16 | BF16/Q8_0 | 2.478 | 0% (baseline) | Reference | |
| | `r2t2-q6_k` | Q6_K | Q6_K | Q6_K | F16 | BF16/Q6_K | 2.060 | +5% | Passes | |
| | `r2t2-q5_k` | Q5_K | Q5_K | Q5_K | F16 | BF16/Q5_K | 1.832 | **-20% (slower)** | Fails Russian (flips to English) | |
| | `r2t2-Da4` | Q4_K | Q4_K | Q6_K | Q4_K | BF16/Q4_K | 1.260 | — | Passes | |
| | **`r2t2-q4_k_m` (M)** | **Q4_K** | **Q4_K** | **Q6_K** | **Q2_K** | **BF16/Q4_K** | **1.187** | **+20%+** | **Floor — Passes All** | |
| | `r2t2-D5` | Q6_K | Q4_K | **Q5_K** | Q4_K | BF16/Q4_K | 1.304 | +18% | Fails German | |
| | `r2t2-E` | Q6_K | Q4_K | **Q4_K** | Q4_K | BF16/Q4_K | 1.260 | +22% | Fails German | |
| | `r2t2-G` | Q4_K | Q4_K | **Q4_K** | Q4_K | BF16/Q4_K | 1.169 | — | Fails German | |
| | `r2t2-P` | Q4_K | **Q3_K** | Q6_K | Q2_K | BF16/Q4_K | 1.093 | — | Empty output cliff | |
| |
| *Note: Q5_K is strictly dominated — it runs ~20% slower than Q8_0 due to non-vectorized kernel routines and fails multilingual validation. Arm M provides both higher throughput and minimal size.* |
| |
| --- |
| |
| ## Validation Protocol |
| |
| - **Probing:** Tested on English (`jfk`), Chinese (`zh`), German (`samples/german.wav`), and native French audio sets. |
| - **Detector:** Evaluated via `detected-language:` outputs and strict word-level diffing against the Q8_0 reference. Arm M matches Q8_0 transcriptions word-for-word without language switching. |
|
|
| --- |
|
|
| ## Usage |
|
|
| ### With `transcribe-cli` |
| ```bash |
| transcribe-cli -m r2t2-q4_k_m.gguf samples/audio.wav |
| ``` |
|
|
| ### Streaming |
| R2T2 supports configurable streaming chunk durations between 80 ms and 2000 ms (default: 320 ms). |
|
|
| --- |
|
|
| ## Provenance and License |
|
|
| - **Original Model:** [NetEase Youdao Confucius4-R2T2](https://huggingface.co/netease-youdao/Confucius4-R2T2) |
| - **License:** Model weights are distributed under the [NetEase Model Use License Agreement](https://huggingface.co/netease-youdao/Confucius4-R2T2). |
|
|
