Hugging Face's logo

Nairod785
/
Confucius4-R2T2-Q4_K_M-GGUF

Automatic Speech Recognition
GGUF
transcribe.cpp
asr
streaming
real-time
low-latency
quantized
q4_k_m
qwen3-asr
confucius4
r2t2
audio.cpp
Model card Files
xet
 Community
1
Confucius4-R2T2-Q4_K_M-GGUF / README.md
Nairod785's picture
Nairod785
Upload README.md with huggingface_hub
897faa8 verified
preview code
|
Raw History Blame
5.61 kB
metadata 
base_model: netease-youdao/Confucius4-R2T2
library_name: transcribe.cpp
pipeline_tag: automatic-speech-recognition
tags:
  - audio
  - speech-recognition
  - asr
  - gguf
  - quantized
  - q4_k_m
license: other
license_name: netease-model-use-license
license_link: https://huggingface.co/netease-youdao/Confucius4-R2T2
language:
  - zh
  - en
  - de
  - fr
  - ru

Confucius4-R2T2 — Q4_K_M GGUF (Arm M)

Optimized Q4_K_M GGUF quantization of netease-youdao/Confucius4-R2T2, designed for native execution in transcribe.cpp.

  • Filename: r2t2-q4_k_m.gguf
  • File size: 1.187 GB (1,186,939,968 bytes)
  • RAM/VRAM footprint: 52% smaller than the 2.478 GB Q8_0 reference.
  • Base Architecture: Qwen3-ASR audio encoder (24 layers, 1024 width, 2048 projected width) + Causal LM decoder (28 layers, 16 query heads, 8 KV heads, 128 head width).
  • Base Checkpoint: 185ce639118ad1362d049ca0d8ed04b6ec5cd6c9

Quantization Summary: Arm M

Uniform quantization breaks multilingual stability on R2T2. Arm M uses an asymmetric per-tensor quantization recipe that reaches the minimum empirical memory floor while preventing language drift and degradation:

Component Tensors Target Dtype Architectural Rationale
Audio Tower 147 BF16 / Q4_K Audio encoder layers quantized to Q4_K while keeping converter-selected projection weights at BF16.
Attention Projections 112 (q, k, v, o_proj) Q4_K Quantized within the LM error budget when paired with Q6_K down projections.
MLP Gate / Up 56 (gate_proj, up_proj) Q4_K Errors pass through SwiGLU non-linearities, bounding error propagation.
MLP Down 28 (down_proj) Q6_K (Floor) Non-negotiable hard floor. down_proj writes straight into the residual stream, compounding errors across all subsequent layers. Quantizing below Q6_K triggers severe English language drift on non-English audio.
Embeddings 1 (embed_tokens) Q2_K Lookup table error enters once and does not compound. Slices 311M parameters down with no measurable fidelity penalty.

Why Arm M Works

1. The Error Budget is Cumulative

A transformer block cannot be quantized in isolation; its tolerance depends on what the rest of the model has spent:

  • Da4 (Attention @ Q4_K, down_proj @ Q6_K) passes.
  • E (Attention @ Q6_K, down_proj @ Q4_K) passes simple probes.
  • G (Both Attention & down_proj @ Q4_K) fails completely.

The LM decoder can afford exactly one of {attention, down_proj} at Q4_K. Arm M allocates that budget to attention while pinning down_proj strictly to Q6_K.

2. down_proj @ Q6_K is the Hard Floor

Every arm evaluated that placed down_proj at Q5_K or Q4_K (D5, E, L, N, K, G, G3) emitted English when given German audio. No arm with down_proj below Q6_K passes multilingual validation.

3. Embeddings are the Cheapest Lever

embed_tokens is 311M parameters. Quantizing it to Q2_K drops the total model size from 1.26 GB (Da4) to 1.187 GB without triggering degradation.

4. Low-Bit Matmul Cliff

Matmuls below Q4_K exhibit a sharp failure cliff rather than gradual degradation:

  • Tower @ Q3_K (r2t2-Dt3) -> empty output.
  • Gate/Up @ Q3_K (r2t2-P) -> empty output.

Arm Evaluation & Benchmarks

Measured on CPU (pooled ratio against Q8_0 baseline on JFK and ZH test audio):

Arm Attention Gate/Up down_proj Embed Tower Size (GB) vs Q8_0 Speed Status
r2t2-q8_0 Q8_0 Q8_0 Q8_0 F16 BF16/Q8_0 2.478 0% (baseline) Reference
r2t2-q6_k Q6_K Q6_K Q6_K F16 BF16/Q6_K 2.060 +5% Passes
r2t2-q5_k Q5_K Q5_K Q5_K F16 BF16/Q5_K 1.832 -20% (slower) Fails Russian (flips to English)
r2t2-Da4 Q4_K Q4_K Q6_K Q4_K BF16/Q4_K 1.260 — Passes
r2t2-q4_k_m (M) Q4_K Q4_K Q6_K Q2_K BF16/Q4_K 1.187 +20%+ Floor — Passes All
r2t2-D5 Q6_K Q4_K Q5_K Q4_K BF16/Q4_K 1.304 +18% Fails German
r2t2-E Q6_K Q4_K Q4_K Q4_K BF16/Q4_K 1.260 +22% Fails German
r2t2-G Q4_K Q4_K Q4_K Q4_K BF16/Q4_K 1.169 — Fails German
r2t2-P Q4_K Q3_K Q6_K Q2_K BF16/Q4_K 1.093 — Empty output cliff

Note: Q5_K is strictly dominated — it runs ~20% slower than Q8_0 due to non-vectorized kernel routines and fails multilingual validation. Arm M provides both higher throughput and minimal size.

Validation Protocol

  • Probing: Tested on English (jfk), Chinese (zh), German (samples/german.wav), and native French audio sets.
  • Detector: Evaluated via detected-language: outputs and strict word-level diffing against the Q8_0 reference. Arm M matches Q8_0 transcriptions word-for-word without language switching.

Usage

With transcribe-cli 

transcribe-cli -m r2t2-q4_k_m.gguf samples/audio.wav

Streaming

R2T2 supports configurable streaming chunk durations between 80 ms and 2000 ms (default: 320 ms).

Provenance and License