README.md

---
license: other
license_name: netease-model-use-license-agreement
license_link: https://github.com/netease-youdao/Confucius4-R2T2/blob/master/MODEL_LICENSE
base_model: netease-youdao/Confucius4-R2T2
base_model_relation: quantized
pipeline_tag: automatic-speech-recognition
library_name: coreml
tags:
- coreml
- apple-neural-engine
- speech-to-text
- asr
- stt
- streaming
- confucius4
- r2t2
---

# Confucius4-R2T2, Core ML for the Apple Neural Engine

[`netease-youdao/Confucius4-R2T2`](https://huggingface.co/netease-youdao/Confucius4-R2T2)
(a real-time speech-recognition fine-tune of Qwen3-ASR-1.7B, 30 languages) converted to Core ML
programs that run on the Apple Neural Engine: the audio encoder in fp16, the Qwen3 decoder and the
LM head palettised to 8 bits. 4.09 GB of bfloat16 becomes 2.8 GB on disk; while loaded, the
Neural Engine holds the weights dequantised to fp16, about 4.5 GB, outside the process.

Converted with the pipeline in the VoiceInk fork's `tools/r2t2-coreml`: the encoder through
coremltools 9 with a fixed 800-frame window (the fused `gelu` replaced by a tanh formulation, whose
constant absolute error otherwise swamps this encoder's small activations), the decoder through
[ANEMLL](https://github.com/Anemll/Anemll) as two stateful chunks whose KV state holds two
sequences side by side ("lanes"), with a 128-row `prefill` of one lane and an `infer` that
decodes one token of each lane in a single pass over the weights, and a head with a two-row
`infer` and a `verify` function that returns the argmax of 128 rows in one call. Requires macOS 15
or later (Core ML stateful models) on Apple silicon.

Word error rate, whole utterances, 100-utterance subsets: LibriSpeech test-clean 2.40 %, FLEURS
pt_br 3.50 %. Against the unconverted bfloat16 weights on MLX, paired on 300 utterances of each:
2.47 % vs 2.42 % and 4.01 % vs 4.01 %.

## Contents

| file | | size |
|---|---|---|
| `R2T2AudioEncoder.mlmodelc` | encoder, fp16, `[1, 128, 800]` mel window + key mask → `[1, 104, 2048]` | 607 MB |
| `r2t2_FFN_PF_lut8_chunk_01of02.mlmodelc` | decoder layers 0–13, functions `prefill` (128 rows of one lane) and `infer` (one row per lane) | 692 MB |
| `r2t2_FFN_PF_lut8_chunk_02of02.mlmodelc` | decoder layers 14–27 and the final norm, same functions | 692 MB |
| `r2t2_lm_head_lut8.mlmodelc` | head, 16-way split: `infer` (two rows → `logits1…16`), `verify` (128 rows → argmaxes) | 306 MB |
| `embed_tokens.f16.bin` | token embeddings, 151 936 × 2048 fp16, row-major, no header | 622 MB |
| `tokenizer.json`, `tokenizer_config.json` | the original tokenizer | 11 MB |
| `MODEL_LICENSE`, `LICENSE-Qwen3-ASR.txt`, `NOTICE`, `SHA256SUMS` | | |

## Use

These files are driven by the R2T2 provider of a VoiceInk fork, which downloads them from here
after the licence is accepted. The runtime re-decodes the whole current piece of audio (up to 30 s)
on every pass, so the live text converges on the same result as an offline decode; what makes that
cheap is keeping the prompt prefix and every completed encoder window in the decoder's KV state,
prefilling only the new rows plus the previous pass's tokens as a draft, confirming the draft with
one `verify` call and decoding only from the first divergence. Passes take 150–250 ms on an M5 Pro.
Offline, two 30 s pieces decode together, one token of each per `infer` call: a 120 s recording
takes about 6 s (8.7 ms per token; 17 ms for a single piece).

For another runtime, the interface:

- Both chunks share one `MLState` of shape `(56, 8, 2048, 128)` fp16: lane *l* owns cache positions
  `l·1024 ..< (l+1)·1024`, written by absolute position, so a caller can rewind and overwrite. The
  causal mask is 0 to attend, −10 000 otherwise; output `output_hidden_states [1, B, 2048]`.
  - `prefill`: `hidden_states [1, 128, 2048]` fp16, `position_ids [128]` int32 (the lane's own
    positions), `causal_mask [1, 1, 128, 1024]`, `current_pos [1]` int32 (absolute: `l·1024 + p`),
    `lane_weights [2, 1, 1]` fp16 one-hot selecting the lane.
  - `infer`: `hidden_states [1, 2, 2048]` (one row per lane), `position_ids [2]`,
    `causal_mask [1, 1, 4, 1024]` (each lane's row twice: the two query heads of a K/V head share a
    matmul row axis), `current_pos [1]` and `current_pos_b [1]` (lane 0 and lane 1, absolute). An
    idle lane takes a zero row, a fully masked row and a scratch position its next real row overwrites.
- Head `infer` takes `hidden_states [1, 2, 2048]` and returns `logits1…16` for both rows; `verify`
  takes 128 rows and returns per row and slice `argmax_val`, `argmax_hi`, `argmax_lo` (`[128, 16]`
  fp16), the slice's best logit and its index as `hi × 64 + lo`.
- Prompt: the Qwen3-ASR chat template, with the encoder rows in place of the `<|audio_pad|>`
  embeddings and, to force a language, the assistant turn opened with `language <Name><asr_text>`.
  The model's `|` marks where it stops trusting its own output.
- Mel: Whisper's recipe (16 kHz, n_fft 400, hop 160, 128 Slaney bins, `log10`, clamped to 8 dB below
  the buffer's maximum, `(x + 4) / 4`).
- The decoder loads under `.cpuAndNeuralEngine` only, and Python coremltools cannot open the
  multifunction packages. The first load by an application compiles the programs (about 50 s);
  they are cached per application binary afterwards.

## Notice

Any modifications made to the original model in this Derivative Work are not endorsed, warranted,
or guaranteed by the original right-holder of the original model, and the original right-holder
disclaims all liability related to this Derivative Work.

This is a conversion of NetEase Youdao's Confucius4-R2T2 and is governed by the original
[MODEL_LICENSE](MODEL_LICENSE). That licence is royalty-free for most users, including commercial
use, but requires a separate licence from NetEase Youdao above 100 million monthly active users or
RMB 1 billion in annual revenue, forbids using the model to improve other AI models and use in the
high-risk scenarios it lists, and binds anyone you redistribute it to. Using these files means
accepting it; keep `MODEL_LICENSE` and `NOTICE` with every copy. The base model, Qwen3-ASR-1.7B, is
Apache 2.0 (`LICENSE-Qwen3-ASR.txt`).