Model card in the style of the MLX conversion's

README.md

@@ -1,157 +1,89 @@
 ---
 license: other
 license_name: netease-model-use-license-agreement
-license_link: https://raw.githubusercontent.com/netease-youdao/Confucius4-R2T2/refs/heads/master/MODEL_LICENSE
+license_link: https://github.com/netease-youdao/Confucius4-R2T2/blob/master/MODEL_LICENSE
 base_model: netease-youdao/Confucius4-R2T2
 pipeline_tag: automatic-speech-recognition
 library_name: coreml
 tags:
 - coreml
 - apple-neural-engine
-- apple-silicon
+- speech-to-text
 - asr
-- speech-recognition
+- stt
 - streaming
-- real-time
-- qwen3-asr
 - confucius4
 - r2t2
-language:
-- en
-- pt
-- zh
-- es
-- fr
-- de
-- it
-- ja
-- ko
-- ru
 ---
 
-# Confucius4-R2T2 for Core ML and the Apple Neural Engine
+# Confucius4-R2T2, Core ML for the Apple Neural Engine
 
-> Any modifications made to the original model in this Derivative Work are not endorsed, warranted,
-> or guaranteed by the original right-holder of the original model, and the original right-holder
-> disclaims all liability related to this Derivative Work.
-
-This is [NetEase Youdao's Confucius4-R2T2](https://huggingface.co/netease-youdao/Confucius4-R2T2)
+[`netease-youdao/Confucius4-R2T2`](https://huggingface.co/netease-youdao/Confucius4-R2T2)
 (a real-time speech-recognition fine-tune of Qwen3-ASR-1.7B, 30 languages) converted to Core ML
-programs that run almost entirely on the Apple Neural Engine: every op of the encoder, 99 % of the
-decoder's, and the head's, with the single-row argmax and a dozen index ops per decoder function on
-the CPU. No re-training or fine-tuning was done; the weights are the original ones, converted and
-palettised.
+programs that run on the Apple Neural Engine: the audio encoder in fp16, the Qwen3 decoder and the
+LM head palettised to 8 bits. 4.09 GB of bfloat16 becomes 2.8 GB on disk; while loaded, the
+Neural Engine holds the weights dequantised to fp16, about 4.5 GB, outside the process.
+
+Converted with the pipeline in the VoiceInk fork's `tools/r2t2-coreml`: the encoder through
+coremltools 9 with a fixed 800-frame window (the fused `gelu` replaced by a tanh formulation, whose
+constant absolute error otherwise swamps this encoder's small activations), the decoder through
+[ANEMLL](https://github.com/Anemll/Anemll) as two stateful chunks with a 128-row `prefill` and a
+1-row `infer` function, and a head with a `verify` function that returns the argmax of 128 rows
+in one call. Requires macOS 15 or later (Core ML stateful models) on Apple silicon.
 
-It is the model used by the R2T2 provider in a fork of [VoiceInk](https://github.com/Beingpax/VoiceInk),
-whose runtime and conversion pipeline are published with that fork. The files here are not a
-general-purpose Core ML model with a single input and output: the decoder is a stateful,
-multifunction program that the runtime drives one pass at a time (see *How it is run*).
+Word error rate, whole utterances, 100-utterance subsets: LibriSpeech test-clean 2.40 %, FLEURS
+pt_br 3.50 %. Against the unconverted bfloat16 weights on MLX, paired on 300 utterances of each:
+2.47 % vs 2.42 % and 4.01 % vs 4.01 %.
 
 ## Contents
 
-| file | what | size |
+| file | | size |
 |---|---|---|
-| `R2T2AudioEncoder.mlmodelc` | audio encoder, fp16, one fixed 800-frame (8 s) mel window with a key mask → 104 rows | 607 MB |
-| `r2t2_FFN_PF_lut8_chunk_01of02.mlmodelc` | decoder layers 0–13, LUT8, functions `prefill` (128 rows) and `infer` (1 row), stateful KV cache | 692 MB |
-| `r2t2_FFN_PF_lut8_chunk_02of02.mlmodelc` | decoder layers 14–27 + final norm, same functions | 692 MB |
-| `r2t2_lm_head_lut8.mlmodelc` | language-model head, LUT8, 16-way split: `infer` (one row → logits) and `verify` (128 rows → 128 argmaxes) | 306 MB |
-| `embed_tokens.f16.bin` | token embedding table, fp16, 151 936 × 2048, memory-mapped by the runtime | 622 MB |
+| `R2T2AudioEncoder.mlmodelc` | encoder, fp16, `[1, 128, 800]` mel window + key mask → `[1, 104, 2048]` | 607 MB |
+| `r2t2_FFN_PF_lut8_chunk_01of02.mlmodelc` | decoder layers 0–13, functions `prefill` (128 rows) and `infer` (1 row) | 692 MB |
+| `r2t2_FFN_PF_lut8_chunk_02of02.mlmodelc` | decoder layers 14–27 and the final norm, same functions | 692 MB |
+| `r2t2_lm_head_lut8.mlmodelc` | head, 16-way split: `infer` (one row → `logits1…16`), `verify` (128 rows → argmaxes) | 306 MB |
+| `embed_tokens.f16.bin` | token embeddings, 151 936 × 2048 fp16, row-major, no header | 622 MB |
 | `tokenizer.json`, `tokenizer_config.json` | the original tokenizer | 11 MB |
-| `MODEL_LICENSE`, `LICENSE-Qwen3-ASR.txt`, `NOTICE`, `SHA256SUMS` | licences, attribution, checksums | |
-
-Requirements: Apple silicon and macOS 15 or later (the decoder uses Core ML stateful models).
-While loaded, the Neural Engine holds the weights dequantised to fp16, about 4.5 GB, outside the
-process; the process itself uses about 200 MB. The first load by a given application compiles the
-programs for the Neural Engine, which takes about 50 s; the compiled programs are cached per
-application binary, so later loads take about a second, and a rebuilt or updated application pays
-the compile once more. The cache also needs free disk: with a nearly full disk (under ~20 GB) every
-load recompiled.
-
-## Interface
-
-The files are meant for a runtime that drives them; they are not a drop-in `MLModel` with audio in
-and text out.
-
-- **Encoder** `R2T2AudioEncoder.mlmodelc`: input `[1, 128, 800]` log-mel frames (Whisper's recipe:
-  16 kHz, n_fft 400, hop 160, Slaney filterbank, `log10`, clamped to 8 dB below the buffer's maximum,
-  `(x + 4) / 4`) plus a key mask for windows shorter than 800 frames (masked keys get −10 000);
-  output `[1, 104, 2048]` rows, 13 per 100 frames.
-- **Decoder chunks**: functions `prefill` (B = 128) and `infer` (B = 1) with inputs
-  `hidden_states [1, B, 2048]` fp16, `position_ids [B]` int32, `causal_mask [1, 1, B, 1024]` fp16
-  (0 to attend, −10 000 otherwise), `current_pos [1]` int32; output `output_hidden_states [1, B, 2048]`
-  (chunk 2's is final-normalised). Both chunks share one `MLState` of shape `(56, 8, 1024, 128)` fp16:
-  layer *l* of chunk *c* uses slots *l* (K) and *28 + l* (V). Rows are written by absolute position,
-  so a caller can rewind and overwrite. Context length 1024.
-- **Head**: `infer` takes `hidden_states [1, 1, 2048]` and returns `logits1…logits16`, each
-  `[1, 1, 9496]` (the vocabulary of 151 936 in 16 slices); `verify` takes `[1, 128, 2048]` and returns
-  per row and per slice `argmax_val`, `argmax_hi`, `argmax_lo` (`[128, 16]` fp16 each), the slice's
-  best logit and its index as `hi × 64 + lo`, exact in fp16.
-- **Embeddings** `embed_tokens.f16.bin`: 151 936 rows of 2048 fp16 values, row-major, no header.
-- The decoder programs load under `.cpuAndNeuralEngine` only: `.all` and `.cpuAndGPU` fail to
-  compile, and `.cpuOnly` cannot load the multifunction packages. Python `coremltools` cannot open
-  the combined packages either; drive them from Swift or Objective-C.
-
-## Accuracy
-
-Word error rate of this conversion, decoding whole utterances, on 100-utterance subsets:
-
-| corpus | this conversion |
-|---|---|
-| LibriSpeech test-clean (English) | 2.40 % |
-| FLEURS pt_br (Brazilian Portuguese) | 3.50 % |
-
-Against the unconverted BF16 weights run on MLX, paired on 300 utterances of each corpus, the
-conversion measured 2.47 % vs 2.42 % (ratio 1.02, 95 % CI 0.96–1.10) on LibriSpeech and
-4.01 % vs 4.01 % (ratio 1.00, CI 0.95–1.05) on FLEURS. The streaming runtime described below ends
-on the same transcript as the whole-file decode.
-
-## How it is run
-
-The runtime that uses these files decodes the whole current piece of audio (up to 30 s) on every
-pass, so the live text converges on the same result as an offline decode. What keeps that cheap
-is what stays in the decoder's KV state between passes: the prompt prefix and the rows of every
-completed encoder window. A pass encodes the partial last window, prefills the new audio rows,
-the prompt scaffold and the previous pass's tokens as a draft, checks the draft with one call of
-the head's `verify` function, and decodes one token at a time only from the first divergence.
-Passes take about 150–250 ms on an M5 Pro; a 30 s piece decodes offline in about 2.5 s
-(20 ms per token).
-
-The prompt is the Qwen3-ASR chat template: system text (hotwords may go here), the user turn with
-`<|audio_start|>`, the encoder's output rows in place of the `<|audio_pad|>` token embeddings,
-`<|audio_end|>`, and the assistant turn, optionally prefixed with `language <Name><asr_text>` to
-force a language. The model answers `language <Name><asr_text>` followed by the text; `|` marks
-where it stops trusting its own output.
-
-## Conversion
-
-Encoder: fp16, the network rewritten in the layout the Neural Engine compiler keeps resident
-(the fused `gelu` op replaced by a tanh formulation, because its constant absolute error is large
-against this encoder's small activations), fixed 800-frame window with a key mask for shorter
-input; 4786/4786 ops on the Neural Engine, 55 dB SNR against the PyTorch encoder.
-
-Decoder: through [ANEMLL](https://github.com/Anemll/Anemll) (patched to return the normalised
-hidden states of every prefill row), two chunks, context 1024, 8-bit palettised weights, a
-128-row `prefill` and a 1-row `infer` function sharing one KV state; prefill and single-step paths
-agree to cosine 1.0. A 6-bit build was measured and rejected (WER ratio 1.16 on Portuguese).
-
-Head: the 16-way split head of the original conversion plus a `verify` function that returns the
-argmax of 128 hidden rows in one call, computed exactly on the Neural Engine.
-
-## Licence
-
-Dual licensing, as for the original model:
-
-- **Weights** (everything in this repository derived from the model): the
-  [NetEase Youdao Model Use License Agreement](MODEL_LICENSE). Using these files means accepting
-  it. In short: royalty-free use including commercial use, with a separate licence required above
-  100 million monthly active users or RMB 1 billion in annual revenue; no use to improve other AI
-  models; no high-risk deployments; keep the NOTICE and MODEL_LICENSE with every copy; anyone you
-  redistribute to is bound by the same terms. The Chinese version of the agreement prevails.
-- **Base model** Qwen3-ASR-1.7B: Apache License 2.0 ([LICENSE-Qwen3-ASR.txt](LICENSE-Qwen3-ASR.txt)).
-- **Conversion pipeline and runtime code**: part of the VoiceInk fork, which is GPL-3.0 like
-  VoiceInk itself. Nothing in this repository is code.
-
-## Attribution
-
-Confucius4-R2T2 by NetEase Youdao. Qwen3-ASR by Alibaba Cloud. Core ML conversion tooling built on
-ANEMLL and coremltools. See [NOTICE](NOTICE).
+| `MODEL_LICENSE`, `LICENSE-Qwen3-ASR.txt`, `NOTICE`, `SHA256SUMS` | | |
+
+## Use
+
+These files are driven by the R2T2 provider of a VoiceInk fork, which downloads them from here
+after the licence is accepted. The runtime re-decodes the whole current piece of audio (up to 30 s)
+on every pass, so the live text converges on the same result as an offline decode; what makes that
+cheap is keeping the prompt prefix and every completed encoder window in the decoder's KV state,
+prefilling only the new rows plus the previous pass's tokens as a draft, confirming the draft with
+one `verify` call and decoding only from the first divergence. Passes take 150–250 ms on an M5 Pro;
+a 30 s piece decodes offline in about 2.5 s.
+
+For another runtime, the interface:
+
+- Decoder inputs `hidden_states [1, B, 2048]` fp16, `position_ids [B]` int32,
+  `causal_mask [1, 1, B, 1024]` fp16 (0 to attend, −10 000 otherwise), `current_pos [1]` int32;
+  output `output_hidden_states [1, B, 2048]`. Both chunks share one `MLState` of shape
+  `(56, 8, 1024, 128)` fp16, written by absolute position, so a caller can rewind and overwrite.
+- Head `verify` output per row and slice: `argmax_val`, `argmax_hi`, `argmax_lo` (`[128, 16]` fp16),
+  the slice's best logit and its index as `hi × 64 + lo`.
+- Prompt: the Qwen3-ASR chat template, with the encoder rows in place of the `<|audio_pad|>`
+  embeddings and, to force a language, the assistant turn opened with `language <Name><asr_text>`.
+  The model's `|` marks where it stops trusting its own output.
+- Mel: Whisper's recipe (16 kHz, n_fft 400, hop 160, 128 Slaney bins, `log10`, clamped to 8 dB below
+  the buffer's maximum, `(x + 4) / 4`).
+- The decoder loads under `.cpuAndNeuralEngine` only, and Python coremltools cannot open the
+  multifunction packages. The first load by an application compiles the programs (about 50 s);
+  they are cached per application binary afterwards.
+
+## Notice
+
+Any modifications made to the original model in this Derivative Work are not endorsed, warranted,
+or guaranteed by the original right-holder of the original model, and the original right-holder
+disclaims all liability related to this Derivative Work.
+
+This is a conversion of NetEase Youdao's Confucius4-R2T2 and is governed by the original
+[MODEL_LICENSE](MODEL_LICENSE). That licence is royalty-free for most users, including commercial
+use, but requires a separate licence from NetEase Youdao above 100 million monthly active users or
+RMB 1 billion in annual revenue, forbids using the model to improve other AI models and use in the
+high-risk scenarios it lists, and binds anyone you redistribute it to. Using these files means
+accepting it; keep `MODEL_LICENSE` and `NOTICE` with every copy. The base model, Qwen3-ASR-1.7B, is
+Apache 2.0 (`LICENSE-Qwen3-ASR.txt`).