File size: 2,132 Bytes
a786ce7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | # AgenticASR-Refiner ONNX (INT4)
INT4 weight-only quantization of the Optimum ONNX export of
`Andrew0425/AgenticASR-Refiner` (Llama-based ASR transcript refiner, 24 layers,
hidden 1536, GQA 16/2, head_dim 128, vocab 130560).
- `model.onnx` + `model.onnx.data` — INT4 (MatMulNBits, block size 32, symmetric,
accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export
(`input_ids` / `attention_mask` / `position_ids` / `past_key_values.*`).
- Quantized with `onnxruntime 1.24.4` `MatMulNBitsQuantizer`
(`onnxruntime.quantization.matmul_nbits_quantizer`).
- Requires ONNX Runtime >= 1.20 (CPU EP supports `MatMulNBits`).
Tokenization uses the original repo tokenizer (`tokenizer.json` at the repo root).
## Verified generation (ONNX Runtime 1.28, CPU)
| Input | Output |
|---|---|
| 我今天去了公司然后然后开了个会，明天再去见张总 | 我今天去了公司然后开了个会，明天再去见张总 |
| 你好你好你好我是那个小李啊 电话是13800138000 | 你好我是小李，电话是13800138000 |
## Usage
```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
input_names = [i.name for i in session.get_inputs()]
kv_names = [n for n in input_names if n.startswith("past_key_values")]
prompt = tokenizer.apply_chat_template(
[{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意，最小修改。"},
{"role": "user", "content": "我今天去了公司然后然后开了个会"}],
tokenize=False, add_generation_prompt=True,
)
ids = tokenizer(prompt).input_ids
pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]
# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts
```
Note: `optimum-onnx 0.1.0` cannot yet run this checkpoint (dummy KV-cache shape
uses `hidden_size // num_heads` = 96 instead of `head_dim` = 128); use the raw
ONNX Runtime loop above, or a future fixed version of optimum-onnx.
|