onnx-int4/README.md

# AgenticASR-Refiner ONNX (INT4)

INT4 weight-only quantization of the Optimum ONNX export of
`Andrew0425/AgenticASR-Refiner` (Llama-based ASR transcript refiner, 24 layers,
hidden 1536, GQA 16/2, head_dim 128, vocab 130560).

- `model.onnx` + `model.onnx.data` — INT4 (MatMulNBits, block size 32, symmetric,
  accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export
  (`input_ids` / `attention_mask` / `position_ids` / `past_key_values.*`).
- Quantized with `onnxruntime 1.24.4` `MatMulNBitsQuantizer`
  (`onnxruntime.quantization.matmul_nbits_quantizer`).
- Requires ONNX Runtime >= 1.20 (CPU EP supports `MatMulNBits`).

Tokenization uses the original repo tokenizer (`tokenizer.json` at the repo root).

## Verified generation (ONNX Runtime 1.28, CPU)

| Input | Output |
|---|---|
| 我今天去了公司然后然后开了个会，明天再去见张总 | 我今天去了公司然后开了个会，明天再去见张总 |
| 你好你好你好我是那个小李啊 电话是13800138000 | 你好我是小李，电话是13800138000 |

## Usage

```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
input_names = [i.name for i in session.get_inputs()]
kv_names = [n for n in input_names if n.startswith("past_key_values")]

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意，最小修改。"},
     {"role": "user", "content": "我今天去了公司然后然后开了个会"}],
    tokenize=False, add_generation_prompt=True,
)
ids = tokenizer(prompt).input_ids
pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]

# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts
```

Note: `optimum-onnx 0.1.0` cannot yet run this checkpoint (dummy KV-cache shape
uses `hidden_size // num_heads` = 96 instead of `head_dim` = 128); use the raw
ONNX Runtime loop above, or a future fixed version of optimum-onnx.