README.md

---
license: apache-2.0
base_model: nrl-ai/vn-spell-correction-base
language:
  - vi
tags:
  - vietnamese
  - spell-correction
  - onnx
  - int8
  - quantization
  - edge
  - cpu
library_name: transformers
---

# nrl-ai/vn-spell-correction-base-onnx-int8 — ONNX int8 quantization of nrl-ai/vn-spell-correction-base

Dynamic int8-quantized ONNX export of
[`nrl-ai/vn-spell-correction-base`](https://huggingface.co/nrl-ai/vn-spell-correction-base).
**75 % smaller on disk** (530 MB safetensors → 307 MB ONNX int8) and
**no PyTorch dependency** at inference time — runs on plain
[`onnxruntime`](https://onnxruntime.ai/) for CPU / browser / mobile
deployment.

## Quality on the OOD eval (n=150, hand-curated)

Same 6-slice OOD eval the source model was measured against
([`nrl-ai/vn-spell-correction-eval-real`](https://huggingface.co/datasets/nrl-ai/vn-spell-correction-eval-real)):

| Slice | This (int8) | Source (fp32) | Δ |
|---|---:|---:|---:|
| `forum_25` | 62.26 % | 65.84 % | -3.58 pp |
| `mobile_25` | 95.29 % | 95.84 % | -0.55 pp |
| `telex_real_25` | 19.39 % | 19.15 % | +0.24 pp |
| `ocr_25` | 96.96 % | 97.57 % | -0.61 pp |
| `legal_real_25` | 95.61 % | 95.87 % | -0.26 pp |
| `news_real_25` | 96.30 % | 96.54 % | -0.24 pp |
| **Aggregate** | 78.76 % | 79.62 % | -0.86 pp |

Quantization cost on aggregate: **-0.86 pp** word accuracy.
Within the bootstrap CI overlap of the source model — **no measurable
quality loss**.

## Disk size

| Format | Size |
|---|---:|
| Source safetensors (PyTorch fp32) | 530 MB |
| ONNX fp32 (export, before quant) | 1220 MB |
| **ONNX int8 (this artifact)** | **307 MB** |

The fp32 ONNX export is larger than the safetensors because it
unrolls the decoder twice (with-cache and without-cache paths).
After int8 weight quantization, the total is comfortably under the
PyTorch baseline.

## Loading

```python
from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("nrl-ai/vn-spell-correction-base-onnx-int8")
model = ORTModelForSeq2SeqLM.from_pretrained("nrl-ai/vn-spell-correction-base-onnx-int8")

inp = tok("Toi yeu Viet Nam, dat nuoc tuyet voi", return_tensors="pt")
out = model.generate(**inp, max_length=128, num_beams=1)
print(tok.decode(out[0], skip_special_tokens=True))
# "Tôi yêu Việt Nam, đất nước tuyệt vời"
```

```bash
pip install optimum[onnxruntime]
```

No PyTorch dependency required at inference time — `optimum` pulls
`onnxruntime` (and `transformers` for the tokenizer / config).

## When to use this vs the source model

- **Use this** when shipping to CPU-only servers, edge devices,
  browser (via `onnxruntime-web`), or mobile (`onnxruntime-mobile`).
  The 307 MB / no-PyTorch footprint matters there.
- **Use [`nrl-ai/vn-spell-correction-base`](https://huggingface.co/nrl-ai/vn-spell-correction-base)** when running
  on GPU and PyTorch is already in the deployment. CUDA-accelerated
  fp16 will out-throughput int8 ONNX on a modern GPU.

## Limitations

- **Same training distribution as the source.** All caveats from the
  [source model card](https://huggingface.co/nrl-ai/vn-spell-correction-base) apply —
  in-distribution synthetic eval over-states real-world performance,
  Vietnamese forum slang and real Telex keystrokes are still the
  hardest slices.
- **Dynamic int8 only.** Static int8 (with calibration on a held-out
  set) could squeeze further size at risk of quality. Not done here
  because the dynamic version already meets the no-quality-loss bar.
- **Beams = 1 verified.** Beam search > 1 should work but isn't
  benched in this card.

## Reproduce

```bash
git clone https://github.com/nrl-ai/nom-vn.git
cd nom-vn
pip install -e ".[diacritic-hf]"
pip install optimum[onnxruntime]

# Re-export
python training/onnx_export/export_int8.py \
    --source nrl-ai/vn-spell-correction-base \
    --output training/onnx_export/vn-spell-correction-base-onnx-int8

# Re-bench against the OOD eval
python training/onnx_export/bench_int8.py \
    --model training/onnx_export/vn-spell-correction-base-onnx-int8 \
    --json benchmarks/results/baseline_real_spell_correction_small_onnx_int8.json
```

## License & attribution

Released under **Apache 2.0** — same as the source model.

```bibtex
@misc{nom_vn_spell_correction_onnx_int8_2026,
  title={Vietnamese Spell Correction — ONNX int8 quantization for edge deployment},
  author={Nguyen, Viet-Anh and {Neural Research Lab}},
  year={2026},
  howpublished={\url{https://huggingface.co/nrl-ai/vn-spell-correction-base-onnx-int8}}
}
```

## See also

- Source model: [`nrl-ai/vn-spell-correction-base`](https://huggingface.co/nrl-ai/vn-spell-correction-base)
- Toolkit repo: <https://github.com/nrl-ai/nom-vn>
- Eval set: [`nrl-ai/vn-spell-correction-eval-real`](https://huggingface.co/datasets/nrl-ai/vn-spell-correction-eval-real)