opus-mt-en-gmq-onnx

ONNX export (fp32 + dynamic int8 quantized) of Helsinki-NLP/opus-mt-en-gmq, a Marian translation model from the Helsinki-NLP OPUS-MT project.

License: apache-2.0 (inherited from the base model; verify at the source link above).

Export

optimum-cli export onnx --model Helsinki-NLP/opus-mt-en-gmq --task text2text-generation-with-past <out>

Quantized to int8 with onnxruntime.quantization.quantize_dynamic (QUInt8 weights).

File layout

./                      fp32 ONNX graphs (encoder_model.onnx, decoder_model.onnx, decoder_with_past_model.onnx) + tokenizer files
./int8/                 int8 dynamic-quantized ONNX graphs

fp32 size: ~1137.3 MB | int8 size: ~528.6 MB

Parity check

Compared PyTorch (MarianMTModel) vs ONNX fp32 (ORTModelForSeq2SeqLM) on 2 sentences, greedy and beam=4 (max_new_tokens=64). Overall: greedy PASS, beam4 PASS.

• src: >>swe<< Hello, how are you today?
  • pytorch greedy: Hej, hur mår du idag?
  • onnx fp32 greedy: Hej, hur mår du idag? (match)
  • pytorch beam4: Hej, hur mår du idag?
  • onnx fp32 beam4: Hej, hur mår du idag? (match)
  • onnx int8 greedy: Hej, hur mår du idag?
• src: Thank you very much.
  • pytorch greedy: Tusen takk.
  • onnx fp32 greedy: Tusen takk. (match)
  • pytorch beam4: Tusen takk.
  • onnx fp32 beam4: Tusen takk. (match)
  • onnx int8 greedy: Tusen takk.

Multi-target verification

This is a multi-target checkpoint: a sentence-initial >>id<< language token is REQUIRED to select the output language (omitting it lets the model pick an arbitrary target). Verified with 3 different target tags that outputs are non-empty and mutually distinct:

• >>dan<< Hello, how are you today? -> Hej, hvordan har du det i dag?
• >>swe<< Hello, how are you today? -> Hej, hur mår du idag?
• >>isl<< Hello, how are you today? -> Hallķ, hvernig hefurđu ūađ í dag?

Usage

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

repo = "TigreGotico/opus-mt-en-gmq-onnx"
tok = AutoTokenizer.from_pretrained(repo)
model = ORTModelForSeq2SeqLM.from_pretrained(repo)  # fp32; pass subfolder="int8" for the quantized graphs
inputs = tok(">>swe<< Hello, how are you today?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

Exported for the OVOS / TigreGotico offline translation stack.