justinchuby/Hy-MT2-1.8B-ONNX

Hy-MT2-1.8B — ONNX

ONNX builds of tencent/Hy-MT2-1.8B, a fast multilingual (33-language) translation model, exported with mobius and packaged for ONNX Runtime GenAI.

Variants

Folder	Precision	EP / Device
f16/cuda	FP16	CUDA
f16/default	FP16	Default (CPU / DML / WebGPU)
bf16/cuda	BF16	CUDA
bf16/default	BF16	Default (DML / WebGPU; bf16 MatMul is not on the CPU EP)
Q4_K_M/cuda	INT4 (K-Quant Q4_K_M)	CUDA
Q4_K_M/default	INT4 (K-Quant Q4_K_M)	Default (CPU)
Q4_RTN/cuda	INT4 (RTN; quantized embeddings + LM head + body)	CUDA
Q4_GPTQ/cuda	INT4 (GPTQ body + RTN embeddings / LM head, tied embeddings)	CUDA
Q4_KQuant_tie/cuda	INT4 (K-Quant body + RTN embeddings / LM head, tied embeddings)	CUDA
Q4_KQuant_tie/cpu	INT4 (K-Quant body + RTN embeddings / LM head, tied embeddings)	CPU
Q4_KQuant_tie/webgpu	INT4 (K-Quant body + RTN embeddings / LM head, tied embeddings)	WebGPU

*/cuda packages set the CUDA execution provider in genai_config.json; */default packages leave provider_options empty so ORT GenAI runs them on whatever EP is configured (CPU by default).

Q4_RTN/cuda is fully weight-quantized: in addition to the body projections, the input embedding table (GatherBlockQuantized) and the tied LM head (MatMulNBits) are INT4, giving the smallest footprint.

Q4_GPTQ/cuda quantizes the same layers to INT4 (group size 32) but uses GPTQ (Hessian-aware, wikitext-calibrated) for the transformer body instead of RTN, with RTN only for the embedding table and LM head. The input embedding and LM head share a single tied INT4 table, so it is both more accurate (GPTQ) and 11% smaller on disk (1.03 GiB vs ~1.16 GiB) than Q4_RTN/cuda. Recommended INT4 CUDA variant.

Q4_KQuant_tie/* uses K-Quant (Q4_K_M) for the transformer body and RTN for the input embedding table and LM head, which share a single tied INT4 table. It has the best quality/size trade-off of the INT4 variants (smallest footprint, highest top-1 agreement with FP16). cuda and webgpu are byte-for-byte identical (fp16 scales, 1.03 GiB); cpu stores fp32 scales (1.19 GiB) since the CPU EP prefers fp32.

Each folder is a complete ORT GenAI package (model.onnx, model.onnx.data, genai_config.json, tokenizer, chat template).

Usage (ONNX Runtime GenAI)

import onnxruntime_genai as og
from transformers import AutoTokenizer

path = "Q4_K_M/default"  # or f16/cuda, bf16/cuda, Q4_K_M/cuda
model = og.Model(path)
tok = AutoTokenizer.from_pretrained("tencent/Hy-MT2-1.8B")

src, tgt = "黄河之水天上来", "English"
messages = [{"role": "user",
             "content": f"Translate the following text into {tgt}:\n{src}"}]
prompt = tok.apply_chat_template(messages, tokenize=False,
                                 add_generation_prompt=True)
ids = tok(prompt, return_tensors="np")["input_ids"].astype("int32")

params = og.GeneratorParams(model)
params.set_search_options(max_length=256, do_sample=False)
gen = og.Generator(model, params)
gen.append_tokens(ids[0])
out = []
while not gen.is_done():
    gen.generate_next_token()
    out.append(gen.get_next_tokens()[0])
print(tok.decode(out, skip_special_tokens=True))
# -> The water of the Yellow River comes from the sky

The Hy-MT BPE vocab uses a custom regex pre-tokenizer that ort-extensions does not currently round-trip, so use the HuggingFace tokenizer to encode / decode and feed raw token IDs to og.Generator.

Build recipes

These models were built with the Olive recipes in microsoft/olive-recipes (tencent-Hy-MT2-1.8B): MobiusBuilder for the FP16/BF16 exports, MobiusBuilder → OnnxKQuantQuantization for the INT4 K-Quant variants, rtn → MobiusBuilder for Q4_RTN, gptq → rtn → MobiusBuilder for Q4_GPTQ, and MobiusBuilder → OnnxKQuantQuantization (body) → OnnxBlockWiseRtnQuantization (embeddings) → GraphSurgeries[TieWordEmbeddings] for Q4_KQuant_tie.

License

Inherits the original Tencent Hunyuan license.