README.md

---
license: apache-2.0
library_name: cortiq
base_model:
  - tencent/Hy-MT2-1.8B
  - tencent/Hy-MT2-7B
  - tencent/Hy-MT2-30B-A3B
base_model_relation: quantized
pipeline_tag: translation
tags:
  - cmf
  - cortiq
  - quantized
  - q4tp
  - q1t
  - ternary
  - translation
  - moe
  - hunyuan
  - 4-bit
language:
  - zh
  - en
  - fr
  - pt
  - es
  - ja
  - tr
  - ru
  - ar
  - ko
  - th
  - it
  - de
  - vi
  - ms
  - id
  - tl
  - hi
  - pl
  - cs
  - nl
  - km
  - my
  - fa
  - gu
  - ur
  - te
  - mr
  - he
  - bn
  - ta
  - uk
  - bo
  - kk
  - mn
  - ug
---

# Tencent Hy-MT2 — CMF (1.8B · 7B · 30B-A3B)

Tencent's [Hy-MT2](https://huggingface.co/collections/tencent/hy-mt2)
translation family — 33 languages, instruction-following translation
(terminology, style, placeholders, structured data) — as
[CMF](https://github.com/infosave2007/cmf) files for `cortiq`: one
memory-mapped file per model with the weights, the vendor tokenizer, the
exact chat template and per-tensor hashes. The same file runs on the CPU,
on Vulkan (NVIDIA / AMD / Intel), DX12 and Apple Metal; no Python, no
PyTorch, no CUDA toolkit.

```bash
cargo install cortiq-cli --version 0.7.0        # first release with HunYuan support

hf download infosave/Hy-MT2-cmf hy-mt2-7b-q4tp.cmf --local-dir .
cortiq verify hy-mt2-7b-q4tp.cmf
cortiq run hy-mt2-7b-q4tp.cmf --greedy --prompt "Translate the following text into English. Note that you should only output the translated result without any additional explanation:

Сегодня хорошая погода, и мы пойдём гулять в парк."
```

**Requires cortiq 0.7.0 or newer.** Earlier runtimes do not know the
`hunyuan_v1_dense` / `hy_v3` architectures (and 0.6.9 crashes on any
4-bit embedding — fixed in the same release).

## Files

| model | file | source | bits | size | wikitext-2 ppl¹ | MT ppl² |
|---|---|---|---|---:|---:|---:|
| Hy-MT2-1.8B | `hy-mt2-1.8b-q4tp.cmf` | bf16 checkpoint | 4.17 | 950.5 MB | 27.13 | 14.02 |
| Hy-MT2-1.8B | `hy-mt2-1.8b-q1t.cmf` | Tencent's 1.25-bit STQ (AngelSlim) | 2.25 | 694.9 MB | 4 460³ | 27.88 |
| Hy-MT2-7B | `hy-mt2-7b-q4tp.cmf` | bf16 checkpoint | 4.17 | 3.93 GB | 90.3⁴ | 98.6⁴ |
| Hy-MT2-30B-A3B | `hy-mt2-30b-a3b-q4tp.cmf` | bf16 checkpoint | 4.17 | 15.83 GB | 9.88 | 18.12 |

¹ Twelve 512-token windows of wikitext-2 test, exact attention, the same
yardstick every card on this account uses. The bf16-class reference for
the 1.8B (`q8_2f`, not published) scores 24.77 — the 4-bit file
sits within 10% (27.13 vs 24.77) of it. These are translation models: their
perplexity on English encyclopedia prose is not what they were trained
for, which is why column ² exists.

² Sixteen chat-formatted sentence pairs (RU/EN/ZH/DE/FR/ES/JA/IT/UK/PT,
1 073 tokens) in the model's own prompt format — the number that tracks
what the file is for. The `q1t` file is Tencent's own 1.25-bit
quantization-aware checkpoint carried over **exactly** (see below); it
is a translation-only specialist and its general-text perplexity is not
meaningful, but its translations are — see the samples further down.

³ Not a conversion defect: llama.cpp with the STQ1_0 kernel (PR #22836)
scores the very same GGUF at 1 793 on its own wikitext-2 protocol (first
twelve 512-token chunks, BOS-anchored, second halves scored), our
cold-window protocol at 4 460 — both say the 1.25-bit checkpoint no
longer models free English prose, while it translates correctly on every
backend. On the same Xeon that llama.cpp build decodes it at 2.2 tok/s;
cortiq's `q1t` kernels at 21.9.

⁴ The 7B uses HunYuan's older tokenizer (vocab 128 167, GPT-4-style
splitting, `<|startoftext|>` … `<|extra_0|>` chat markup) and is sensitive
to running without its BOS in the middle of a corpus; its translations are
the best of the three dense files. Compare sizes on column ², not here.

Every q4tp file was quantized straight from the bf16 safetensors,
streamed shard by shard; nothing here is a re-quantization of a
lower-precision file. The tied `lm_head` of the dense models is not
written twice: the embedding serves both ends, as in the source.

## What the runtime had to learn

Two things distinguish HunYuan from the Qwen/Llama block, and both ride
in the file header rather than in flags:

- **Dense 1.8B / 7B (`hunyuan_v1_dense`)** apply the per-head q/k RMSNorm
  *after* RoPE (`query_layernorm` / `key_layernorm` on the rotated
  vectors). That is not a reordering you can fold into the weights — the
  rotation preserves a head's norm but not its elementwise norm weights —
  so the engine carries an explicit `qk_norm_after_rope` flag through the
  CPU, Metal and WGSL rope kernels. Their "dynamic" NTK-alpha RoPE
  (`alpha = 1000`) is one rescaled base, `10000 · 1000^(128/126) =
  11 158 840`, applied at conversion; the full 262 144-token window is
  native, nothing is rescaled per position.
- **30B-A3B (`hy_v3`)** is 48 layers with layer 0 dense (6 912 wide) and
  47 sparse layers of 128 experts, top-8 by *sigmoid* score plus a
  selection bias (`expert_bias`, DeepSeek-V3 style: bias for the choice,
  unbiased scores for the weights), renormalized and multiplied by
  `router_scaling_factor = 2.826`, plus one always-on shared expert of
  768. The converter maps `mlp.router.gate` / `mlp.shared_mlp` onto the
  canonical layout and the header carries the router constants; the wgpu
  whole-token graph learned the two pieces it lacked — the routed scale
  and a shared expert without a gate — so the 30B decodes on the card
  as one graph (12 submits per token) instead of falling to the per-op
  path (145 submits, 1.2 tok/s) the way every sigmoid-scaled MoE did
  before 0.7.0.

## The 1.25-bit file, exactly

Tencent ships `Hy-MT2-1.8B-1.25Bit-GGUF`: a quantization-aware ternary
checkpoint in llama.cpp's `STQ1_0` type (256-weight blocks, one f16
scale, and in every group of four lanes exactly one zero and three ±1 —
5 bits per 4 weights). `cortiq import-gguf` decodes that block layout
natively and re-encodes each 32-group as CMF `q1t` (ternary, base-3
packed, f16 scale, empty outlier overlay): the reconstruction is
**bit-for-bit** the values the GGUF stores — no calibration, no second
quantizer. The cost of the general-purpose container is size: 2.25
bits/weight against 1.31, so the file is 695 MB against 462, with the
token table at `q8_2f` (Tencent stores it at 6.5 bits).

```bash
cortiq import-gguf Hy-MT2-1.8B-1.25Bit.gguf --quant q1t \
    --tokenizer-dir ./Hy-MT2-1.8B --output hy-mt2-1.8b-q1t.cmf
```

`--tokenizer-dir` matters: HunYuan's pre-tokenizer splits digits in runs
of one to three and isolates CJK before the byte-level step, which a
tokenizer rebuilt from GGUF metadata cannot express. The flag embeds the
vendor `tokenizer.json` and chat template verbatim, so the ternary file
tokenizes exactly like the 4-bit ones.

## Measured

Steady-state decode, single stream, `cortiq bench --core --tokens 128
--ignore-eos`, cortiq 0.7.1. The dense files are latency-bound on a
discrete card (a 1 GB model needs ~8 submits per token), so the CPU
matters as much as the GPU there; the MoE row is where the card counts.

| file | RTX PRO 4000 Blackwell 24 GB, Vulkan | Xeon E5-2690 v4 (14C), CPU | Apple M4 24 GB, Metal | M4, CPU (10 cores) |
|---|---:|---:|---:|---:|
| `hy-mt2-1.8b-q4tp.cmf` | 137.7 | 29.1 | 72.8 | 56.9 |
| `hy-mt2-1.8b-q1t.cmf` | 63.7 | 21.9 | 66.9 | 38.1 |
| `hy-mt2-7b-q4tp.cmf` | 77.2 | 9.3 | 23.1 | 16.7 |
| `hy-mt2-30b-a3b-q4tp.cmf` | 57.4 | 11.0 | 32.5⁵ | 22.7 |

### The MoE on any card (dynamic loading)

The 30B's whole-token graph is not all-or-nothing: as many leading layers
as the VRAM budget admits stay resident on the card, the host finishes the
rest — one boundary crossing per token, no expert paging. Measured on the
RTX PRO 4000 with `CMF_GPU_VRAM_MB` capped to what each card size would
auto-detect (the CPU alone: 10.5 tok/s on this 14-core Xeon):

| VRAM budget | 4 GB | 6 GB | 8 GB | 12 GB | 16 GB | 24 GB |
|---|---:|---:|---:|---:|---:|---:|
| layers on the card | 7/48 | 14/48 | 20/48 | 33/48 | **48/48** | **48/48** |
| decode, tok/s | 9.0 | 12.5 | 14.2 | 19.6 | 51.9 | **57.4** |

Perplexity and the greedy continuation do not move along the ladder — the
split changes where a layer runs, never what it computes. `CMF_GPU_VRAM_MB`
overrides the auto-detected budget when you want to cap it by hand. The
16 GB point held 45 layers in 0.7.0; with the prompt on the graph (0.7.1)
the per-op prefill arena no longer competes for the budget and the whole
stack fits (16.8 GB resident), so a 16 GB card decodes at the full rate.

⁵ The 15.8 GB file is larger than one Metal buffer (13.6 GB on a 24 GB
M4), so its weights map as two overlapping windows; 0.7.1 taught the
expert kernels to address them (before, every MoE layer of a windowed
file ran on the CPU: 17.9 tok/s, under the M4's own CPU). The experts themselves stream at the
card's bandwidth (~12 ms of the 31 ms token); the rest is the fixed cost
of the ~12 dispatches each of the 48 layers needs, which is why the
dense 7B, with a third of the layers' bytes per layer, is not faster.

Prompt ingest: on the RTX PRO 4000 the dense files take 180 (1.8B) and
118 (7B) tok/s. The 30B ingests at **65 tok/s** (41-token prompt) and
56 tok/s at 512 tokens — through the same resident graph that decodes,
one position at a time; `CMF_BATCH_K=32` switches the prompt to the
batched graph (32 positions per submit) for 80 / 71 tok/s. In 0.7.0 the
30B's prompt went through the chunked host prefill, where every expert
ran on the CPU: 8 tok/s, ten seconds before the first token of a
paragraph. The graph route needs the whole stack on the card; on the
VRAM ladder below the chunked path stays (7–12 tok/s of ingest), because
walking a prompt through a device prefix finishes every position on the
host. On the M4: 469 tok/s for the 1.8B q4tp, 218 for the ternary file,
122 for the 7B, 74 for the 30B.

**First answer vs. the rest.** On a discrete card the weights are uploaded
when the whole-token graph is first built — 18.5 GB for the 30B, ~26 s on
this box — and `cortiq run` folds that into its printed "decode" figure
for a one-shot prompt, which then reads a few tok/s. The per-token cost
after the upload is the table above (`cortiq bench --core` measures it
after an untimed warm-up). `cortiq serve` pays the upload once per
process, so every request after the first decodes at the steady rate.

## Prompting

The models have no system prompt. The chat template is embedded, so
`cortiq run` and `cortiq serve` wrap a plain user message correctly;
what goes into the message is Tencent's instruction, with the language
name written out in the prompt's language:

```
Translate the following text into {target_lang}. Note that you should only output the translated result without any additional explanation:

{source_text}
```

```
将以下文本翻译为{target_lang}，注意只需要输出翻译后的结果，不要额外解释：

{source_text}
```

Terminology, style, personalization, delimiter-preserving and
structured-data prompts are documented on the
[source card](https://huggingface.co/tencent/Hy-MT2-30B-A3B#hy-mt2-translation-task-instruction-examples-chinese-english-comparison).
Recommended sampling (Tencent): 1.8B / 7B — temperature 0.7, top-p 0.6,
top-k 20, repetition penalty 1.05; 30B-A3B — temperature 0.7, top-p 1.0,
no repetition penalty. `--greedy` is the deterministic choice for
evaluation.

Samples from the 1.8B files, greedy, identical on CPU and GPU:

| prompt | q4tp | q1t (1.25-bit) |
|---|---|---|
| RU→EN «Сегодня хорошая погода, и мы пойдём гулять в парк, а вечером посмотрим фильм.» | Today the weather is nice, and we're going to go for a walk in the park. In the evening, we'll watch a movie. | Today, the weather is nice. We'll go for a walk in the park, and in the evening, we'll watch a movie. |
| EN→DE "Please keep the {placeholder} and the number 12345 exactly as they are." | Bitte behalten Sie den {Placeholder} und die Zahl 12345 genau so, wie sie sind. | Bitte lassen Sie {placeholder} und die Zahl 12345 genau so, wie sie sind. |
| ZH→EN «今天天气真好，我们去公园散步吧。» | The weather is really nice today. Let's go for a walk in the park. | The weather is really nice today; let's go for a walk in the park. |

## Server

```bash
cortiq serve hy-mt2-7b-q4tp.cmf --port 8080
curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"hy-mt2-7b-q4tp","temperature":0,"messages":[{"role":"user","content":"Translate the following text into French. Note that you should only output the translated result without any additional explanation:\n\nThe meeting is at ten."}]}'
```

`/v1/chat/completions`, `/v1/completions` and `/v1/models` are live;
`--ollama` adds an Ollama-compatible listener.

## Reproduce

```bash
cortiq convert --model tencent/Hy-MT2-1.8B    --quant q4tp --output hy-mt2-1.8b-q4tp.cmf
cortiq convert --model tencent/Hy-MT2-7B      --quant q4tp --output hy-mt2-7b-q4tp.cmf
cortiq convert --model tencent/Hy-MT2-30B-A3B --quant q4tp --output hy-mt2-30b-a3b-q4tp.cmf
```

Streaming from the hub: peak disk is the output file. Every file was
size- and hash-verified after upload (`cortiq verify`).

Weights derive from Tencent's release and remain under its Apache-2.0
terms. The CMF container and the cortiq runtime are Apache-2.0 as well
(see the repository's LICENSE and PATENTS.md).