---
license: apache-2.0
base_model:
  - tencent/Hy-MT2-1.8B
  - febilly/Hy-MT2-1.8B-StreamRevise-LoRA
base_model_relation: quantized
pipeline_tag: translation
tags:
  - translation
  - streaming-translation
  - simultaneous-translation
  - subtitles
  - speech-translation
  - gguf
  - llama.cpp
  - quantized
language:
  - zh
  - en
  - ja
---

# Hy-MT2-1.8B-StreamRevise — GGUF

4-bit llama.cpp build of **StreamRevise**, a translation model for **live subtitles**. Every time the ASR
hypothesis updates, you pass the model its own previous translation of the sentence in progress; it decides
whether to keep that text and extend it, or rewrite part of it because the meaning changed. Subtitles grow
smoothly instead of flickering.

**1.07 GB on disk, and about 1.4 GB resident at `n_ctx 2048`.** Small enough to ship with a desktop app.
See [Footprint](#footprint) for how that scales.

Adapter, full prompt spec, training details:
[**Hy-MT2-1.8B-StreamRevise-LoRA**](https://huggingface.co/febilly/Hy-MT2-1.8B-StreamRevise-LoRA) ·
Base model: [`tencent/Hy-MT2-1.8B`](https://huggingface.co/tencent/Hy-MT2-1.8B)

> **中文简介**：实时字幕翻译模型的 4-bit 本地部署版。语音边说边识别，源文一直在变——每次更新时把模型上一版
> 译文一起喂回去，它自己决定保留已显示的部分继续接，还是因为意思变了而改写，让字幕平滑增长不闪烁。
> 1.07 GB，纯 CPU 可跑（约 2.1 GB 内存），有显卡则单次请求 <0.1 秒。

---

## Files

| file | size | what it is |
|---|---|---|
| `Hy-MT2-1.8B-StreamRevise-Q4_K_M.gguf` | 1.07 GB | **the model.** Q4_K_M with imatrix calibration, token embeddings at `Q4_K` |
| `Hy-MT2-1.8B-StreamRevise.imatrix.gguf` | 2.3 MB | importance matrix, only needed if you want to re-quantize at another bit width |

Only one quantization is published. The alternatives were measured and the differences sit inside the noise
floor, so the smallest one shipped:

| variant | size | verdict |
|---|---|---|
| **Q4_K_M, embeddings `Q4_K`** | **1.07 GB** | published here |
| Q4_K_M, embeddings `Q8_0` | 1.19 GB | +0.5pp exact match — noise |
| Q4_K_M, embeddings `F16` | 1.42 GB | no measurable gain |
| Q4_K_M, defaults | 1.92 GB | `llama-quantize` promotes embeddings to F32. No quality gain, 470 MB wasted |

Embeddings are tied to `lm_head` in this architecture, which is why their bit width gets its own column.
imatrix calibration used real streaming-translation prompts, which is what makes 4-bit embeddings safe here.

---

## Run it

```bash
llama-server -m Hy-MT2-1.8B-StreamRevise-Q4_K_M.gguf -c 4096 --port 8080 -ngl 99 --no-webui
```

Drop `-ngl 99` (or set `0`) on CPU-only machines. Then POST to `/completion`:

```json
{
  "prompt": "<｜hy_begin▁of▁sentence｜><｜hy_User｜>{PROMPT}<｜hy_Assistant｜>",
  "n_predict": 256,
  "temperature": 0,
  "cache_prompt": true
}
```

Three things to get right:

1. **`temperature: 0`** — greedy, not the base model's `0.7`. Consecutive updates hand the model nearly the
   same prompt and you want nearly the same output back; sampling introduces changes that have nothing to do
   with the source changing, and those land on screen as flicker. All numbers below are greedy.
2. **`cache_prompt: true`** — updates within one utterance share a long prefix. KV reuse is what makes
   per-update latency negligible.
3. **`{PROMPT}` must use the StreamRevise layout**, not a plain translation instruction.

### A warning you can ignore

llama.cpp prints this on load:

```
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
```

It is harmless here. Hy-MT2 declares a single-value `eos_token_id`, and this build carries an explicit
`tokenizer.ggml.eom_token_id = 120020` that llama.cpp does not fold into its `special_eog_ids` set — hence
the complaint. Generation still terminates correctly: expect `stop_type: eos` and a `predicted_n` far below
your `n_predict`. If instead the model never stops, you are running a GGUF converted without the
end-of-message fix — see below.

---

## Prompt format

Full spec and a copy-paste renderer are in the
[LoRA repo](https://huggingface.co/febilly/Hy-MT2-1.8B-StreamRevise-LoRA#prompt-format). Short version.

First chunk of a new sentence, nothing to revise yet:

```text
Translate the following text into {TARGET_LANGUAGE}. Note that you should only output the translated result without any additional explanation:

{CURRENT_SOURCE}
```

Every update after that:

```text
[Background Information]
Recent source utterances:
{UP_TO_10_PREVIOUS_SOURCE_SENTENCES}

Previous version of the current source:
{PREVIOUS_SOURCE}

Previous translation of the current source:
{PREVIOUS_TRANSLATION}

When the source meaning has not changed, preserve the still-correct prefix of the previous translation whenever possible. When content is added or corrected, accuracy and completeness take priority.

Please translate the following text into {TARGET_LANGUAGE}, taking the provided background information into consideration.

[Source Text]
{CURRENT_SOURCE}
```

Background blocks are each optional, joined by a blank line, always in that order. `{TARGET_LANGUAGE}` is a
full English language name. `Recent source utterances` carries **source text only**, never the translations.

The model keeps no state between requests — the caller owns the revision chain and sends it in full every
time. That means you can retry, reorder, or drop requests freely.

---

## Speed

| | short prompt (33 tok) | long prompt (137 tok) |
|---|---|---|
| GPU (A40, CUDA, `-ngl 99`) | **0.06 s**, 256 tok/s decode | **0.06 s**, 337 tok/s decode |
| CPU (56-thread server) | 0.53 s, 17 tok/s decode | 1.16 s, 17 tok/s decode |

Rough expectations elsewhere: 8-core desktop CPU ~10–20 tok/s; Apple Silicon (Metal) ~30–60+ tok/s; any
discrete GPU with ≥2 GB VRAM (CUDA or Vulkan) ~50–300 tok/s.

For live subtitles: under 100 ms per update on a GPU, 0.5–1.2 s on CPU alone — usable, if not instant.

## Footprint

Weights are ~1.02 GiB once loaded. The KV cache is the only part that scales with your settings: this is a
32-layer model with 4 KV heads at head_dim 128, so **64 KiB per token** at f16.

| `n_ctx` | KV cache | ballpark resident |
|---|---|---|
| 1024 | 64 MiB | ~1.3 GB |
| **2048** | **128 MiB** | **~1.4 GB** |
| 4096 | 256 MiB | ~1.5 GB |
| 8192 | 512 MiB | ~1.8 GB |

Measured: raising `-c` from 2048 to 4096 costs exactly 135 MB. `--parallel` costs nothing extra — `-c` is the
total KV budget, divided among slots, not multiplied by them.

Two things push the number above these estimates, and neither is the model's doing: the CUDA context and
allocator pool (often 300–500 MB, and `nvidia-smi` reports it against your process), and compute buffers
sized by `-b`/`-ub`. If you are counting VRAM on a small card, lower `-c` first — that is the part you
control.

## What 4-bit costs

843 trajectories / 2,438 states, same prompts and greedy decoding on both sides.

| | bf16 | this build |
|---|---|---|
| state exact match | 0.287 | 0.282 |
| mean state similarity | 0.814 | 0.801 |
| final exact match | 0.199 | 0.211 |
| characters erased per append | 2.56 | 2.32 |
| prefix preserved (append transition) | 0.798 | 0.824 |
| prefix preserved (final transition) | 0.917 | 0.925 |
| empty-output rate | 0.000 | 0.000 |

About 1.3pp of similarity and half a point of exact match. The stability rows come out level or slightly
ahead, which is within noise. Full table in the
[LoRA repo](https://huggingface.co/febilly/Hy-MT2-1.8B-StreamRevise-LoRA#numbers).

---

## If you convert Hy-MT2 yourself

llama.cpp's HunYuan converter doesn't write an `eog`/`eom` token when the source model has a **single-valued**
`eos_token_id` — and Hy-MT2 does. Without a fix, generation never stops and you get endless repetition.

**This build already has the fix applied**; its metadata carries `tokenizer.ggml.eom_token_id = 120020`. If
you're converting from scratch, append this to `_fix_special_tokens()` in `conversion/hunyuan.py`:

```python
eos = self.hparams.get("eos_token_id")
if eos is not None:
    ids = eos if isinstance(eos, list) else [eos]
    self.gguf_writer.add_eom_token_id(int(ids[0]))
```

Converted and quantized with llama.cpp `b10442`.

---

## Limitations

- **Stability is a tendency, not a guarantee.** Nothing enforces prefix preservation; worst case a single
  update rewrites the whole line. If your UI can't tolerate that, only display the prefix that has held
  steady for N updates.
- **Prompt format matters a lot** — off-format prompts lose quality and stability.
- **Language coverage is uneven**: zh/en/ja are what it was trained on and measured on. Other directions
  fall back to base-model behaviour, untested.
- **Greedy decoding assumed.**
- Metric definitions are project-internal; don't compare them against numbers from simultaneous-MT papers.

## License

Apache 2.0, same as the base model [`tencent/Hy-MT2-1.8B`](https://huggingface.co/tencent/Hy-MT2-1.8B).