Initial release: Q4_K_M imatrix build of StreamRevise

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Hy-MT2-1.8B-StreamRevise-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Hy-MT2-1.8B-StreamRevise.imatrix.gguf filter=lfs diff=lfs merge=lfs -text

Hy-MT2-1.8B-StreamRevise-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaf0fecb53a50f533377f975ad507294fd504eb94619f2f19119a5e81fdb06b0
+size 1069288736

Hy-MT2-1.8B-StreamRevise.imatrix.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8829921f1bf1789280090368132fe0a348da8c2a6d67c012da222a5283eb17a9
+size 2393696

README.md

@@ -0,0 +1,234 @@
+---
+license: apache-2.0
+base_model:
+  - tencent/Hy-MT2-1.8B
+  - febilly/Hy-MT2-1.8B-StreamRevise-LoRA
+base_model_relation: quantized
+pipeline_tag: translation
+tags:
+  - translation
+  - streaming-translation
+  - simultaneous-translation
+  - subtitles
+  - speech-translation
+  - gguf
+  - llama.cpp
+  - quantized
+language:
+  - zh
+  - en
+  - ja
+---
+
+# Hy-MT2-1.8B-StreamRevise — GGUF
+
+4-bit llama.cpp build of **StreamRevise**, a translation model for **live subtitles**. Every time the ASR
+hypothesis updates, you pass the model its own previous translation of the sentence in progress; it decides
+whether to keep that text and extend it, or rewrite part of it because the meaning changed. Subtitles grow
+smoothly instead of flickering.
+
+**1.07 GB on disk, and about 1.4 GB resident at `n_ctx 2048`.** Small enough to ship with a desktop app.
+See [Footprint](#footprint) for how that scales.
+
+Adapter, full prompt spec, training details:
+[**Hy-MT2-1.8B-StreamRevise-LoRA**](https://huggingface.co/febilly/Hy-MT2-1.8B-StreamRevise-LoRA) ·
+Base model: [`tencent/Hy-MT2-1.8B`](https://huggingface.co/tencent/Hy-MT2-1.8B)
+
+> **中文简介**：实时字幕翻译模型的 4-bit 本地部署版。语音边说边识别，源文一直在变——每次更新时把模型上一版
+> 译文一起喂回去，它自己决定保留已显示的部分继续接，还是因为意思变了而改写，让字幕平滑增长不闪烁。
+> 1.07 GB，纯 CPU 可跑（约 2.1 GB 内存），有显卡则单次请求 <0.1 秒。
+
+---
+
+## Files
+
+| file | size | what it is |
+|---|---|---|
+| `Hy-MT2-1.8B-StreamRevise-Q4_K_M.gguf` | 1.07 GB | **the model.** Q4_K_M with imatrix calibration, token embeddings at `Q4_K` |
+| `Hy-MT2-1.8B-StreamRevise.imatrix.gguf` | 2.3 MB | importance matrix, only needed if you want to re-quantize at another bit width |
+
+Only one quantization is published. The alternatives were measured and the differences sit inside the noise
+floor, so the smallest one shipped:
+
+| variant | size | verdict |
+|---|---|---|
+| **Q4_K_M, embeddings `Q4_K`** | **1.07 GB** | published here |
+| Q4_K_M, embeddings `Q8_0` | 1.19 GB | +0.5pp exact match — noise |
+| Q4_K_M, embeddings `F16` | 1.42 GB | no measurable gain |
+| Q4_K_M, defaults | 1.92 GB | `llama-quantize` promotes embeddings to F32. No quality gain, 470 MB wasted |
+
+Embeddings are tied to `lm_head` in this architecture, which is why their bit width gets its own column.
+imatrix calibration used real streaming-translation prompts, which is what makes 4-bit embeddings safe here.
+
+---
+
+## Run it
+
+```bash
+llama-server -m Hy-MT2-1.8B-StreamRevise-Q4_K_M.gguf -c 4096 --port 8080 -ngl 99 --no-webui
+```
+
+Drop `-ngl 99` (or set `0`) on CPU-only machines. Then POST to `/completion`:
+
+```json
+{
+  "prompt": "<｜hy_begin▁of▁sentence｜><｜hy_User｜>{PROMPT}<｜hy_Assistant｜>",
+  "n_predict": 256,
+  "temperature": 0,
+  "cache_prompt": true
+}
+```
+
+Three things to get right:
+
+1. **`temperature: 0`** — greedy, not the base model's `0.7`. Consecutive updates hand the model nearly the
+   same prompt and you want nearly the same output back; sampling introduces changes that have nothing to do
+   with the source changing, and those land on screen as flicker. All numbers below are greedy.
+2. **`cache_prompt: true`** — updates within one utterance share a long prefix. KV reuse is what makes
+   per-update latency negligible.
+3. **`{PROMPT}` must use the StreamRevise layout**, not a plain translation instruction.
+
+### A warning you can ignore
+
+llama.cpp prints this on load:
+
+```
+load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
+load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
+```
+
+It is harmless here. Hy-MT2 declares a single-value `eos_token_id`, and this build carries an explicit
+`tokenizer.ggml.eom_token_id = 120020` that llama.cpp does not fold into its `special_eog_ids` set — hence
+the complaint. Generation still terminates correctly: expect `stop_type: eos` and a `predicted_n` far below
+your `n_predict`. If instead the model never stops, you are running a GGUF converted without the
+end-of-message fix — see below.
+
+---
+
+## Prompt format
+
+Full spec and a copy-paste renderer are in the
+[LoRA repo](https://huggingface.co/febilly/Hy-MT2-1.8B-StreamRevise-LoRA#prompt-format). Short version.
+
+First chunk of a new sentence, nothing to revise yet:
+
+```text
+Translate the following text into {TARGET_LANGUAGE}. Note that you should only output the translated result without any additional explanation:
+
+{CURRENT_SOURCE}
+```
+
+Every update after that:
+
+```text
+[Background Information]
+Recent source utterances:
+{UP_TO_10_PREVIOUS_SOURCE_SENTENCES}
+
+Previous version of the current source:
+{PREVIOUS_SOURCE}
+
+Previous translation of the current source:
+{PREVIOUS_TRANSLATION}
+
+When the source meaning has not changed, preserve the still-correct prefix of the previous translation whenever possible. When content is added or corrected, accuracy and completeness take priority.
+
+Please translate the following text into {TARGET_LANGUAGE}, taking the provided background information into consideration.
+
+[Source Text]
+{CURRENT_SOURCE}
+```
+
+Background blocks are each optional, joined by a blank line, always in that order. `{TARGET_LANGUAGE}` is a
+full English language name. `Recent source utterances` carries **source text only**, never the translations.
+
+The model keeps no state between requests — the caller owns the revision chain and sends it in full every
+time. That means you can retry, reorder, or drop requests freely.
+
+---
+
+## Speed
+
+| | short prompt (33 tok) | long prompt (137 tok) |
+|---|---|---|
+| GPU (A40, CUDA, `-ngl 99`) | **0.06 s**, 256 tok/s decode | **0.06 s**, 337 tok/s decode |
+| CPU (56-thread server) | 0.53 s, 17 tok/s decode | 1.16 s, 17 tok/s decode |
+
+Rough expectations elsewhere: 8-core desktop CPU ~10–20 tok/s; Apple Silicon (Metal) ~30–60+ tok/s; any
+discrete GPU with ≥2 GB VRAM (CUDA or Vulkan) ~50–300 tok/s.
+
+For live subtitles: under 100 ms per update on a GPU, 0.5–1.2 s on CPU alone — usable, if not instant.
+
+## Footprint
+
+Weights are ~1.02 GiB once loaded. The KV cache is the only part that scales with your settings: this is a
+32-layer model with 4 KV heads at head_dim 128, so **64 KiB per token** at f16.
+
+| `n_ctx` | KV cache | ballpark resident |
+|---|---|---|
+| 1024 | 64 MiB | ~1.3 GB |
+| **2048** | **128 MiB** | **~1.4 GB** |
+| 4096 | 256 MiB | ~1.5 GB |
+| 8192 | 512 MiB | ~1.8 GB |
+
+Measured: raising `-c` from 2048 to 4096 costs exactly 135 MB. `--parallel` costs nothing extra — `-c` is the
+total KV budget, divided among slots, not multiplied by them.
+
+Two things push the number above these estimates, and neither is the model's doing: the CUDA context and
+allocator pool (often 300–500 MB, and `nvidia-smi` reports it against your process), and compute buffers
+sized by `-b`/`-ub`. If you are counting VRAM on a small card, lower `-c` first — that is the part you
+control.
+
+## What 4-bit costs
+
+843 trajectories / 2,438 states, same prompts and greedy decoding on both sides.
+
+| | bf16 | this build |
+|---|---|---|
+| state exact match | 0.287 | 0.282 |
+| mean state similarity | 0.814 | 0.801 |
+| final exact match | 0.199 | 0.211 |
+| characters erased per append | 2.56 | 2.32 |
+| prefix preserved (append transition) | 0.798 | 0.824 |
+| prefix preserved (final transition) | 0.917 | 0.925 |
+| empty-output rate | 0.000 | 0.000 |
+
+About 1.3pp of similarity and half a point of exact match. The stability rows come out level or slightly
+ahead, which is within noise. Full table in the
+[LoRA repo](https://huggingface.co/febilly/Hy-MT2-1.8B-StreamRevise-LoRA#numbers).
+
+---
+
+## If you convert Hy-MT2 yourself
+
+llama.cpp's HunYuan converter doesn't write an `eog`/`eom` token when the source model has a **single-valued**
+`eos_token_id` — and Hy-MT2 does. Without a fix, generation never stops and you get endless repetition.
+
+**This build already has the fix applied**; its metadata carries `tokenizer.ggml.eom_token_id = 120020`. If
+you're converting from scratch, append this to `_fix_special_tokens()` in `conversion/hunyuan.py`:
+
+```python
+eos = self.hparams.get("eos_token_id")
+if eos is not None:
+    ids = eos if isinstance(eos, list) else [eos]
+    self.gguf_writer.add_eom_token_id(int(ids[0]))
+```
+
+Converted and quantized with llama.cpp `b10442`.
+
+---
+
+## Limitations
+
+- **Stability is a tendency, not a guarantee.** Nothing enforces prefix preservation; worst case a single
+  update rewrites the whole line. If your UI can't tolerate that, only display the prefix that has held
+  steady for N updates.
+- **Prompt format matters a lot** — off-format prompts lose quality and stability.
+- **Language coverage is uneven**: zh/en/ja are what it was trained on and measured on. Other directions
+  fall back to base-model behaviour, untested.
+- **Greedy decoding assumed.**
+- Metric definitions are project-internal; don't compare them against numbers from simultaneous-MT papers.
+
+## License
+
+Apache 2.0, same as the base model [`tencent/Hy-MT2-1.8B`](https://huggingface.co/tencent/Hy-MT2-1.8B).