How does this compare to HY-MT1.5-7B?
Hi there, I'd like to thank you for the detailed instructions on how to get this properly working with llama.cpp, I couldn't find this information anywhere else. Right now I'm using Tencent's HY-MT1.5-7B for my translation workflows and it does a pretty decent job for most languages I use it for, have you done any tests comparing the two and would this be a substantial improvement?
Thanks! I have not tested the HY-MT model yet, but it looks like it runs on llama.cpp so I can make a quant and see how it does. I have BLEU results on opus and flores200 on my HF benchlm space for various MT models. This model is benching better than plamo but worse than madlad400. I finalized my recommended minimalist prompt templates today for this model tested across the 4B, 12B, and 27B models to work. There is no information anywhere on this just have to experiment with it, but the model rides on top of gemma3 so at least has some instruct following ability.
@rsbdev I ran some evals on a HY-MT1.5-7B quant (very close to lossless Q6_K_H mixed precision quant) results are here: https://huggingface.co/spaces/steampunque/benchlm . It can't translate en->de and translategemma12b benches better on most of the tested languages. How it performs out in the wild on real translation tasks against other models would have to be evaluated manually, the evals may or may not be realistic on picking up good translations due to limitations of the comparison metric (bleu). Most likely need a human fluent in source and target languages to accurately assess translation quality vs others.
I tested them both, and tencent/HY-MT1.5-7B-GGUF HY-MT1.5-7B-Q8_0.gguf is better than the TranslateGemma models I tried.
I tested them both, and
tencent/HY-MT1.5-7B-GGUF HY-MT1.5-7B-Q8_0.ggufis better than the TranslateGemma models I tried.
It looks like there is a HY-MT2 series released now across a bunch of model sizes. I will make MP quants and benches for them shortly. If
they fixed the en->de and the benches look sane I will upload.
@Taylor-eOS I created MP quants for HY-MT1.5-7B https://huggingface.co/steampunque/HY-MT1.5-7B-MP-GGUF and HY-MT2-7B https://huggingface.co/steampunque/HY-MT2-7B-MP-GGUF and ran bleu benches across flores200 and opus. translategemma benches higher than HY-MT1.5-7B and lower than HY-MT2-7B, but madlad400 still benches better than HY-MT2-7B. Note I corrected a prompting issue with HY-MT as described on the model pages so all translation results should be valid. I'll also check out the HY 30B moe model later (not yet supported in llama.cpp)