MTP Assistant?
#2
by meepen - opened
Hey, with the https://github.com/ggml-org/llama.cpp/pull/23398 PR MTP is possible against https://huggingface.co/google/gemma-4-31B-it-assistant - is there a chance you can use the same training data on that model and put out an MTP assistant with this model?
Thank you!
Hi! Sorry for the belated answer.
I think it would be possible, but more in the sense of distillation of the main model to a lot of questions and then training the assistant model on it.
Otherwise, I got feedback that my latest merges (Versipellis and Dark-Gemistry) have MTP accept rates of 40-80%. While I can't verify first-hand, you could try if it works better with those models.
Let me know if it works, I might re-merge then with some more filters.