Needs the projector file in GGUF as well...
#1
by igzbar - opened
I think for the model to be complete and be capable of being run via the llava-cli module of llama.cpp, you also need to quantize the projector .safetensors file to GGUF (it's small so Q8_0 would be totally usable).
I tried but the ProjectorModel architecture did not work properly for me. :/
If someone else successfully does it then feel free to share.
I’ve tried using mmproj from another repo but it doesn’t work (with LM Studio), it keeps saying that the model is not capable to read image
I got it running in LMStudio using the vision adapter from koboldcpp for Llama3-8b.
My main model file is the Q5_K_M version.
But performance isn't great, probably because of mismatching quantization accuracies (?)..