How to use this with llama-cpp or ollama to create image embeddings?
First of all, thanks for the quant! I'm trying to use this to create image embedding with llama-cpp (python binding) or ollama but don't know how.
That's likely because the extra files for vision tasks are not provided by us, we only provide the llm portion. I have it somewhere on my todo list to also provide other files automatically, but this is very model specific, and they usually do not need quantizing, only extraction with the model-specific code. Normally, they are easy to come by, but in this case... not.
@nicoboss do you know how to extract these files? I could potentially do this as part of quantization or conversion.
Hello guys,
Thanks again for the quant! Has someone found a code sample on how to use this model using llama.cpp with Python bindings (or not)?
things have changed since then, it might work. I willo re-queue the job and see if we get an mmproj file.
anyway, q8 and f16 mmproj files are now both available, as well as imatrix quants soon. sorr,y no code samples :=)