Eric111/vicuna-13b-v1.5-16k-gguf

This is a half precision (float16) gguf file for vicuna-13b-v1.5-16k.

Conversion code:

import os import subprocess

from huggingface_hub import snapshot_download

model_id="lmsys/vicuna-13b-v1.5-16k" snapshot_download(repo_id=model_id, local_dir="vicuna-hf", local_dir_use_symlinks=False, revision="main")

subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git"])

subprocess.run(["pip", "install", "-r", "llama.cpp/requirements.txt"])

subprocess.run(["python", "llama.cpp/convert_hf_to_gguf.py", "vicuna-hf", "--outfile", "vicuna-13b-v1.5-16k.gguf", "--outtype", "f16"])

(modified from https://www.substratus.ai/blog/converting-hf-model-gguf-model)