Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:# Run inference directly in the terminal:
llama cli -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:# Run inference directly in the terminal:
./llama-cli -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:# Run inference directly in the terminal:
./build/bin/llama-cli -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:Use Docker
docker model run hf.co/Raydev/talkie-1930-13b-yarn32k-tf-GGUF:This repo contains the GGUF and quantized version of talkie-1930-13b-yarn-32k-tf It is based off of the base version of Talkie-1930, and therefore is not meant for actual chat conversations.
It is recommended to use llama.cpp to load this model.
Included files:
| Quant File | Size |
|---|---|
| talkie-1930-13b-yarn32k-Q2_K.gguf | 5.1G |
| talkie-1930-13b-yarn32k-Q3_K_S.gguf | 5.8G |
| talkie-1930-13b-yarn32k-Q3_K_M.gguf | 6.4G |
| talkie-1930-13b-yarn32k-Q4_K_S.gguf | 7.4G |
| talkie-1930-13b-yarn32k-Q4_K_M.gguf | 8.0G |
| talkie-1930-13b-yarn32k-Q5_K_S-ffn5_0.gguf | 9.2G |
| talkie-1930-13b-yarn32k-Q5_K_M-ffn5_0.gguf | 9.2G |
| talkie-1930-13b-yarn32k-Q6_K.gguf | 11G |
| talkie-1930-13b-yarn32k-Q8_0.gguf | 14G |
| talkie-1930-13b-yarn32k-f16.gguf | 25G |
Note:
ffn5_0 refer to those files having their ffn_down forced to be q5_0 rather than letting the usually determined 5_1, this has to do with the fact that Talkie's ffn_down is not divisble by 256 meaning the base llama-quantize chooses 5_1, although when I first quantized with 5_1, the models output were completely broken; repeating the same character over and over rather than generating text, irregardless of input. I am pretty sure the fact the ffn_down cannot be divided by 256 is just a architectural design decision made by the upstream talkie-1930-base model.
i have found this issue with the talkie-1930-13b-it-hf-GGUF models as well? Not necessarily the q5_1 issue, as I haven't verified it yet. But the exact mode of failure with a repeating '>' token or other control token with their Q5_K_M & K_S models.
- Downloads last month
- 142
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Model tree for Raydev/talkie-1930-13b-yarn32k-tf-GGUF
Base model
talkie-lm/talkie-1930-13b-base
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF:# Run inference directly in the terminal: llama cli -hf Raydev/talkie-1930-13b-yarn32k-tf-GGUF: