Draft Models
Collection
Tiny "draft" models for speculative decoding. • 14 items • Updated • 7
How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0# Run inference directly in the terminal:
llama cli -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0# Run inference directly in the terminal:
./llama-cli -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0# Run inference directly in the terminal:
./build/bin/llama-cli -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0Use Docker
docker model run hf.co/jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0Quick Links
A 0.6B parameter draft (speculative decoding) model for use with Kimi-K2-Instruct.
See Kimi-K2-Instruct-DRAFT-0.6B-v3.0 for the models in transformers format, and a detailed explanation of how the model was created.
I've included the Q4_0 quants for 3 different context lengths:
- Kimi-K2-Instruct-DRAFT-0.6B-32k-Q4_0.gguf
- Kimi-K2-Instruct-DRAFT-0.6B-64k-Q4_0.gguf
- Kimi-K2-Instruct-DRAFT-0.6B-128k-Q4_0.gguf
NOTES:
- The 14 heads of
Qwen2.5-0.5Bdoesn't allow for any of the other 4-bit quants to be made (and experimentation has shown using more or less than 4-bits for speculative decoding is a waste of time anwyay). - Due to
llama.cppusing "static-YaRN" the scaling factor remains constant regardless of input length! Only use the longer context versions when processing long contexts is required... - If you want to recreate these, then the
TikToken/SentencePiecetokenizer mismatch requires a small hack toconvert_hf_to_gguf.py(see main model page for details).
- Downloads last month
- 115
Hardware compatibility
Log In to add your hardware
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0# Run inference directly in the terminal: llama cli -hf jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF:Q4_0