Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model:
|
| 4 |
+
- Qwen/Qwen2.5-0.5B-Instruct
|
| 5 |
+
datasets:
|
| 6 |
+
- agentlans/common-crawl-sample
|
| 7 |
+
- bigcode/the-stack-smol-xl
|
| 8 |
+
- rombodawg/Everything_Instruct
|
| 9 |
+
tags:
|
| 10 |
+
- draft
|
| 11 |
+
- speculative-decoding
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
A `0.6B` parameter draft (speculative decoding) model for use with [Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct).
|
| 15 |
+
|
| 16 |
+
See [Kimi-K2-Instruct-DRAFT-0.6B-v3.0](https://huggingface.co/jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0) for the models in `transformers` format, and a detailed explanation of how the model was created.
|
| 17 |
+
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
I've included the `Q4_0` quants for 3 different context lengths:
|
| 21 |
+
|
| 22 |
+
- [Kimi-K2-Instruct-DRAFT-0.6B-32k-Q4_0.gguf](https://huggingface.co/jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF/resolve/main/Kimi-K2-Instruct-DRAFT-0.6B-32k-Q4_0.gguf)
|
| 23 |
+
- [Kimi-K2-Instruct-DRAFT-0.6B-64k-Q4_0.gguf](https://huggingface.co/jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF/resolve/main/Kimi-K2-Instruct-DRAFT-0.6B-64k-Q4_0.gguf)
|
| 24 |
+
- [Kimi-K2-Instruct-DRAFT-0.6B-128k-Q4_0.gguf](https://huggingface.co/jukofyork/Kimi-K2-Instruct-DRAFT-0.6B-v3.0-GGUF/resolve/main/Kimi-K2-Instruct-DRAFT-0.6B-128k-Q4_0.gguf)
|
| 25 |
+
|
| 26 |
+
---
|
| 27 |
+
|
| 28 |
+
## NOTES:
|
| 29 |
+
|
| 30 |
+
- The 14 heads of `Qwen2.5-0.5B` doesn't allow for any of the other 4-bit quants to be made (and experimentation has shown using more or less than 4-bits for speculative decoding is a waste of time anwyay).
|
| 31 |
+
- Due to `llama.cpp` using "static-YaRN" the scaling factor remains constant regardless of input length! Only use the longer context versions when processing long contexts is required...
|
| 32 |
+
- If you want to recreate these, then the `TikToken` / `SentencePiece` tokenizer mismatch requires a small hack to `convert_hf_to_gguf.py` (see main model page for details).
|