| --- |
| license: apache-2.0 |
| license_note: derived from Qwen2.5-0.5B (Apache-2.0); base license retained |
| base_model: Qwen/Qwen2.5-0.5B |
| tags: |
| - speculative-decoding |
| - draft-model |
| - llama.cpp |
| - code-generation |
| language: |
| - en |
| --- |
| |
| # Horizon Draft 0.5B |
|
|
| A tiny **standalone draft model** for speculative decoding with |
| `DeepSeek-R1-Distill-Qwen-7B` as the target, built for consumer CPUs and |
| `llama.cpp -md`. Part of the [Horizon](https://github.com/Vexp-ai/horizon) |
| project: a verification-first layer for local LLMs, by the team behind |
| [vexp](https://vexp.dev). |
|
|
| ## What it is |
|
|
| - **Base:** Qwen2.5-0.5B, full-finetuned for one epoch (~2.5 h on a single |
| 24 GB GPU, ~$3 of compute) on 30k reasoning traces distilled from |
| DeepSeek-R1 (13k math, 13k code, 4k science), formatted with the |
| **target's chat template**. |
| - **Vocabulary-aligned with the target:** trained with the target's |
| tokenizer and the embedding padded to n_vocab 152064, so it passes |
| llama.cpp's strict speculative vocabulary check and drops straight into |
| `-md`. A stock Qwen2.5-0.5B does not pair (different special tokens, |
| 151936 vocab), and through permissive paths it reaches only τ≈1. |
| |
| ## Measured results (mainstream DDR4 desktop, Ryzen 9 3900X, 12 threads) |
| |
| | Config | Code generation | Reasoning segment | RSS | |
| |---|---|---|---| |
| | 7B Q4_K_M autoregressive | 8.1 t/s | 8.1 t/s | 7.7 GB | |
| | + this drafter (Q8_0, γ=8) | **10.7-15.1 t/s (mean 13.3, 1.65×)** | 7.1-9.4 (neutral) | 8.3 GB | |
| | + same-family 1.5B draft | 7.1-10.9 (no gain) | 5.3 (hurts) | 9.4 GB | |
|
|
| Acceptance on three coding tasks: **36-56%** on code, 20-30% on |
| chain-of-thought text. Two practical lessons we ship with the numbers: |
| on CPU the draft must be nearly free (a 1.5B draft with similar acceptance |
| gains nothing), and speculation pays on code, not on reasoning: enable it |
| per segment. |
|
|
| ## Usage (llama.cpp) |
|
|
| ```bash |
| llama-speculative -m DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf \ |
| -md horizon-draft-0.5b-q8_0.gguf \ |
| --spec-draft-n-max 8 -t 12 -n 512 --temp 0 -p "..." |
| ``` |
|
|
| Caveat: some recent `llama-server` builds silently skip speculative |
| decoding (unified KV cache default) while still loading the draft. Verify |
| acceptance stats are non-zero; the `llama-speculative` binary is the |
| reliable path today. |
|
|
| ## Limitations |
|
|
| - Target-specific by design: it predicts DeepSeek-R1-Distill-Qwen-7B's |
| output distribution. Pair it with other targets at your own risk. |
| - Trained for one epoch on 30k traces: acceptance has headroom (more |
| epochs, more diverse code styles, DSpark-style parallel drafting are the |
| declared next steps). |
| - Not an instruct model: do not use it standalone for generation. |
|
|
| ## Reproduce |
|
|
| Training script (`train/train_draft.py`), corpus recipe and the full |
| measurement logs are in the |
| [Horizon repository](https://github.com/Vexp-ai/horizon). |
|
|
