File size: 5,222 Bytes
c8ee49b 1d7426f c8ee49b 0609ed2 c8ee49b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | ---
license: apache-2.0
library_name: transformers
base_model: meta-models/Muse-Glimmer-30B
datasets:
- DaoCloud/Muse-Glimmer-OPB-100K
tags:
- speculative-decoding
- dspark
- dflash
- speculators
- vllm
- muse-glimmer
---
# Muse Glimmer 30B DSpark
A DSpark drafter for [Muse Glimmer 30B](https://huggingface.co/meta-models/Muse-Glimmer-30B), warm-started from the official five-layer [Muse Glimmer DFlash assistant](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant).
This checkpoint adds a rank-256 Markov head and a confidence head to the warm-started DFlash backbone — about 103M new parameters, growing the checkpoint from 4.76 GiB to 4.95 GiB (+4%) — then trains on 99,984 Muse-Glimmer-generated, on-policy OpenPerfectBlend conversations. Its main benefit is much stronger acceptance deeper into a 15-token proposal.
## Results
The following results use the same BF16 Muse Glimmer 30B verifier, greedy decoding, and 15 speculative proposals per draft call. Higher accepted length is better.
### Speculators evaluation subsets
| Subset | Official DFlash | DSpark | Change |
|---|---:|---:|---:|
| HumanEval | 3.887 | **4.641** | +19% |
| Math reasoning | 3.867 | **5.237** | +35% |
| QA | 3.192 | **3.768** | +18% |
| Question | 2.963 | **3.441** | +16% |
| RAG | 4.048 | **5.182** | +28% |
| Summarization | 3.551 | **4.568** | +29% |
| Tool call | 3.087 | **3.805** | +23% |
| Translation | 3.627 | **4.948** | +36% |
| Writing | 2.974 | **3.455** | +16% |
### Open-source benchmarks
| Subset | Official DFlash | DSpark | Change |
|---|---:|---:|---:|
| GSM8K | 3.734 | **4.914** | +32% |
| MATH500 | 4.139 | **4.869** | +18% |
| MBPP | 3.647 | **4.390** | +20% |
| AIME 2026 | 3.870 | **4.271** | +10% |
Accepted length is calculated from the raw server counters:
```text
accepted_length = 1 + accepted_tokens / draft_calls
```
### Per-position acceptance
Per-position acceptance curves for the same runs as the tables above. DSpark shows slightly lower position-0 acceptance but substantially stronger acceptance deeper into the proposal, with the largest gains toward the tail.
## Warm start
The inherited five-layer DFlash body is already trained, while the Markov and confidence heads are newly initialized. Applying the same `6e-4` peak learning rate to every parameter caused the warm-started body to lose some early-token accuracy during the high-LR phase. The released run uses:
```text
DFlash body: 3e-4
Markov/confidence heads: 6e-4
```
## Training recipe
| Setting | Value |
|---|---|
| Verifier | Muse Glimmer 30B BF16 |
| Initialization | Official Muse Glimmer five-layer DFlash assistant |
| Draft | 5×Qwen3 GQA, causal SWA 2048, BF16 |
| Hidden-state IDs | `[2, 14, 26, 38, 50]` |
| DSpark heads | Markov rank 256 + confidence head |
| Proposals | `block_size=15`, `sample_from_anchor=true` |
| Data | 99,984 conversations / 148,900 train-turn rows |
| Reasoning mix | low 64,997; medium 10,000; high 19,991; xhigh 4,996 |
| Packed length | 16,384 tokens/rank |
| Anchors | up to 512 per packed sequence |
| Position decay | gamma 4 |
| Training | 3 epochs, TP4 producer + FSDP4 trainer, 8×H200 |
| Optimizer | AdamW, weight decay 0.01, gradient clip 1 |
| Peak LR | body `3e-4`; Markov/confidence heads `6e-4` |
| Schedule | cosine to zero, 4% warmup |
| Loss | 0.1 CE + 0.9 TV; confidence BCE weight 1.0 |
## Training data
[DaoCloud/Muse-Glimmer-OPB-100K](https://huggingface.co/datasets/DaoCloud/Muse-Glimmer-OPB-100K)
## Example: Run with vLLM
This is a drafter checkpoint, not a standalone language model. Pair it with Muse Glimmer 30B or a compatible quantized version:
```bash
vllm serve meta-models/Muse-Glimmer-30B \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser muse_glimmer \
--reasoning-parser muse_glimmer \
--speculative-config \
'{"method":"dspark","model":"DaoCloud/Muse-Glimmer-30B-DSpark","num_speculative_tokens":15}'
```
## Checkpoint
The BF16 checkpoint contains 62 tensors and is 5,318,887,946 bytes (4.95 GiB):
```text
SHA-256 be0d03dd05d99049eac573a2654d0e65f86bc4beeedc1bb213798a506189240f
```
## Notes
- This technical preview was trained and validated with patched development versions of [vllm-project/speculators](https://github.com/vllm-project/speculators) and [vllm-project/vllm](https://github.com/vllm-project/vllm). Stock releases may not yet include the required Muse Glimmer / DFlash / DSpark support.
- The model was trained with a maximum proposal length of 15 tokens, but `num_speculative_tokens` is a serving-time tuning parameter. The optimal value should be determined based on workload, hardware, and latency.
- The confidence head is included in the checkpoint, but the tested vLLM runtime does not yet use it for confidence-scheduled verification.
- Multilingual, agentic, and multimodal speculative decoding has not yet been benchmarked and is left for future work.
Released under Apache-2.0, matching Muse Glimmer and its official DFlash assistant.
|