---
license: apache-2.0
library_name: transformers
base_model: meta-models/Muse-Glimmer-30B
datasets:
  - DaoCloud/Muse-Glimmer-OPB-100K
tags:
  - speculative-decoding
  - dspark
  - dflash
  - speculators
  - vllm
  - muse-glimmer
---

# Muse Glimmer 30B DSpark

A DSpark drafter for [Muse Glimmer 30B](https://huggingface.co/meta-models/Muse-Glimmer-30B), warm-started from the official five-layer [Muse Glimmer DFlash assistant](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant).

This checkpoint adds a rank-256 Markov head and a confidence head to the warm-started DFlash backbone — about 103M new parameters, growing the checkpoint from 4.76 GiB to 4.95 GiB (+4%) — then trains on 99,984 Muse-Glimmer-generated, on-policy OpenPerfectBlend conversations. Its main benefit is much stronger acceptance deeper into a 15-token proposal.

## Results

The following results use the same BF16 Muse Glimmer 30B verifier, greedy decoding, and 15 speculative proposals per draft call. Higher accepted length is better.

### Speculators evaluation subsets

| Subset | Official DFlash | DSpark | Change |
|---|---:|---:|---:|
| HumanEval | 3.887 | **4.641** | +19% |
| Math reasoning | 3.867 | **5.237** | +35% |
| QA | 3.192 | **3.768** | +18% |
| Question | 2.963 | **3.441** | +16% |
| RAG | 4.048 | **5.182** | +28% |
| Summarization | 3.551 | **4.568** | +29% |
| Tool call | 3.087 | **3.805** | +23% |
| Translation | 3.627 | **4.948** | +36% |
| Writing | 2.974 | **3.455** | +16% |

### Open-source benchmarks

| Subset | Official DFlash | DSpark | Change |
|---|---:|---:|---:|
| GSM8K | 3.734 | **4.914** | +32% |
| MATH500 | 4.139 | **4.869** | +18% |
| MBPP | 3.647 | **4.390** | +20% |
| AIME 2026 | 3.870 | **4.271** | +10% |

Accepted length is calculated from the raw server counters:

```text
accepted_length = 1 + accepted_tokens / draft_calls
```

### Per-position acceptance 

![Per-position acceptance rate on all 13 subsets: Official DFlash vs DSpark](per_position_acceptance.png)

Per-position acceptance curves for the same runs as the tables above. DSpark shows slightly lower position-0 acceptance but substantially stronger acceptance deeper into the proposal, with the largest gains toward the tail.

## Warm start

The inherited five-layer DFlash body is already trained, while the Markov and confidence heads are newly initialized. Applying the same `6e-4` peak learning rate to every parameter caused the warm-started body to lose some early-token accuracy during the high-LR phase. The released run uses:

```text
DFlash body:             3e-4
Markov/confidence heads: 6e-4
```

![Training curves comparing uniform and differential learning rates](image.png)

## Training recipe

| Setting | Value |
|---|---|
| Verifier | Muse Glimmer 30B BF16 |
| Initialization | Official Muse Glimmer five-layer DFlash assistant |
| Draft | 5×Qwen3 GQA, causal SWA 2048, BF16 |
| Hidden-state IDs | `[2, 14, 26, 38, 50]` |
| DSpark heads | Markov rank 256 + confidence head |
| Proposals | `block_size=15`, `sample_from_anchor=true` |
| Data | 99,984 conversations / 148,900 train-turn rows |
| Reasoning mix | low 64,997; medium 10,000; high 19,991; xhigh 4,996 |
| Packed length | 16,384 tokens/rank |
| Anchors | up to 512 per packed sequence |
| Position decay | gamma 4 |
| Training | 3 epochs, TP4 producer + FSDP4 trainer, 8×H200 |
| Optimizer | AdamW, weight decay 0.01, gradient clip 1 |
| Peak LR | body `3e-4`; Markov/confidence heads `6e-4` |
| Schedule | cosine to zero, 4% warmup |
| Loss | 0.1 CE + 0.9 TV; confidence BCE weight 1.0 |

## Training data

[DaoCloud/Muse-Glimmer-OPB-100K](https://huggingface.co/datasets/DaoCloud/Muse-Glimmer-OPB-100K)

## Example: Run with vLLM

This is a drafter checkpoint, not a standalone language model. Pair it with Muse Glimmer 30B or a compatible quantized version:

```bash
vllm serve meta-models/Muse-Glimmer-30B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --speculative-config \
  '{"method":"dspark","model":"DaoCloud/Muse-Glimmer-30B-DSpark","num_speculative_tokens":15}'
```

## Checkpoint

The BF16 checkpoint contains 62 tensors and is 5,318,887,946 bytes (4.95 GiB):

```text
SHA-256  be0d03dd05d99049eac573a2654d0e65f86bc4beeedc1bb213798a506189240f
```

## Notes

- This technical preview was trained and validated with patched development versions of [vllm-project/speculators](https://github.com/vllm-project/speculators) and [vllm-project/vllm](https://github.com/vllm-project/vllm). Stock releases may not yet include the required Muse Glimmer / DFlash / DSpark support.
- The model was trained with a maximum proposal length of 15 tokens, but `num_speculative_tokens` is a serving-time tuning parameter. The optimal value should be determined based on workload, hardware, and latency.
- The confidence head is included in the checkpoint, but the tested vLLM runtime does not yet use it for confidence-scheduled verification.
- Multilingual, agentic, and multimodal speculative decoding has not yet been benchmarked and is left for future work.

Released under Apache-2.0, matching Muse Glimmer and its official DFlash assistant.