Keep MLX draft comparisons and highlight only column-best speeds
Browse files
README.md
CHANGED
|
@@ -27,29 +27,15 @@ This checkpoint is the **DFlash DPACE branch** adapted to:
|
|
| 27 |
|
| 28 |
## Results
|
| 29 |
|
| 30 |
-
### A100 / SGLang
|
| 31 |
-
|
| 32 |
-
| Draft | Overall tok/s |
|
| 33 |
-
| --- | ---: |
|
| 34 |
-
| Target only | 175.60 |
|
| 35 |
-
| Official DSpark | **354.69** |
|
| 36 |
-
| Draftfit DSpark | **347.94** |
|
| 37 |
-
| Draftfit DFlash CE | 294.77 |
|
| 38 |
-
| **Draftfit DFlash DPACE (Mine)** | **304.85** |
|
| 39 |
-
| DFlash2 | not measured |
|
| 40 |
-
|
| 41 |
### MLX / M5 Pro
|
| 42 |
|
| 43 |
| Draft | Overall tok/s |
|
| 44 |
| --- | ---: |
|
| 45 |
-
| Target only | 50.98 |
|
| 46 |
| Official DSpark | 105.38 |
|
| 47 |
| Draftfit DSpark | **116.15** |
|
| 48 |
-
| DFlash CE | 89.61 |
|
| 49 |
-
|
|
| 50 |
-
| DFlash2 | 80.75 |
|
| 51 |
-
|
| 52 |
-
A100 and MLX use different measurement methods, so their absolute numbers should **not** be compared directly.
|
| 53 |
|
| 54 |
Validation set: **53 held-out conversations**
|
| 55 |
|
|
@@ -62,12 +48,11 @@ Validation set: **53 held-out conversations**
|
|
| 62 |
|
| 63 |
| Draft | PerfectBlend | OpenCode | APIGen-MT | Nemotron Agentic |
|
| 64 |
| --- | ---: | ---: | ---: | ---: |
|
| 65 |
-
| Target only | 50.76 | 51.04 | 51.47 | 51.81 |
|
| 66 |
| Official DSpark | 107.83 | 112.19 | 107.79 | 85.86 |
|
| 67 |
| Draftfit DSpark | **118.29** | **122.28** | **116.04** | **91.99** |
|
| 68 |
-
| DFlash CE | 89.02 | 99.18 | 92.49 | 79.31 |
|
| 69 |
-
|
|
| 70 |
-
| DFlash2 | 78.37 | 92.76 | 87.49 | 70.41 |
|
| 71 |
|
| 72 |
Full prompt-free summary: [`evaluation-v4.json`](evaluation-v4.json)
|
| 73 |
|
|
@@ -75,14 +60,14 @@ Full prompt-free summary: [`evaluation-v4.json`](evaluation-v4.json)
|
|
| 75 |
|
| 76 |
| Draft | Best use |
|
| 77 |
| --- | --- |
|
| 78 |
-
|
|
| 79 |
-
|
|
| 80 |
-
|
|
| 81 |
-
|
|
| 82 |
|
| 83 |
For this MiniCPM5-SFT experiment, **DSpark is the strongest practical baseline**.
|
| 84 |
|
| 85 |
-
Draftfit DSpark reached **
|
| 86 |
|
| 87 |
These measurements do **not** establish a general speed advantage.
|
| 88 |
|
|
@@ -93,12 +78,7 @@ These measurements do **not** establish a general speed advantage.
|
|
| 93 |
| `main` | DFlash DPACE, primary release |
|
| 94 |
| `v5` | Additional 1,000-step continual-training stage on a fresh split |
|
| 95 |
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
- Draftfit DSpark [`v3-best`](https://huggingface.co/jayyun98/MiniCPM5-2B-SFT-Draftfit-DSpark/tree/v3-best): **360.26 tok/s**
|
| 99 |
-
- Matching official DSpark: **361.36 tok/s**
|
| 100 |
-
|
| 101 |
-
Different revisions used different held-out sets, so these numbers should not be used to rank revisions directly.
|
| 102 |
|
| 103 |
## Validation notes
|
| 104 |
|
|
@@ -116,7 +96,7 @@ Training mixture per stage:
|
|
| 116 |
Important limitations:
|
| 117 |
|
| 118 |
- DSpark/DFlash serving correctness still requires exact token/state and acceptance-length validation.
|
| 119 |
-
- DFlash2 was tested on MLX but not
|
| 120 |
- A successfully loaded draft checkpoint does **not** prove correct speculative decoding.
|
| 121 |
- Reported speed is specific to the tested target, hardware, verifier settings and prompt set.
|
| 122 |
|
|
|
|
| 27 |
|
| 28 |
## Results
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
### MLX / M5 Pro
|
| 31 |
|
| 32 |
| Draft | Overall tok/s |
|
| 33 |
| --- | ---: |
|
|
|
|
| 34 |
| Official DSpark | 105.38 |
|
| 35 |
| Draftfit DSpark | **116.15** |
|
| 36 |
+
| Draftfit DFlash CE | 89.61 |
|
| 37 |
+
| Draftfit DFlash DPACE | 89.97 |
|
| 38 |
+
| Draftfit DFlash2 | 80.75 |
|
|
|
|
|
|
|
| 39 |
|
| 40 |
Validation set: **53 held-out conversations**
|
| 41 |
|
|
|
|
| 48 |
|
| 49 |
| Draft | PerfectBlend | OpenCode | APIGen-MT | Nemotron Agentic |
|
| 50 |
| --- | ---: | ---: | ---: | ---: |
|
|
|
|
| 51 |
| Official DSpark | 107.83 | 112.19 | 107.79 | 85.86 |
|
| 52 |
| Draftfit DSpark | **118.29** | **122.28** | **116.04** | **91.99** |
|
| 53 |
+
| Draftfit DFlash CE | 89.02 | 99.18 | 92.49 | 79.31 |
|
| 54 |
+
| Draftfit DFlash DPACE | 89.84 | 103.20 | 91.62 | 76.97 |
|
| 55 |
+
| Draftfit DFlash2 | 78.37 | 92.76 | 87.49 | 70.41 |
|
| 56 |
|
| 57 |
Full prompt-free summary: [`evaluation-v4.json`](evaluation-v4.json)
|
| 58 |
|
|
|
|
| 60 |
|
| 61 |
| Draft | Best use |
|
| 62 |
| --- | --- |
|
| 63 |
+
| DSpark | Best first choice when adapting an existing DSpark draft to a new target |
|
| 64 |
+
| DFlash CE | Simple block-parallel DFlash baseline |
|
| 65 |
+
| DFlash DPACE | Test position-weighted training objectives against CE |
|
| 66 |
+
| DFlash2 | Research richer candidate ranking / grouped-convolution drafts |
|
| 67 |
|
| 68 |
For this MiniCPM5-SFT experiment, **DSpark is the strongest practical baseline**.
|
| 69 |
|
| 70 |
+
Draftfit DSpark reached **116.15 tok/s vs 105.38 tok/s** for the official DSpark in the tested MLX setup.
|
| 71 |
|
| 72 |
These measurements do **not** establish a general speed advantage.
|
| 73 |
|
|
|
|
| 78 |
| `main` | DFlash DPACE, primary release |
|
| 79 |
| `v5` | Additional 1,000-step continual-training stage on a fresh split |
|
| 80 |
|
| 81 |
+
Different revisions used different held-out sets, so their measurements should not be used to rank revisions directly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
|
| 83 |
## Validation notes
|
| 84 |
|
|
|
|
| 96 |
Important limitations:
|
| 97 |
|
| 98 |
- DSpark/DFlash serving correctness still requires exact token/state and acceptance-length validation.
|
| 99 |
+
- DFlash2 was tested on MLX but not with SGLang 0.5.16.
|
| 100 |
- A successfully loaded draft checkpoint does **not** prove correct speculative decoding.
|
| 101 |
- Reported speed is specific to the tested target, hardware, verifier settings and prompt set.
|
| 102 |
|