jayyun98 commited on
Commit
4eea54b
·
verified ·
1 Parent(s): b175f7a

Keep MLX draft comparisons and highlight only column-best speeds

Browse files
Files changed (1) hide show
  1. README.md +13 -33
README.md CHANGED
@@ -27,29 +27,15 @@ This checkpoint is the **DFlash DPACE branch** adapted to:
27
 
28
  ## Results
29
 
30
- ### A100 / SGLang
31
-
32
- | Draft | Overall tok/s |
33
- | --- | ---: |
34
- | Target only | 175.60 |
35
- | Official DSpark | **354.69** |
36
- | Draftfit DSpark | **347.94** |
37
- | Draftfit DFlash CE | 294.77 |
38
- | **Draftfit DFlash DPACE (Mine)** | **304.85** |
39
- | DFlash2 | not measured |
40
-
41
  ### MLX / M5 Pro
42
 
43
  | Draft | Overall tok/s |
44
  | --- | ---: |
45
- | Target only | 50.98 |
46
  | Official DSpark | 105.38 |
47
  | Draftfit DSpark | **116.15** |
48
- | DFlash CE | 89.61 |
49
- | **Draftfit DFlash DPACE (Mine)** | **89.97** |
50
- | DFlash2 | 80.75 |
51
-
52
- A100 and MLX use different measurement methods, so their absolute numbers should **not** be compared directly.
53
 
54
  Validation set: **53 held-out conversations**
55
 
@@ -62,12 +48,11 @@ Validation set: **53 held-out conversations**
62
 
63
  | Draft | PerfectBlend | OpenCode | APIGen-MT | Nemotron Agentic |
64
  | --- | ---: | ---: | ---: | ---: |
65
- | Target only | 50.76 | 51.04 | 51.47 | 51.81 |
66
  | Official DSpark | 107.83 | 112.19 | 107.79 | 85.86 |
67
  | Draftfit DSpark | **118.29** | **122.28** | **116.04** | **91.99** |
68
- | DFlash CE | 89.02 | 99.18 | 92.49 | 79.31 |
69
- | **Draftfit DFlash DPACE (Mine)** | **89.84** | **103.20** | **91.62** | **76.97** |
70
- | DFlash2 | 78.37 | 92.76 | 87.49 | 70.41 |
71
 
72
  Full prompt-free summary: [`evaluation-v4.json`](evaluation-v4.json)
73
 
@@ -75,14 +60,14 @@ Full prompt-free summary: [`evaluation-v4.json`](evaluation-v4.json)
75
 
76
  | Draft | Best use |
77
  | --- | --- |
78
- | **DSpark** | Best first choice when adapting an existing DSpark draft to a new target |
79
- | **DFlash CE** | Simple block-parallel DFlash baseline |
80
- | **DFlash DPACE** | Test position-weighted training objectives against CE |
81
- | **DFlash2** | Research richer candidate ranking / grouped-convolution drafts |
82
 
83
  For this MiniCPM5-SFT experiment, **DSpark is the strongest practical baseline**.
84
 
85
- Draftfit DSpark reached **347.94 tok/s vs 354.69 tok/s** for the official DSpark on A100, while reaching **116.15 vs 105.38 tok/s** in the tested MLX setup.
86
 
87
  These measurements do **not** establish a general speed advantage.
88
 
@@ -93,12 +78,7 @@ These measurements do **not** establish a general speed advantage.
93
  | `main` | DFlash DPACE, primary release |
94
  | `v5` | Additional 1,000-step continual-training stage on a fresh split |
95
 
96
- Best observed DSpark A100 result:
97
-
98
- - Draftfit DSpark [`v3-best`](https://huggingface.co/jayyun98/MiniCPM5-2B-SFT-Draftfit-DSpark/tree/v3-best): **360.26 tok/s**
99
- - Matching official DSpark: **361.36 tok/s**
100
-
101
- Different revisions used different held-out sets, so these numbers should not be used to rank revisions directly.
102
 
103
  ## Validation notes
104
 
@@ -116,7 +96,7 @@ Training mixture per stage:
116
  Important limitations:
117
 
118
  - DSpark/DFlash serving correctness still requires exact token/state and acceptance-length validation.
119
- - DFlash2 was tested on MLX but not on A100 with SGLang 0.5.16.
120
  - A successfully loaded draft checkpoint does **not** prove correct speculative decoding.
121
  - Reported speed is specific to the tested target, hardware, verifier settings and prompt set.
122
 
 
27
 
28
  ## Results
29
 
 
 
 
 
 
 
 
 
 
 
 
30
  ### MLX / M5 Pro
31
 
32
  | Draft | Overall tok/s |
33
  | --- | ---: |
 
34
  | Official DSpark | 105.38 |
35
  | Draftfit DSpark | **116.15** |
36
+ | Draftfit DFlash CE | 89.61 |
37
+ | Draftfit DFlash DPACE | 89.97 |
38
+ | Draftfit DFlash2 | 80.75 |
 
 
39
 
40
  Validation set: **53 held-out conversations**
41
 
 
48
 
49
  | Draft | PerfectBlend | OpenCode | APIGen-MT | Nemotron Agentic |
50
  | --- | ---: | ---: | ---: | ---: |
 
51
  | Official DSpark | 107.83 | 112.19 | 107.79 | 85.86 |
52
  | Draftfit DSpark | **118.29** | **122.28** | **116.04** | **91.99** |
53
+ | Draftfit DFlash CE | 89.02 | 99.18 | 92.49 | 79.31 |
54
+ | Draftfit DFlash DPACE | 89.84 | 103.20 | 91.62 | 76.97 |
55
+ | Draftfit DFlash2 | 78.37 | 92.76 | 87.49 | 70.41 |
56
 
57
  Full prompt-free summary: [`evaluation-v4.json`](evaluation-v4.json)
58
 
 
60
 
61
  | Draft | Best use |
62
  | --- | --- |
63
+ | DSpark | Best first choice when adapting an existing DSpark draft to a new target |
64
+ | DFlash CE | Simple block-parallel DFlash baseline |
65
+ | DFlash DPACE | Test position-weighted training objectives against CE |
66
+ | DFlash2 | Research richer candidate ranking / grouped-convolution drafts |
67
 
68
  For this MiniCPM5-SFT experiment, **DSpark is the strongest practical baseline**.
69
 
70
+ Draftfit DSpark reached **116.15 tok/s vs 105.38 tok/s** for the official DSpark in the tested MLX setup.
71
 
72
  These measurements do **not** establish a general speed advantage.
73
 
 
78
  | `main` | DFlash DPACE, primary release |
79
  | `v5` | Additional 1,000-step continual-training stage on a fresh split |
80
 
81
+ Different revisions used different held-out sets, so their measurements should not be used to rank revisions directly.
 
 
 
 
 
82
 
83
  ## Validation notes
84
 
 
96
  Important limitations:
97
 
98
  - DSpark/DFlash serving correctness still requires exact token/state and acceptance-length validation.
99
+ - DFlash2 was tested on MLX but not with SGLang 0.5.16.
100
  - A successfully loaded draft checkpoint does **not** prove correct speculative decoding.
101
  - Reported speed is specific to the tested target, hardware, verifier settings and prompt set.
102