Add T5Gemma 2 DSpark Base checkpoint and benchmarks

README.md

@@ -0,0 +1,262 @@
+---
+language:
+- en
+license: gemma
+library_name: vllm
+base_model: google/t5gemma-2-1b-1b
+datasets:
+- HuggingFaceH4/ultrachat_200k
+- Aeala/ShareGPT_Vicuna_unfiltered
+- ise-uiuc/Magicoder-Evol-Instruct-110K
+- Salesforce/xlam-function-calling-60k
+- glaiveai/glaive-function-calling-v2
+- TIGER-Lab/MathInstruct
+tags:
+- speculative-decoding
+- speculators
+- vllm
+- t5gemma2
+- dspark
+---
+
+# T5Gemma 2 1B-1B DSpark Base
+
+This repository contains a three-layer DSpark speculative draft model trained for the frozen [google/t5gemma-2-1b-1b](https://huggingface.co/google/t5gemma-2-1b-1b) verifier.
+
+"Base" denotes the reduced three-layer draft. [DSpark Large](https://huggingface.co/d0rj/t5gemma-2-1b-1b.dspark-large) uses five draft transformer layers with the same data, vocabulary, target-state selection, Markov head, confidence head, and training budget. The smaller model trades a modest amount of acceptance accuracy for substantially lower draft compute.
+
+This is a **speculative draft checkpoint**, not a standalone language model. The repository includes inference weights, complete optimizer/scheduler state, validation metrics, continuation metadata, exact training commands, both DSpark TensorBoard runs, and raw benchmark artifacts for the Base and Large comparisons.
+
+## Model details
+
+| Item | Value |
+|---|---|
+| Draft architecture | `DSparkDraftModel` |
+| Verifier | `google/t5gemma-2-1b-1b` |
+| Draft transformer layers | 3 |
+| Large comparison | 5 draft transformer layers |
+| Draft hidden / intermediate size | 1,152 / 6,912 |
+| Draft vocabulary | 32,000 tokens mapped from the verifier vocabulary |
+| Speculative block | 8 positions: one anchor and up to 7 proposed tokens |
+| Default proposals per step | 7 |
+| Runtime override | 1-7 proposed tokens |
+| Verifier decoder states | 2, 7, 13, 18, 21 |
+| Markov head | Vanilla first-order bias, rank 256 |
+| Confidence head | Enabled; conditioned on the Markov embedding |
+| Precision | BF16 |
+| Selected checkpoint | Epoch 5 (sixth/final epoch), global step 146,486 |
+
+## Usage with vLLM
+
+Install a compatible vLLM build and the [T5Gemma 2 vLLM plugin](https://github.com/d0rj/t5gemma2-vllm-plugin):
+
+```bash
+pip install "git+https://github.com/d0rj/t5gemma2-vllm-plugin.git"
+```
+
+The plugin is discovered automatically through vLLM entry points; `VLLM_PLUGINS` is not required.
+
+### Serve the Hub checkpoint with its default K=7
+
+```bash
+vllm serve d0rj/t5gemma-2-1b-1b.dspark-base \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --served-model-name t5gemma-2-1b-1b-dspark-base \
+  --trust-remote-code \
+  --no-enable-chunked-prefill \
+  --max-model-len 2048 \
+  --max-num-seqs 1
+```
+
+The plugin reads the verifier reference from this checkpoint and loads `google/t5gemma-2-1b-1b` automatically.
+
+### Override the number of proposed tokens
+
+The adapter uses vLLM's DFlash scheduler for every DFlash-family checkpoint and selects DSpark from the checkpoint architecture.
+
+```bash
+vllm serve google/t5gemma-2-1b-1b \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --served-model-name t5gemma-2-1b-1b-dspark-base-k3 \
+  --trust-remote-code \
+  --no-enable-chunked-prefill \
+  --max-model-len 2048 \
+  --max-num-seqs 1 \
+  --speculative-config '{"model":"d0rj/t5gemma-2-1b-1b.dspark-base","method":"dflash","num_speculative_tokens":3}'
+```
+
+Set `num_speculative_tokens` to an integer from 1 through 7. K=3 and K=7 were benchmarked below. CUDA Graph remains enabled when `--enforce-eager` is omitted.
+
+```bash
+curl http://127.0.0.1:8000/v1/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "t5gemma-2-1b-1b-dspark-base",
+    "prompt": "Solve step by step.\n\nQuestion: A train travels 60 miles in 2 hours. What is its average speed?\nAnswer:",
+    "max_tokens": 128,
+    "temperature": 0
+  }'
+```
+
+This checkpoint is not intended to be loaded independently through Transformers `AutoModelForSeq2SeqLM`.
+
+## Training
+
+Training used online hidden-state extraction. The frozen verifier and trainable draft shared one RTX 5070 Ti 16 GB GPU under WSL2; hidden states were generated in process and were not stored for the complete dataset.
+
+### Data mixture
+
+The deterministically shuffled seed-42 mixture contained exactly 200,000 examples.
+
+| Dataset | Examples | Share | Coverage |
+|---|---:|---:|---|
+| [UltraChat 200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) | 60,000 | 30% | General instruction following and dialogue |
+| [ShareGPT Vicuna unfiltered](https://huggingface.co/datasets/Aeala/ShareGPT_Vicuna_unfiltered) | 20,000 | 10% | Multi-turn conversation |
+| [Magicoder Evol Instruct](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) | 50,000 | 25% | Code and software tasks |
+| [xLAM function calling](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) | 40,000 | 20% | Tool and function calling |
+| [Glaive function calling v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) | 20,000 | 10% | Tool use and function calling |
+| [MathInstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct) | 10,000 | 5% | Mathematics and reasoning |
+
+The prepared data used a 90%/10% train/validation split. Encoder inputs were truncated to 2,048 tokens and decoder targets to 1,024 tokens. Exact source counts and converters are in [training/data_config.json](training/data_config.json).
+
+### Hyperparameters
+
+| Parameter | Value |
+|---|---|
+| Draft layers | 3 |
+| Epochs | 6 |
+| Optimizer | AdamW |
+| Learning rate / weight decay | 6e-4 / 0.01 |
+| Scheduler | Linear; automatic 1% warmup |
+| Packed token budget | 2,048 |
+| Online extraction microbatch | 32 |
+| Maximum anchors / block size | 256 / 8 |
+| Hidden-state dtype | BF16 |
+| Noise standard deviation | 0.05 |
+| Main loss | 0.1 cross-entropy + 0.9 total variation |
+| Confidence loss weight | 1.0 |
+| Verifier | Frozen |
+| Dataloader workers | 0 |
+| Seed | 42 |
+| Training revision | `4bd884b204230bbbec44dbb6538eb82593c4016f` |
+| Software | Speculators 0.7.0.dev72; Transformers 5.12.1; PyTorch 2.12.0 |
+
+The launcher is [training/train.sh](training/train.sh). [train_command.txt](train_command.txt) records the resolved command and software versions.
+
+### TensorBoard
+
+Both requested DSpark runs are included:
+
+- `dspark_base_t5gemma2_mixture_200k`: this three-layer model;
+- `dspark_t5gemma2_mixture_200k`: the five-layer Large model.
+
+```bash
+tensorboard --logdir tensorboard --port 6006
+```
+
+## Validation comparison
+
+These are teacher-forced validation metrics from each selected epoch-5 checkpoint. Runtime acceptance is lower because later speculative positions depend on every preceding draft token being accepted.
+
+| Metric | DSpark Large (5L) | DSpark Base (3L) |
+|---|---:|---:|
+| Validation loss | 0.6738 | 0.6961 |
+| Full-block token accuracy | 45.73% | 43.67% |
+| Teacher-forced acceptance rate | 42.77% | 40.55% |
+| Teacher-forced acceptance length | 2.557 | 2.418 |
+| Confidence loss | 0.2815 | 0.2832 |
+| Confidence absolute error | 0.1886 | 0.1896 |
+| Position 1 accuracy | 66.60% | 64.26% |
+| Position 2 accuracy | 55.52% | 53.21% |
+| Position 3 accuracy | 47.79% | 45.51% |
+| Position 4 accuracy | 42.48% | 40.30% |
+| Position 5 accuracy | 38.69% | 36.74% |
+| Position 6 accuracy | 35.74% | 33.96% |
+| Position 7 accuracy | 33.21% | 31.62% |
+
+Unrounded Base values are in [val_metrics.json](val_metrics.json); both runs can be inspected in TensorBoard.
+
+## vLLM performance: Base vs Large
+
+Both campaigns used identical prompt snapshots: their per-dataset SHA-256 hashes match. Settings were also identical within each batch size: greedy decoding, seed 0, maximum model length 2,048, maximum output 128, `--max-num-batched-tokens 4096`, GPU memory utilization 0.85, chunked prefill disabled, CUDA Graph enabled, and 8 excluded warm-up requests.
+
+BS1 used concurrency/max-num-seqs 1; BS4 used 4. Dataset sizes were MATH-500 500, GSM8K 1,319, HumanEval 164, MBPP 500, MT-Bench 160 independent turns, and UltraChat `test_sft` 5,000.
+
+The raw verifier throughput drifted between the Large and Base benchmark campaigns, particularly at BS4. Therefore each draft is compared against the baseline measured in its own campaign. The `Normalized gain of 3L` column is:
+
+```text
+(Base draft / Base campaign verifier) / (Large draft / Large campaign verifier)
+```
+
+This avoids presenting baseline drift as an architectural speedup. A value above 1.0 means the three-layer draft improved normalized end-to-end throughput relative to Large.
+
+Within paired columns, values are always `Large (5L) / Base (3L)`. Latency, TTFT, and ITL show mean/p95 milliseconds. Acceptance rate is accepted draft tokens divided by all proposals; p1 is first-position acceptance; acceptance length is `1 + accepted_tokens / draft_steps`.
+
+### Batch size 1
+
+| Dataset | K | Verifier tok/s (5L run / 3L run) | Draft tok/s (5L / 3L) | Speedup (5L / 3L) | Normalized gain of 3L | Latency mean/p95 ms (5L / 3L) | TTFT mean/p95 ms (5L / 3L) | ITL mean/p95 ms (5L / 3L) | Accept rate (5L / 3L) | p1 (5L / 3L) | Accept len (5L / 3L) |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| math500 | 3 | 113.45 / 123.60 | 131.01 / 152.19 | 1.155x / 1.231x | 1.066x | 972.82/1383.58 / 837.38/1146.01 | 139.16/243.83 / 119.19/213.45 | 6.59/9.67 / 5.68/8.04 | 44.80% / 41.56% | 65.41% / 63.26% | 2.344 / 2.247 |
+| math500 | 7 | 113.45 / 123.60 | 136.48 / 149.95 | 1.203x / 1.213x | 1.008x | 932.87/1415.98 / 849.09/1242.33 | 142.46/239.27 / 129.85/256.74 | 6.26/10.08 / 5.69/8.78 | 22.84% / 20.32% | 64.00% / 62.06% | 2.599 / 2.423 |
+| gsm8k | 3 | 115.56 / 127.30 | 111.95 / 128.83 | 0.969x / 1.012x | 1.045x | 1140.44/1627.45 / 990.98/1346.59 | 137.90/232.76 / 111.31/207.46 | 7.91/11.64 / 6.94/9.58 | 31.19% / 27.13% | 53.50% / 48.78% | 1.936 / 1.814 |
+| gsm8k | 7 | 115.56 / 127.30 | 111.09 / 121.81 | 0.961x / 0.957x | 0.995x | 1149.49/1690.77 / 1048.34/1480.12 | 142.42/238.71 / 127.43/265.25 | 7.94/11.94 / 7.26/10.53 | 14.63% / 12.43% | 53.36% / 47.85% | 2.024 / 1.870 |
+| humaneval | 3 | 93.66 / 112.12 | 130.99 / 162.20 | 1.399x / 1.447x | 1.034x | 530.74/905.70 / 428.57/818.54 | 146.04/236.92 / 100.60/181.40 | 6.10/8.98 / 5.12/7.48 | 60.43% / 57.68% | 77.12% / 75.54% | 2.813 / 2.730 |
+| humaneval | 7 | 93.66 / 112.12 | 145.41 / 149.24 | 1.553x / 1.331x | 0.857x | 455.86/852.21 / 444.15/816.37 | 143.63/241.35 / 160.06/265.69 | 5.51/9.66 / 4.94/8.73 | 34.97% / 33.09% | 73.45% / 73.07% | 3.448 / 3.317 |
+| mbpp | 3 | 117.61 / 130.38 | 146.73 / 175.87 | 1.248x / 1.349x | 1.081x | 840.11/1144.96 / 700.90/919.01 | 133.81/238.02 / 108.79/204.45 | 5.95/8.07 / 5.01/6.63 | 52.37% / 50.96% | 71.03% / 70.10% | 2.571 / 2.529 |
+| mbpp | 7 | 117.61 / 130.38 | 158.24 / 175.67 | 1.346x / 1.347x | 1.001x | 773.38/1069.82 / 696.66/978.64 | 144.88/240.50 / 144.75/269.57 | 5.46/7.70 / 4.79/6.83 | 28.85% / 27.43% | 68.24% / 67.37% | 3.020 / 2.920 |
+| mt_bench | 3 | 116.89 / 127.47 | 121.09 / 144.01 | 1.036x / 1.130x | 1.091x | 1037.41/1565.51 / 872.29/1313.48 | 141.77/240.41 / 97.45/185.92 | 7.31/11.71 / 6.34/10.21 | 37.06% / 32.28% | 59.00% / 54.41% | 2.112 / 1.968 |
+| mt_bench | 7 | 116.89 / 127.47 | 122.24 / 132.15 | 1.046x / 1.037x | 0.991x | 1027.68/1596.33 / 950.60/1560.65 | 141.25/255.56 / 132.69/258.32 | 7.28/11.79 / 6.66/11.56 | 17.42% / 14.79% | 56.41% / 52.76% | 2.220 / 2.035 |
+| ultrachat | 3 | 62.83 / 66.23 | 78.10 / 83.93 | 1.243x / 1.267x | 1.019x | 1628.21/2287.42 / 1515.15/2205.47 | 171.98/258.65 / 154.38/250.98 | 11.54/16.24 / 10.78/15.62 | 32.41% / 30.25% | 55.57% / 53.35% | 1.972 / 1.907 |
+| ultrachat | 7 | 62.83 / 66.23 | 78.70 / 83.77 | 1.253x / 1.265x | 1.010x | 1616.26/2290.96 / 1518.37/2203.91 | 176.72/263.78 / 165.57/262.97 | 11.40/16.28 / 10.72/15.63 | 15.45% / 14.19% | 55.33% / 53.12% | 2.081 / 1.993 |
+
+### Batch size 4
+
+| Dataset | K | Verifier tok/s (5L run / 3L run) | Draft tok/s (5L / 3L) | Speedup (5L / 3L) | Normalized gain of 3L | Latency mean/p95 ms (5L / 3L) | TTFT mean/p95 ms (5L / 3L) | ITL mean/p95 ms (5L / 3L) | Accept rate (5L / 3L) | p1 (5L / 3L) | Accept len (5L / 3L) |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| math500 | 3 | 266.74 / 327.25 | 286.34 / 424.44 | 1.073x / 1.297x | 1.208x | 1774.14/2512.93 / 1195.68/1773.15 | 237.01/434.45 / 143.30/297.68 | 12.18/17.88 / 8.35/12.35 | 44.99% / 41.55% | 65.67% / 63.32% | 2.350 / 2.246 |
+| math500 | 7 | 266.74 / 327.25 | 294.75 / 394.37 | 1.105x / 1.205x | 1.091x | 1725.25/2646.27 / 1286.87/1958.87 | 247.76/461.03 / 169.23/310.46 | 11.72/18.88 / 8.87/13.72 | 22.89% / 20.28% | 63.98% / 61.78% | 2.602 / 2.419 |
+| gsm8k | 3 | 280.58 / 344.96 | 267.82 / 376.84 | 0.954x / 1.092x | 1.144x | 1904.86/2717.59 / 1352.79/1901.42 | 232.41/436.56 / 146.36/300.51 | 13.20/19.22 / 9.52/13.59 | 31.15% / 27.04% | 53.33% / 48.59% | 1.934 / 1.811 |
+| gsm8k | 7 | 280.58 / 344.96 | 256.27 / 361.79 | 0.913x / 1.049x | 1.148x | 1991.49/2946.94 / 1409.89/2024.30 | 240.23/457.41 / 149.38/303.52 | 13.82/21.07 / 9.94/14.45 | 14.63% / 12.44% | 53.19% / 47.59% | 2.024 / 1.871 |
+| humaneval | 3 | 194.21 / 265.07 | 234.79 / 371.59 | 1.209x / 1.402x | 1.160x | 1172.97/2107.93 / 731.68/1438.71 | 265.30/502.88 / 153.84/371.52 | 14.78/29.63 / 9.22/16.54 | 59.91% / 57.23% | 76.53% / 75.49% | 2.797 / 2.717 |
+| humaneval | 7 | 194.21 / 265.07 | 210.60 / 388.96 | 1.084x / 1.467x | 1.353x | 1276.63/2616.22 / 695.93/1258.20 | 564.26/1681.50 / 170.66/440.41 | 12.49/25.28 / 9.07/18.30 | 35.09% / 33.28% | 74.01% / 73.49% | 3.457 / 3.329 |
+| mbpp | 3 | 275.24 / 345.00 | 310.80 / 478.08 | 1.129x / 1.386x | 1.227x | 1592.68/2178.77 / 1031.81/1450.64 | 240.73/455.66 / 149.58/290.25 | 11.35/15.68 / 7.49/10.49 | 52.29% / 51.15% | 71.25% / 70.32% | 2.569 / 2.535 |
+| mbpp | 7 | 275.24 / 345.00 | 274.50 / 482.65 | 0.997x / 1.399x | 1.403x | 1794.33/3171.44 / 1016.76/1519.82 | 643.14/2119.01 / 157.02/310.51 | 10.10/14.88 / 7.46/11.04 | 28.83% / 27.60% | 68.32% / 67.48% | 3.018 / 2.932 |
+| mt_bench | 3 | 309.23 / 359.97 | 282.93 / 395.71 | 0.915x / 1.099x | 1.201x | 1771.67/2604.84 / 1267.36/1889.96 | 252.12/503.80 / 149.93/296.09 | 12.36/19.15 / 8.95/13.28 | 36.54% / 32.00% | 58.03% / 54.34% | 2.096 / 1.960 |
+| mt_bench | 7 | 309.23 / 359.97 | 224.25 / 356.55 | 0.725x / 0.991x | 1.366x | 2225.17/3657.94 / 1404.46/2271.47 | 838.84/2441.89 / 182.32/363.33 | 11.43/22.87 / 9.92/17.31 | 17.22% / 14.60% | 55.30% / 51.91% | 2.205 / 2.022 |
+| ultrachat | 3 | 155.46 / 169.47 | 190.92 / 231.21 | 1.228x / 1.364x | 1.111x | 2665.16/3441.95 / 2200.76/2860.88 | 264.37/429.11 / 203.06/327.66 | 19.01/24.80 / 15.82/20.66 | 32.40% / 30.23% | 55.58% / 53.29% | 1.972 / 1.907 |
+| ultrachat | 7 | 155.46 / 169.47 | 189.60 / 230.66 | 1.220x / 1.361x | 1.116x | 2683.34/3469.80 / 2205.89/2923.48 | 287.66/433.69 / 208.65/335.64 | 18.97/25.07 / 15.81/21.20 | 15.40% / 14.16% | 55.21% / 53.16% | 2.078 / 1.991 |
+
+Full Base reports are in [benchmarks/base](benchmarks/base); the complete Large suite reports are in [benchmarks/large](benchmarks/large). These include all quantiles, prompt hashes, request counts, baseline rows, CSV tables, and raw JSON.
+
+## Repository contents
+
+- `model.safetensors`: selected three-layer DSpark weights.
+- `config.json`, `config.py`: architecture and verifier linkage.
+- `optimizer_state_dict.pt`: complete AdamW state.
+- `scheduler_state_dict.pt`: scheduler state.
+- `training_state.json`: epoch/global-step continuation metadata.
+- `train_command.txt`: exact training command and package versions.
+- `val_metrics.json`: selected-checkpoint validation metrics.
+- `training/`: data metadata and launcher.
+- `tensorboard/`: Base and Large DSpark TensorBoard events.
+- `benchmarks/`: Base and Large bs1/bs4 reports.
+
+For optimizer-level continuation, restore the root checkpoint files as an epoch directory in a compatible Speculators checkpoint tree and point `checkpoint_best` to it.
+
+## Limitations
+
+- Performance depends on hardware, workload, batch size, and vLLM version; universal acceleration is not claimed.
+- Base and Large were measured in separate benchmark campaigns. Prompt hashes and settings match, but the two co-run verifier measurements demonstrate system-level drift.
+- The plugin path was validated for single-GPU text serving. Tensor/pipeline parallelism, quantized serving, and the vision path require separate validation.
+- Teacher-forced validation accuracy does not guarantee equal runtime acceptance.
+- K=3 reduces draft work but changes block length relative to eight-position training; K=7 matches the trained block shape.
+- Generated outputs retain the behavior and limitations of the verifier and training data.
+
+## License and attribution
+
+This checkpoint is derived from T5Gemma 2 and is distributed under the Gemma license. Using the verifier requires accepting Google's Gemma terms. Each dataset retains its own license and conditions.
+
+Training used the [vLLM Speculators project](https://github.com/vllm-project/speculators) with T5Gemma 2, DSpark, online-extraction, CUDA Graph, and logging extensions from the accompanying development branch.
+

benchmarks/base/bs1/comparison.csv

@@ -0,0 +1,19 @@
+dataset,run,method,num_speculative_tokens,requests,output_tokens,request_throughput_rps,output_throughput_tps,speedup_vs_baseline,latency_mean_ms,latency_p50_ms,latency_p95_ms,latency_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,acceptance_rate,acceptance_pos_1,acceptance_pos_2,acceptance_pos_3,mean_acceptance_length
+math500,baseline,baseline,,500,63642,0.9710887696354362,123.60406295427687,1.0,1029.6528329752618,1011.0086214990588,1183.8821351528165,1352.1066228914415,137.17931756953476,125.86030399688752,254.1806304405327,259.12194621312665,7.065699614096616,6.948497625951028,8.149714840967464,9.294363555888458,7.066986512325109,6.949724716546406,8.150898662537166,,,,,
+math500,dspark_k3,dspark,3,500,63731,1.1940337177445612,152.19392573115726,1.2313019660806477,837.3801779437345,816.9088775030104,1146.009227049944,1370.9137859771727,119.19492300227284,106.72050050197868,213.44604070691275,258.3343633514596,5.6791052364180485,5.593892366200956,8.0383705036205,9.9072340297603,5.680338308854861,5.595103622135508,8.040039556758208,0.4156275683926265,0.6326171187037689,0.3862627685804861,0.2280028178936245,2.2468827051778795
+math500,dspark_k7,dspark,7,500,63668,1.1775733318385104,149.94747778298856,1.2131274182989966,849.0922174340812,839.2276294980547,1242.3280104500004,1512.9399566508073,129.85324780782685,118.26306749571813,256.7381934488367,268.83207743172534,5.692451923082981,5.603080988136545,8.775799772777777,10.470896226674313,5.693717860424883,5.604267641709472,8.777032548852267,0.2032188585715368,0.6206151981210698,0.36616410334116223,0.20482612319115084,2.422532010000758
+gsm8k,baseline,baseline,,1319,168500,0.99646643680205,127.29688749139153,1.0,1003.4296112389243,979.7711369901663,1147.1265090964152,1165.5087842789362,121.82556504392818,98.20042399223894,253.87967379938343,259.2248212755658,6.953847864544167,6.954055936964903,7.2364834158344005,7.467789549431415,6.955123454672103,6.955686165340699,7.237693709486319,,,,,
+gsm8k,dspark_k3,dspark,3,1319,168418,1.0089794437230273,128.83267623422654,1.0120646213202884,990.9843431904721,969.3124250043184,1346.5886881953336,1535.857625339122,111.30873365058301,95.77244399406482,207.4599851010134,258.9313290748395,6.9409634880545115,6.796889598449274,9.581487464620439,10.733646140910508,6.942225311685386,6.797988936978788,9.582715377937562,0.27131373077960613,0.48776908551389264,0.22053412462908012,0.10563798219584569,1.8139411923388185
+gsm8k,dspark_k7,dspark,7,1319,168457,0.9537832973729679,121.81309547047616,0.9569212403462244,1048.3418924380537,1024.0400490001775,1480.1234933955127,1657.6216600838231,127.43488244511649,108.85212599532679,265.2473596011987,272.91192704375135,7.264521106390136,7.101249275637395,10.530224299220105,11.874427473184914,7.265778957962821,7.102425023638387,10.531432203184004,0.12425870324814467,0.4784887142079961,0.21339195505667877,0.10025647003963628,1.8698109227370128
+humaneval,baseline,baseline,,164,11593,1.5860477240562036,112.1161662498998,1.0,630.4013797863018,523.5549619974336,1166.231724152749,1309.8181371219107,107.49666936503561,68.79427550302353,235.7000987394713,257.95137990207877,7.297341666050263,7.1785622135738,8.49481829626492,9.019322222237124,7.301280505914361,7.182021026551093,8.49614809654986,,,,,
+humaneval,dspark_k3,dspark,3,164,11403,2.3328174923608316,162.20193820360097,1.4467310436040344,428.57000031148254,387.70782500796486,818.5407340446543,898.805770642066,100.60342578719606,86.00447800563416,181.3961499959987,206.1478519160301,5.122034968621708,4.931764990461113,7.476613033529377,8.192469397143373,5.12586804041208,4.935711371615048,7.481607739016129,0.5768270082990895,0.7553782934493595,0.5569253081943437,0.4181774232535654,2.7304810248972684
+humaneval,dspark_k7,dspark,7,164,10873,2.251028218808984,149.24042575067125,1.3311231621854054,444.1504760051896,404.9552284996025,816.3704766055158,1066.9646934645427,160.0552513159417,160.3144644977874,265.6927762531268,272.09467787863105,4.944825696685838,4.49954001268619,8.732022160872603,10.318099194431854,4.949079180467763,4.503242393642178,8.749463808814781,0.33094945594945596,0.7306511056511057,0.538083538083538,0.382985257985258,3.316646191646192
+mbpp,baseline,baseline,,500,61467,1.060577545227861,130.38103994504186,1.0,942.7637862757256,959.554503999243,1136.071662903123,1164.7922664470389,128.43507730966667,113.83070349984337,256.7646375973709,260.46189150976716,6.666311532516699,6.6432852756234775,7.121261330345493,7.26582164827209,6.669662882298301,6.644507716595485,7.122378109070412,,,,,
+mbpp,dspark_k3,dspark,3,500,61643,1.426522036352137,175.87019577370955,1.3488939484440547,700.897503109707,703.9270094974199,919.0087530580053,1045.5224799610733,108.79212251797435,91.88079299929086,204.44850095518615,252.88790728605818,5.011328370238571,4.792381047239568,6.631624132701181,11.0604020699849,5.013954450995089,4.79360114569697,6.632888974820347,0.5095989615358338,0.7010043041606887,0.49940561590489857,0.32838696454191435,2.5287968846075017
+mbpp,dspark_k7,dspark,7,500,61199,1.4351988286444157,175.6654662284192,1.347323708282014,696.6608542179456,697.1229435002897,978.6413718429683,1134.9274124257497,144.74703813757515,139.05306548986118,269.57134354670416,274.25451427989174,4.7933126349011435,4.529900153696456,6.829579977194199,12.186756946117372,4.796161684500074,4.531059523688732,6.830801591285134,0.27433826068703066,0.6736976821349007,0.462388017253638,0.2917950419490923,2.9203678248092144
+mt_bench,baseline,baseline,,160,20102,1.0145474949999134,127.46521090305163,1.0,985.5352811689045,984.0788365036133,1160.0982246920466,1228.7251105705216,135.88225862440595,116.72725749667734,258.65039654891007,261.56884412805084,6.802873811459109,6.694544555138573,7.932162838574364,8.945381395555675,6.808100097450948,6.695755173252367,7.93327857724984,,,,,
+mt_bench,dspark_k3,dspark,3,160,20102,1.146245568010398,144.01142755090638,1.1298096675212783,872.2913337876889,843.7197764942539,1313.481805704941,1433.86794267004,97.44770207444162,75.82400399405742,185.91803135059308,229.50114874707637,6.340698793466556,5.966741342590682,10.206342749464907,12.089351741436634,6.344304410333824,5.967888015807757,10.32622098520203,0.3227736367202825,0.5441349548842683,0.2823656335817968,0.14182032169478226,1.9683209101608474
+mt_bench,dspark_k7,dspark,7,160,20102,1.0518269898218957,132.14891343374842,1.0367449478764772,950.6035467624315,909.5398534991546,1560.6508335607941,1760.411608337017,132.69152166194544,122.75661650346592,258.31930338608794,269.76503039040836,6.65775048881359,6.321849094407714,11.558155716175955,13.248131914494198,6.6624969313339735,6.32326457077476,11.559374125591408,0.14785346059984694,0.5276458101688062,0.257859092287476,0.1271606186192257,2.034974224198929
+ultrachat,baseline,baseline,,5000,636089,0.5205862124506341,66.22783265830228,1.0,1920.7680450268758,1795.524564004154,3165.0052286931896,3293.7335572592565,134.13350982365372,127.32078000408364,226.59587289090268,258.95474469623883,14.14775085821201,13.20624435037479,23.33603281298938,24.09970887749654,14.149112711234228,13.207468862130167,23.337274902295345,,,,,
+ultrachat,dspark_k3,dspark,3,5000,635846,0.6599485226793063,83.92512567030923,1.2672183627586737,1515.1526615684213,1470.6976669913274,2205.4712326978915,2478.7801955970654,154.38469593704212,149.33674249914475,250.97870194367718,286.67091090654156,10.78135369824052,10.431902874002185,15.61618643974625,17.696921136958977,10.78267526592669,10.433149968439395,15.617432073222732,0.30246536910398036,0.5334846321484933,0.2569174118884798,0.11699406327496806,1.907396107311941
+ultrachat,dspark_k7,dspark,7,5000,636040,0.6585516278400763,83.77303547428042,1.2649218932846762,1518.3681885767612,1479.9964584963163,2203.905933455099,2513.144597544161,165.56593623632216,164.75157599779777,262.971773991012,305.4997507331428,10.715100522278119,10.43376617324282,15.631104971251936,17.89630725780433,10.71641033546772,10.434974913300184,15.632339404468743,0.14185244726020002,0.5312465705748265,0.25502381392840456,0.11516911181415417,1.9929671308214

benchmarks/base/bs1/comparison.md

@@ -0,0 +1,20 @@
+| Dataset | Run | Req | Output tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | TPOT mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| math500 | baseline | 500 | 123.60 | 1.000x | 1029.65/1183.88 | 137.18/254.18 | 7.07/8.15 | 7.07/8.15 | — | -/-/- | — |
+| math500 | dspark_k3 | 500 | 152.19 | 1.231x | 837.38/1146.01 | 119.19/213.45 | 5.68/8.04 | 5.68/8.04 | 41.56% | 63.26%/38.63%/22.80% | 2.247 |
+| math500 | dspark_k7 | 500 | 149.95 | 1.213x | 849.09/1242.33 | 129.85/256.74 | 5.69/8.78 | 5.69/8.78 | 20.32% | 62.06%/36.62%/20.48% | 2.423 |
+| gsm8k | baseline | 1319 | 127.30 | 1.000x | 1003.43/1147.13 | 121.83/253.88 | 6.95/7.24 | 6.96/7.24 | — | -/-/- | — |
+| gsm8k | dspark_k3 | 1319 | 128.83 | 1.012x | 990.98/1346.59 | 111.31/207.46 | 6.94/9.58 | 6.94/9.58 | 27.13% | 48.78%/22.05%/10.56% | 1.814 |
+| gsm8k | dspark_k7 | 1319 | 121.81 | 0.957x | 1048.34/1480.12 | 127.43/265.25 | 7.26/10.53 | 7.27/10.53 | 12.43% | 47.85%/21.34%/10.03% | 1.870 |
+| humaneval | baseline | 164 | 112.12 | 1.000x | 630.40/1166.23 | 107.50/235.70 | 7.30/8.49 | 7.30/8.50 | — | -/-/- | — |
+| humaneval | dspark_k3 | 164 | 162.20 | 1.447x | 428.57/818.54 | 100.60/181.40 | 5.12/7.48 | 5.13/7.48 | 57.68% | 75.54%/55.69%/41.82% | 2.730 |
+| humaneval | dspark_k7 | 164 | 149.24 | 1.331x | 444.15/816.37 | 160.06/265.69 | 4.94/8.73 | 4.95/8.75 | 33.09% | 73.07%/53.81%/38.30% | 3.317 |
+| mbpp | baseline | 500 | 130.38 | 1.000x | 942.76/1136.07 | 128.44/256.76 | 6.67/7.12 | 6.67/7.12 | — | -/-/- | — |
+| mbpp | dspark_k3 | 500 | 175.87 | 1.349x | 700.90/919.01 | 108.79/204.45 | 5.01/6.63 | 5.01/6.63 | 50.96% | 70.10%/49.94%/32.84% | 2.529 |
+| mbpp | dspark_k7 | 500 | 175.67 | 1.347x | 696.66/978.64 | 144.75/269.57 | 4.79/6.83 | 4.80/6.83 | 27.43% | 67.37%/46.24%/29.18% | 2.920 |
+| mt_bench | baseline | 160 | 127.47 | 1.000x | 985.54/1160.10 | 135.88/258.65 | 6.80/7.93 | 6.81/7.93 | — | -/-/- | — |
+| mt_bench | dspark_k3 | 160 | 144.01 | 1.130x | 872.29/1313.48 | 97.45/185.92 | 6.34/10.21 | 6.34/10.33 | 32.28% | 54.41%/28.24%/14.18% | 1.968 |
+| mt_bench | dspark_k7 | 160 | 132.15 | 1.037x | 950.60/1560.65 | 132.69/258.32 | 6.66/11.56 | 6.66/11.56 | 14.79% | 52.76%/25.79%/12.72% | 2.035 |
+| ultrachat | baseline | 5000 | 66.23 | 1.000x | 1920.77/3165.01 | 134.13/226.60 | 14.15/23.34 | 14.15/23.34 | — | -/-/- | — |
+| ultrachat | dspark_k3 | 5000 | 83.93 | 1.267x | 1515.15/2205.47 | 154.38/250.98 | 10.78/15.62 | 10.78/15.62 | 30.25% | 53.35%/25.69%/11.70% | 1.907 |
+| ultrachat | dspark_k7 | 5000 | 83.77 | 1.265x | 1518.37/2203.91 | 165.57/262.97 | 10.72/15.63 | 10.72/15.63 | 14.19% | 53.12%/25.50%/11.52% | 1.993 |

benchmarks/base/bs1/results.json

@@ -0,0 +1,560 @@
+[
+  {
+    "dataset": "math500",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 63642,
+    "request_throughput_rps": 0.9710887696354362,
+    "output_throughput_tps": 123.60406295427687,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1029.6528329752618,
+    "latency_p50_ms": 1011.0086214990588,
+    "latency_p95_ms": 1183.8821351528165,
+    "latency_p99_ms": 1352.1066228914415,
+    "ttft_mean_ms": 137.17931756953476,
+    "ttft_p50_ms": 125.86030399688752,
+    "ttft_p95_ms": 254.1806304405327,
+    "ttft_p99_ms": 259.12194621312665,
+    "itl_mean_ms": 7.065699614096616,
+    "itl_p50_ms": 6.948497625951028,
+    "itl_p95_ms": 8.149714840967464,
+    "itl_p99_ms": 9.294363555888458,
+    "tpot_mean_ms": 7.066986512325109,
+    "tpot_p50_ms": 6.949724716546406,
+    "tpot_p95_ms": 8.150898662537166,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 1.1940337177445612,
+    "output_throughput_tps": 152.19392573115726,
+    "speedup_vs_baseline": 1.2313019660806477,
+    "latency_mean_ms": 837.3801779437345,
+    "latency_p50_ms": 816.9088775030104,
+    "latency_p95_ms": 1146.009227049944,
+    "latency_p99_ms": 1370.9137859771727,
+    "ttft_mean_ms": 119.19492300227284,
+    "ttft_p50_ms": 106.72050050197868,
+    "ttft_p95_ms": 213.44604070691275,
+    "ttft_p99_ms": 258.3343633514596,
+    "itl_mean_ms": 5.6791052364180485,
+    "itl_p50_ms": 5.593892366200956,
+    "itl_p95_ms": 8.0383705036205,
+    "itl_p99_ms": 9.9072340297603,
+    "tpot_mean_ms": 5.680338308854861,
+    "tpot_p50_ms": 5.595103622135508,
+    "tpot_p95_ms": 8.040039556758208,
+    "acceptance_rate": 0.4156275683926265,
+    "acceptance_pos_1": 0.6326171187037689,
+    "acceptance_pos_2": 0.3862627685804861,
+    "acceptance_pos_3": 0.2280028178936245,
+    "mean_acceptance_length": 2.2468827051778795
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 1.1775733318385104,
+    "output_throughput_tps": 149.94747778298856,
+    "speedup_vs_baseline": 1.2131274182989966,
+    "latency_mean_ms": 849.0922174340812,
+    "latency_p50_ms": 839.2276294980547,
+    "latency_p95_ms": 1242.3280104500004,
+    "latency_p99_ms": 1512.9399566508073,
+    "ttft_mean_ms": 129.85324780782685,
+    "ttft_p50_ms": 118.26306749571813,
+    "ttft_p95_ms": 256.7381934488367,
+    "ttft_p99_ms": 268.83207743172534,
+    "itl_mean_ms": 5.692451923082981,
+    "itl_p50_ms": 5.603080988136545,
+    "itl_p95_ms": 8.775799772777777,
+    "itl_p99_ms": 10.470896226674313,
+    "tpot_mean_ms": 5.693717860424883,
+    "tpot_p50_ms": 5.604267641709472,
+    "tpot_p95_ms": 8.777032548852267,
+    "acceptance_rate": 0.2032188585715368,
+    "acceptance_pos_1": 0.6206151981210698,
+    "acceptance_pos_2": 0.36616410334116223,
+    "acceptance_pos_3": 0.20482612319115084,
+    "mean_acceptance_length": 2.422532010000758
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 1319,
+    "output_tokens": 168500,
+    "request_throughput_rps": 0.99646643680205,
+    "output_throughput_tps": 127.29688749139153,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1003.4296112389243,
+    "latency_p50_ms": 979.7711369901663,
+    "latency_p95_ms": 1147.1265090964152,
+    "latency_p99_ms": 1165.5087842789362,
+    "ttft_mean_ms": 121.82556504392818,
+    "ttft_p50_ms": 98.20042399223894,
+    "ttft_p95_ms": 253.87967379938343,
+    "ttft_p99_ms": 259.2248212755658,
+    "itl_mean_ms": 6.953847864544167,
+    "itl_p50_ms": 6.954055936964903,
+    "itl_p95_ms": 7.2364834158344005,
+    "itl_p99_ms": 7.467789549431415,
+    "tpot_mean_ms": 6.955123454672103,
+    "tpot_p50_ms": 6.955686165340699,
+    "tpot_p95_ms": 7.237693709486319,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 1.0089794437230273,
+    "output_throughput_tps": 128.83267623422654,
+    "speedup_vs_baseline": 1.0120646213202884,
+    "latency_mean_ms": 990.9843431904721,
+    "latency_p50_ms": 969.3124250043184,
+    "latency_p95_ms": 1346.5886881953336,
+    "latency_p99_ms": 1535.857625339122,
+    "ttft_mean_ms": 111.30873365058301,
+    "ttft_p50_ms": 95.77244399406482,
+    "ttft_p95_ms": 207.4599851010134,
+    "ttft_p99_ms": 258.9313290748395,
+    "itl_mean_ms": 6.9409634880545115,
+    "itl_p50_ms": 6.796889598449274,
+    "itl_p95_ms": 9.581487464620439,
+    "itl_p99_ms": 10.733646140910508,
+    "tpot_mean_ms": 6.942225311685386,
+    "tpot_p50_ms": 6.797988936978788,
+    "tpot_p95_ms": 9.582715377937562,
+    "acceptance_rate": 0.27131373077960613,
+    "acceptance_pos_1": 0.48776908551389264,
+    "acceptance_pos_2": 0.22053412462908012,
+    "acceptance_pos_3": 0.10563798219584569,
+    "mean_acceptance_length": 1.8139411923388185
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.9537832973729679,
+    "output_throughput_tps": 121.81309547047616,
+    "speedup_vs_baseline": 0.9569212403462244,
+    "latency_mean_ms": 1048.3418924380537,
+    "latency_p50_ms": 1024.0400490001775,
+    "latency_p95_ms": 1480.1234933955127,
+    "latency_p99_ms": 1657.6216600838231,
+    "ttft_mean_ms": 127.43488244511649,
+    "ttft_p50_ms": 108.85212599532679,
+    "ttft_p95_ms": 265.2473596011987,
+    "ttft_p99_ms": 272.91192704375135,
+    "itl_mean_ms": 7.264521106390136,
+    "itl_p50_ms": 7.101249275637395,
+    "itl_p95_ms": 10.530224299220105,
+    "itl_p99_ms": 11.874427473184914,
+    "tpot_mean_ms": 7.265778957962821,
+    "tpot_p50_ms": 7.102425023638387,
+    "tpot_p95_ms": 10.531432203184004,
+    "acceptance_rate": 0.12425870324814467,
+    "acceptance_pos_1": 0.4784887142079961,
+    "acceptance_pos_2": 0.21339195505667877,
+    "acceptance_pos_3": 0.10025647003963628,
+    "mean_acceptance_length": 1.8698109227370128
+  },
+  {
+    "dataset": "humaneval",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 164,
+    "output_tokens": 11593,
+    "request_throughput_rps": 1.5860477240562036,
+    "output_throughput_tps": 112.1161662498998,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 630.4013797863018,
+    "latency_p50_ms": 523.5549619974336,
+    "latency_p95_ms": 1166.231724152749,
+    "latency_p99_ms": 1309.8181371219107,
+    "ttft_mean_ms": 107.49666936503561,
+    "ttft_p50_ms": 68.79427550302353,
+    "ttft_p95_ms": 235.7000987394713,
+    "ttft_p99_ms": 257.95137990207877,
+    "itl_mean_ms": 7.297341666050263,
+    "itl_p50_ms": 7.1785622135738,
+    "itl_p95_ms": 8.49481829626492,
+    "itl_p99_ms": 9.019322222237124,
+    "tpot_mean_ms": 7.301280505914361,
+    "tpot_p50_ms": 7.182021026551093,
+    "tpot_p95_ms": 8.49614809654986,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 2.3328174923608316,
+    "output_throughput_tps": 162.20193820360097,
+    "speedup_vs_baseline": 1.4467310436040344,
+    "latency_mean_ms": 428.57000031148254,
+    "latency_p50_ms": 387.70782500796486,
+    "latency_p95_ms": 818.5407340446543,
+    "latency_p99_ms": 898.805770642066,
+    "ttft_mean_ms": 100.60342578719606,
+    "ttft_p50_ms": 86.00447800563416,
+    "ttft_p95_ms": 181.3961499959987,
+    "ttft_p99_ms": 206.1478519160301,
+    "itl_mean_ms": 5.122034968621708,
+    "itl_p50_ms": 4.931764990461113,
+    "itl_p95_ms": 7.476613033529377,
+    "itl_p99_ms": 8.192469397143373,
+    "tpot_mean_ms": 5.12586804041208,
+    "tpot_p50_ms": 4.935711371615048,
+    "tpot_p95_ms": 7.481607739016129,
+    "acceptance_rate": 0.5768270082990895,
+    "acceptance_pos_1": 0.7553782934493595,
+    "acceptance_pos_2": 0.5569253081943437,
+    "acceptance_pos_3": 0.4181774232535654,
+    "mean_acceptance_length": 2.7304810248972684
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 2.251028218808984,
+    "output_throughput_tps": 149.24042575067125,
+    "speedup_vs_baseline": 1.3311231621854054,
+    "latency_mean_ms": 444.1504760051896,
+    "latency_p50_ms": 404.9552284996025,
+    "latency_p95_ms": 816.3704766055158,
+    "latency_p99_ms": 1066.9646934645427,
+    "ttft_mean_ms": 160.0552513159417,
+    "ttft_p50_ms": 160.3144644977874,
+    "ttft_p95_ms": 265.6927762531268,
+    "ttft_p99_ms": 272.09467787863105,
+    "itl_mean_ms": 4.944825696685838,
+    "itl_p50_ms": 4.49954001268619,
+    "itl_p95_ms": 8.732022160872603,
+    "itl_p99_ms": 10.318099194431854,
+    "tpot_mean_ms": 4.949079180467763,
+    "tpot_p50_ms": 4.503242393642178,
+    "tpot_p95_ms": 8.749463808814781,
+    "acceptance_rate": 0.33094945594945596,
+    "acceptance_pos_1": 0.7306511056511057,
+    "acceptance_pos_2": 0.538083538083538,
+    "acceptance_pos_3": 0.382985257985258,
+    "mean_acceptance_length": 3.316646191646192
+  },
+  {
+    "dataset": "mbpp",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 61467,
+    "request_throughput_rps": 1.060577545227861,
+    "output_throughput_tps": 130.38103994504186,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 942.7637862757256,
+    "latency_p50_ms": 959.554503999243,
+    "latency_p95_ms": 1136.071662903123,
+    "latency_p99_ms": 1164.7922664470389,
+    "ttft_mean_ms": 128.43507730966667,
+    "ttft_p50_ms": 113.83070349984337,
+    "ttft_p95_ms": 256.7646375973709,
+    "ttft_p99_ms": 260.46189150976716,
+    "itl_mean_ms": 6.666311532516699,
+    "itl_p50_ms": 6.6432852756234775,
+    "itl_p95_ms": 7.121261330345493,
+    "itl_p99_ms": 7.26582164827209,
+    "tpot_mean_ms": 6.669662882298301,
+    "tpot_p50_ms": 6.644507716595485,
+    "tpot_p95_ms": 7.122378109070412,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 1.426522036352137,
+    "output_throughput_tps": 175.87019577370955,
+    "speedup_vs_baseline": 1.3488939484440547,
+    "latency_mean_ms": 700.897503109707,
+    "latency_p50_ms": 703.9270094974199,
+    "latency_p95_ms": 919.0087530580053,
+    "latency_p99_ms": 1045.5224799610733,
+    "ttft_mean_ms": 108.79212251797435,
+    "ttft_p50_ms": 91.88079299929086,
+    "ttft_p95_ms": 204.44850095518615,
+    "ttft_p99_ms": 252.88790728605818,
+    "itl_mean_ms": 5.011328370238571,
+    "itl_p50_ms": 4.792381047239568,
+    "itl_p95_ms": 6.631624132701181,
+    "itl_p99_ms": 11.0604020699849,
+    "tpot_mean_ms": 5.013954450995089,
+    "tpot_p50_ms": 4.79360114569697,
+    "tpot_p95_ms": 6.632888974820347,
+    "acceptance_rate": 0.5095989615358338,
+    "acceptance_pos_1": 0.7010043041606887,
+    "acceptance_pos_2": 0.49940561590489857,
+    "acceptance_pos_3": 0.32838696454191435,
+    "mean_acceptance_length": 2.5287968846075017
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 1.4351988286444157,
+    "output_throughput_tps": 175.6654662284192,
+    "speedup_vs_baseline": 1.347323708282014,
+    "latency_mean_ms": 696.6608542179456,
+    "latency_p50_ms": 697.1229435002897,
+    "latency_p95_ms": 978.6413718429683,
+    "latency_p99_ms": 1134.9274124257497,
+    "ttft_mean_ms": 144.74703813757515,
+    "ttft_p50_ms": 139.05306548986118,
+    "ttft_p95_ms": 269.57134354670416,
+    "ttft_p99_ms": 274.25451427989174,
+    "itl_mean_ms": 4.7933126349011435,
+    "itl_p50_ms": 4.529900153696456,
+    "itl_p95_ms": 6.829579977194199,
+    "itl_p99_ms": 12.186756946117372,
+    "tpot_mean_ms": 4.796161684500074,
+    "tpot_p50_ms": 4.531059523688732,
+    "tpot_p95_ms": 6.830801591285134,
+    "acceptance_rate": 0.27433826068703066,
+    "acceptance_pos_1": 0.6736976821349007,
+    "acceptance_pos_2": 0.462388017253638,
+    "acceptance_pos_3": 0.2917950419490923,
+    "mean_acceptance_length": 2.9203678248092144
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.0145474949999134,
+    "output_throughput_tps": 127.46521090305163,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 985.5352811689045,
+    "latency_p50_ms": 984.0788365036133,
+    "latency_p95_ms": 1160.0982246920466,
+    "latency_p99_ms": 1228.7251105705216,
+    "ttft_mean_ms": 135.88225862440595,
+    "ttft_p50_ms": 116.72725749667734,
+    "ttft_p95_ms": 258.65039654891007,
+    "ttft_p99_ms": 261.56884412805084,
+    "itl_mean_ms": 6.802873811459109,
+    "itl_p50_ms": 6.694544555138573,
+    "itl_p95_ms": 7.932162838574364,
+    "itl_p99_ms": 8.945381395555675,
+    "tpot_mean_ms": 6.808100097450948,
+    "tpot_p50_ms": 6.695755173252367,
+    "tpot_p95_ms": 7.93327857724984,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.146245568010398,
+    "output_throughput_tps": 144.01142755090638,
+    "speedup_vs_baseline": 1.1298096675212783,
+    "latency_mean_ms": 872.2913337876889,
+    "latency_p50_ms": 843.7197764942539,
+    "latency_p95_ms": 1313.481805704941,
+    "latency_p99_ms": 1433.86794267004,
+    "ttft_mean_ms": 97.44770207444162,
+    "ttft_p50_ms": 75.82400399405742,
+    "ttft_p95_ms": 185.91803135059308,
+    "ttft_p99_ms": 229.50114874707637,
+    "itl_mean_ms": 6.340698793466556,
+    "itl_p50_ms": 5.966741342590682,
+    "itl_p95_ms": 10.206342749464907,
+    "itl_p99_ms": 12.089351741436634,
+    "tpot_mean_ms": 6.344304410333824,
+    "tpot_p50_ms": 5.967888015807757,
+    "tpot_p95_ms": 10.32622098520203,
+    "acceptance_rate": 0.3227736367202825,
+    "acceptance_pos_1": 0.5441349548842683,
+    "acceptance_pos_2": 0.2823656335817968,
+    "acceptance_pos_3": 0.14182032169478226,
+    "mean_acceptance_length": 1.9683209101608474
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.0518269898218957,
+    "output_throughput_tps": 132.14891343374842,
+    "speedup_vs_baseline": 1.0367449478764772,
+    "latency_mean_ms": 950.6035467624315,
+    "latency_p50_ms": 909.5398534991546,
+    "latency_p95_ms": 1560.6508335607941,
+    "latency_p99_ms": 1760.411608337017,
+    "ttft_mean_ms": 132.69152166194544,
+    "ttft_p50_ms": 122.75661650346592,
+    "ttft_p95_ms": 258.31930338608794,
+    "ttft_p99_ms": 269.76503039040836,
+    "itl_mean_ms": 6.65775048881359,
+    "itl_p50_ms": 6.321849094407714,
+    "itl_p95_ms": 11.558155716175955,
+    "itl_p99_ms": 13.248131914494198,
+    "tpot_mean_ms": 6.6624969313339735,
+    "tpot_p50_ms": 6.32326457077476,
+    "tpot_p95_ms": 11.559374125591408,
+    "acceptance_rate": 0.14785346059984694,
+    "acceptance_pos_1": 0.5276458101688062,
+    "acceptance_pos_2": 0.257859092287476,
+    "acceptance_pos_3": 0.1271606186192257,
+    "mean_acceptance_length": 2.034974224198929
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 5000,
+    "output_tokens": 636089,
+    "request_throughput_rps": 0.5205862124506341,
+    "output_throughput_tps": 66.22783265830228,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1920.7680450268758,
+    "latency_p50_ms": 1795.524564004154,
+    "latency_p95_ms": 3165.0052286931896,
+    "latency_p99_ms": 3293.7335572592565,
+    "ttft_mean_ms": 134.13350982365372,
+    "ttft_p50_ms": 127.32078000408364,
+    "ttft_p95_ms": 226.59587289090268,
+    "ttft_p99_ms": 258.95474469623883,
+    "itl_mean_ms": 14.14775085821201,
+    "itl_p50_ms": 13.20624435037479,
+    "itl_p95_ms": 23.33603281298938,
+    "itl_p99_ms": 24.09970887749654,
+    "tpot_mean_ms": 14.149112711234228,
+    "tpot_p50_ms": 13.207468862130167,
+    "tpot_p95_ms": 23.337274902295345,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.6599485226793063,
+    "output_throughput_tps": 83.92512567030923,
+    "speedup_vs_baseline": 1.2672183627586737,
+    "latency_mean_ms": 1515.1526615684213,
+    "latency_p50_ms": 1470.6976669913274,
+    "latency_p95_ms": 2205.4712326978915,
+    "latency_p99_ms": 2478.7801955970654,
+    "ttft_mean_ms": 154.38469593704212,
+    "ttft_p50_ms": 149.33674249914475,
+    "ttft_p95_ms": 250.97870194367718,
+    "ttft_p99_ms": 286.67091090654156,
+    "itl_mean_ms": 10.78135369824052,
+    "itl_p50_ms": 10.431902874002185,
+    "itl_p95_ms": 15.61618643974625,
+    "itl_p99_ms": 17.696921136958977,
+    "tpot_mean_ms": 10.78267526592669,
+    "tpot_p50_ms": 10.433149968439395,
+    "tpot_p95_ms": 15.617432073222732,
+    "acceptance_rate": 0.30246536910398036,
+    "acceptance_pos_1": 0.5334846321484933,
+    "acceptance_pos_2": 0.2569174118884798,
+    "acceptance_pos_3": 0.11699406327496806,
+    "mean_acceptance_length": 1.907396107311941
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.6585516278400763,
+    "output_throughput_tps": 83.77303547428042,
+    "speedup_vs_baseline": 1.2649218932846762,
+    "latency_mean_ms": 1518.3681885767612,
+    "latency_p50_ms": 1479.9964584963163,
+    "latency_p95_ms": 2203.905933455099,
+    "latency_p99_ms": 2513.144597544161,
+    "ttft_mean_ms": 165.56593623632216,
+    "ttft_p50_ms": 164.75157599779777,
+    "ttft_p95_ms": 262.971773991012,
+    "ttft_p99_ms": 305.4997507331428,
+    "itl_mean_ms": 10.715100522278119,
+    "itl_p50_ms": 10.43376617324282,
+    "itl_p95_ms": 15.631104971251936,
+    "itl_p99_ms": 17.89630725780433,
+    "tpot_mean_ms": 10.71641033546772,
+    "tpot_p50_ms": 10.434974913300184,
+    "tpot_p95_ms": 15.632339404468743,
+    "acceptance_rate": 0.14185244726020002,
+    "acceptance_pos_1": 0.5312465705748265,
+    "acceptance_pos_2": 0.25502381392840456,
+    "acceptance_pos_3": 0.11516911181415417,
+    "mean_acceptance_length": 1.9929671308214
+  }
+]

benchmarks/base/bs1/run_config.json

@@ -0,0 +1,86 @@
+{
+  "args": {
+    "base_model": "google/t5gemma-2-1b-1b",
+    "dflash_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+    "dflare_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+    "dspark_model": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints/5",
+    "speculative_token_counts": [
+      3,
+      7
+    ],
+    "methods": [
+      "dspark"
+    ],
+    "datasets": [
+      "math500",
+      "gsm8k",
+      "humaneval",
+      "mbpp",
+      "mt_bench",
+      "ultrachat"
+    ],
+    "num_prompts": 0,
+    "ultrachat_num_prompts": 5000,
+    "warmup": 8,
+    "concurrency": 1,
+    "max_num_seqs": 1,
+    "max_num_batched_tokens": 4096,
+    "max_model_len": 2048,
+    "max_tokens": 128,
+    "gpu_memory_utilization": 0.85,
+    "host": "127.0.0.1",
+    "port": 8011,
+    "temperature": 0.0,
+    "top_p": 1.0,
+    "seed": 0,
+    "timeout_s": 180.0,
+    "server_timeout_s": 600.0,
+    "enforce_eager": false,
+    "include_baseline": true
+  },
+  "runs": [
+    {
+      "name": "baseline",
+      "checkpoint": null,
+      "num_speculative_tokens": null
+    },
+    {
+      "name": "dspark_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dspark_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    }
+  ],
+  "prompt_sets": {
+    "math500": {
+      "count": 500,
+      "sha256": "78a67699bc700588d0e23da6457ea3afc00d5635897d456a6245f1e964449c4a"
+    },
+    "gsm8k": {
+      "count": 1319,
+      "sha256": "d591bbef95c9d5b17f6f26999bde24def2f6fbf25d7e20953b9ac89c65ffbb8e"
+    },
+    "humaneval": {
+      "count": 164,
+      "sha256": "de1066c5b81dc0f99eab434315cee2c3c68c5a522b8c25d53c73f9c9042b44c8"
+    },
+    "mbpp": {
+      "count": 500,
+      "sha256": "c596be7f77834627fd0a05755691e30dbf4f6724425842de6d03348614e5fecd"
+    },
+    "mt_bench": {
+      "count": 160,
+      "sha256": "8761bfece58bd26e5b93c33e9aad480728ea9950a7d257aaff267db5c7b98010"
+    },
+    "ultrachat": {
+      "count": 5000,
+      "sha256": "5dac9e9dc0d63372c701dbebbb18f3f22a568e9b62cc0ff1b4fb88fc41bf5bf7"
+    }
+  },
+  "benchmark_fingerprint": "454343a0220046bba5e4c80bdedae69aac76c1988cbff6920427a5788099f674",
+  "output_dir": "/mnt/d/Projects/dflash_t5/benchmark-results/t5gemma2_dspark_base_suite_bs1_fixed"
+}

benchmarks/base/bs4/comparison.csv

@@ -0,0 +1,19 @@
+dataset,run,method,num_speculative_tokens,requests,output_tokens,request_throughput_rps,output_throughput_tps,speedup_vs_baseline,latency_mean_ms,latency_p50_ms,latency_p95_ms,latency_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,acceptance_rate,acceptance_pos_1,acceptance_pos_2,acceptance_pos_3,mean_acceptance_length
+math500,baseline,baseline,,500,63527,2.575660280906157,327.24794133025085,1.0,1549.881507652055,1541.6923305019736,1901.3991399056977,2073.3104983172957,147.72230004466837,128.53012699633837,277.2130677534733,387.54749131272547,11.11915976237464,11.026409515830519,13.801987814431913,15.665761921923757,11.12106789862526,11.028098413518146,13.803411649602399,,,,,
+math500,dspark_k3,dspark,3,500,63539,3.34003084405518,424.44443960084413,1.2970117944073019,1195.6849749402609,1169.5264599984512,1773.1451036757787,1944.9941020199901,143.30231556599028,116.45177099853754,297.67805395094905,373.04166853777133,8.352826905473885,8.158615838546549,12.354124409472469,14.635059152750648,8.35462870098348,8.163252110248358,12.355534362218949,0.415486407892723,0.6332426800409706,0.38554727510330944,0.22766926853388902,2.246459223678169
+math500,dspark_k7,dspark,7,500,63493,3.1055917217703812,394.36667038073364,1.2051005386852782,1286.8654769104323,1246.8089515023166,1958.8731202602503,2254.721420374407,169.23283278750023,150.04799248708878,310.4628508415771,383.799409790663,8.871197280140835,8.589971834756103,13.716522226358514,15.951512530517089,8.87296793212422,8.60067761416536,13.71767588804694,0.20277687384748888,0.6178056188306758,0.36526195899772207,0.20611237661351556,2.4194381169324224
+gsm8k,baseline,baseline,,1319,168533,2.699786987185922,344.9607280602009,1.0,1479.6602965273742,1454.6987209760118,1803.740513609955,1914.7201473527819,168.7780613106763,145.47203300753608,336.32914681511363,491.5234538837098,10.337075278453744,10.18635707075055,12.4034695235158,13.359812111263784,10.338842536074479,10.187574787391716,12.40562485355891,,,,,
+gsm8k,dspark_k3,dspark,3,1319,168415,2.95131608515978,376.8354044595787,1.0924008845256588,1352.7894092793806,1323.1997079856228,1901.4164583000816,2116.5834879834433,146.35591101451402,119.20796197955497,300.50935930921696,387.0458965114083,9.517948084655174,9.351653133988322,13.592088056107666,15.135234009802675,9.519542839852642,9.352777228419354,13.593510024599716,0.27039498963955716,0.48590297454239884,0.220014867325282,0.10526712705099062,1.8111849689186714
+gsm8k,dspark_k7,dspark,7,1319,168438,2.833061661782182,361.7856256158205,1.0487733709579932,1409.892868064269,1389.3212949915323,2024.2955645022444,2287.086103030015,149.37952628049933,121.9771110045258,303.5228224063758,355.9325602342128,9.94289976587715,9.82163717335926,14.449290831630137,16.439139549680032,9.944535710893788,9.823021669386277,14.453224437911699,0.1243881347729851,0.47593713555839395,0.21317265507858055,0.10223801854834232,1.870716943410896
+humaneval,baseline,baseline,,164,11241,3.867290467605202,265.07446430701265,1.0,1027.5289629021356,879.5409715094138,2018.1860155600586,2274.2641233184268,140.52152795707892,120.44445698847994,278.58940106234513,480.18000137846695,13.187347418172193,12.668324928583226,19.83492009233357,28.834143813500756,13.194289122212655,12.670714736622909,19.844658022235045,,,,,
+humaneval,dspark_k3,dspark,3,164,11202,5.440176459827299,371.59058965235,1.4018347283047567,731.6780699141573,653.8903180044144,1438.7055883620633,1758.196029517276,153.84280301219493,109.24666900245938,371.5150598072796,427.1234392697809,9.217930821861229,8.395432637747984,16.543497819817173,21.385043967787766,9.224179054706173,8.398416746314833,16.54860166646416,0.5723147241266732,0.7548971596474046,0.5506856023506367,0.41136141038197843,2.7169441723800194
+humaneval,dspark_k7,dspark,7,164,11135,5.728781174033297,388.9632827613461,1.467373644527463,695.9294298721943,634.961468007532,1258.2039341956258,1984.7797918543813,170.66249135340757,130.76092200935818,440.41123243077857,548.8665195452631,9.06925487325878,7.807209323622503,18.29704301704432,30.691704785244347,9.07486906638103,7.822465188736613,18.303107731801564,0.33277512988964747,0.7348963029756538,0.5404268109407875,0.38352870453862337,3.3294259092275325
+mbpp,baseline,baseline,,500,61587,2.8009211403112766,345.00066053670116,1.0,1426.2509520059102,1449.8898644960718,1792.6750187427385,1927.6900081126948,146.00965341442497,129.28127399936784,278.12606179941207,302.88602881017135,10.539905415700249,10.33841663381217,12.799749751224665,13.565437609112568,10.546513464177922,10.339677704733752,12.800697692052097,,,,,
+mbpp,dspark_k3,dspark,3,500,61810,3.8673346513351055,478.07990959804573,1.385736215270775,1031.8113529087277,1022.938920505112,1450.6354179771733,1667.4116877623596,149.58043881563935,120.52947049960494,290.24569394096017,465.6294314624392,7.4866564872650105,7.0952005747625035,10.490002014466166,15.616462420633345,7.490425515753587,7.09647927951163,10.49589030935033,0.5115081479053122,0.7031512518952587,0.5011678891939516,0.3302053026267262,2.5345244437159367
+mbpp,dspark_k7,dspark,7,500,61424,3.9288605862280823,482.6526652969474,1.3989905542386718,1016.7640253802527,1009.4089240010362,1519.8205350112396,1755.0660978292578,157.01642078207806,127.09367199568078,310.5101487963111,555.6603015074506,7.464304052053599,6.92667360632959,11.035447346997291,15.66697385307634,7.469533758903007,6.928639874063014,11.131861498385357,0.27595293416337563,0.6748067713025748,0.46773199298212337,0.29465598179145525,2.9316705391436293
+mt_bench,baseline,baseline,,160,20102,2.865123484091851,359.96695173258996,1.0,1379.8839098995813,1349.3874825071543,1778.2662457597326,2064.43852098775,230.34865414410888,188.74082650290802,469.9252384045379,599.9244086013641,9.196491927825377,8.739159606336653,12.184589993809974,13.513825134104701,9.201697317973778,8.741404259885389,12.185761682743273,,,,,
+mt_bench,dspark_k3,dspark,3,160,20228,3.1300295907514477,395.7139910107518,1.0993064477338947,1267.3618169241308,1240.8099970052717,1889.9596753937649,2197.152719998557,149.93388268667331,120.75363399344496,296.08547244570207,370.8032631434613,8.953320217924055,8.487721767728438,13.275213856385248,16.593942496504845,8.957264486664604,8.49002677173625,13.297254559146252,0.3200388161086851,0.5434255215914604,0.27666181465308104,0.14002911208151383,1.9601164483260554
+mt_bench,dspark_k7,dspark,7,160,20102,2.8379656094158685,356.5549042529862,0.9905212201754052,1404.4557639432242,1343.6405550019117,2271.4736478112154,2877.03235346009,182.31840012485918,159.79043499100953,363.3274522420834,631.4187010040041,9.922806420825252,9.43615103145515,17.314218461555217,20.890300572353894,9.927143236485232,9.4376937125298,17.32922917495226,0.1459540058496301,0.5190686471296668,0.258129265355279,0.12515054195102368,2.0216780409474104
+ultrachat,baseline,baseline,,5000,635560,1.3331967798759812,169.4653090835957,1.0,2999.209397312632,3004.6992159914225,3652.681129911798,3792.2580027181543,191.445676819887,180.53925450658426,319.0085294045276,436.0589518790949,22.255913944838188,22.191040874087605,26.93092076657395,27.96015746255569,22.257870607914793,22.192596555178742,26.93375055238194,,,,,
+ultrachat,dspark_k3,dspark,3,5000,636234,1.816992286110778,231.20645403228093,1.364329108314604,2200.762252031482,2194.020344497403,2860.881272106781,3141.828603268659,203.0626388980716,195.85809300770052,327.66038095142005,418.8016837404577,15.817088125396433,15.717495220445654,20.662014169011663,22.842756694141052,15.818637421764908,15.7196392047615,20.663277460188848,0.3022621517716569,0.5328723151898772,0.2570481426653975,0.11686599745969595,1.9067864553149707
+ultrachat,dspark_k7,dspark,7,5000,636210,1.8127834563773697,230.66219255636926,1.361117468841872,2205.888447939884,2192.8646555024898,2923.480463263696,3272.379539081885,208.6473120003706,199.3813320004847,335.6416549038841,448.6493355646964,15.814628500703911,15.656986732382371,21.199671716881205,23.549442959891635,15.81632527800587,15.658670244142634,21.20066768934127,0.14160791914766432,0.5315988273762544,0.25404655416494404,0.11493529271745531,1.9912554340336501

benchmarks/base/bs4/comparison.md

@@ -0,0 +1,20 @@
+| Dataset | Run | Req | Output tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | TPOT mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| math500 | baseline | 500 | 327.25 | 1.000x | 1549.88/1901.40 | 147.72/277.21 | 11.12/13.80 | 11.12/13.80 | — | -/-/- | — |
+| math500 | dspark_k3 | 500 | 424.44 | 1.297x | 1195.68/1773.15 | 143.30/297.68 | 8.35/12.35 | 8.35/12.36 | 41.55% | 63.32%/38.55%/22.77% | 2.246 |
+| math500 | dspark_k7 | 500 | 394.37 | 1.205x | 1286.87/1958.87 | 169.23/310.46 | 8.87/13.72 | 8.87/13.72 | 20.28% | 61.78%/36.53%/20.61% | 2.419 |
+| gsm8k | baseline | 1319 | 344.96 | 1.000x | 1479.66/1803.74 | 168.78/336.33 | 10.34/12.40 | 10.34/12.41 | — | -/-/- | — |
+| gsm8k | dspark_k3 | 1319 | 376.84 | 1.092x | 1352.79/1901.42 | 146.36/300.51 | 9.52/13.59 | 9.52/13.59 | 27.04% | 48.59%/22.00%/10.53% | 1.811 |
+| gsm8k | dspark_k7 | 1319 | 361.79 | 1.049x | 1409.89/2024.30 | 149.38/303.52 | 9.94/14.45 | 9.94/14.45 | 12.44% | 47.59%/21.32%/10.22% | 1.871 |
+| humaneval | baseline | 164 | 265.07 | 1.000x | 1027.53/2018.19 | 140.52/278.59 | 13.19/19.83 | 13.19/19.84 | — | -/-/- | — |
+| humaneval | dspark_k3 | 164 | 371.59 | 1.402x | 731.68/1438.71 | 153.84/371.52 | 9.22/16.54 | 9.22/16.55 | 57.23% | 75.49%/55.07%/41.14% | 2.717 |
+| humaneval | dspark_k7 | 164 | 388.96 | 1.467x | 695.93/1258.20 | 170.66/440.41 | 9.07/18.30 | 9.07/18.30 | 33.28% | 73.49%/54.04%/38.35% | 3.329 |
+| mbpp | baseline | 500 | 345.00 | 1.000x | 1426.25/1792.68 | 146.01/278.13 | 10.54/12.80 | 10.55/12.80 | — | -/-/- | — |
+| mbpp | dspark_k3 | 500 | 478.08 | 1.386x | 1031.81/1450.64 | 149.58/290.25 | 7.49/10.49 | 7.49/10.50 | 51.15% | 70.32%/50.12%/33.02% | 2.535 |
+| mbpp | dspark_k7 | 500 | 482.65 | 1.399x | 1016.76/1519.82 | 157.02/310.51 | 7.46/11.04 | 7.47/11.13 | 27.60% | 67.48%/46.77%/29.47% | 2.932 |
+| mt_bench | baseline | 160 | 359.97 | 1.000x | 1379.88/1778.27 | 230.35/469.93 | 9.20/12.18 | 9.20/12.19 | — | -/-/- | — |
+| mt_bench | dspark_k3 | 160 | 395.71 | 1.099x | 1267.36/1889.96 | 149.93/296.09 | 8.95/13.28 | 8.96/13.30 | 32.00% | 54.34%/27.67%/14.00% | 1.960 |
+| mt_bench | dspark_k7 | 160 | 356.55 | 0.991x | 1404.46/2271.47 | 182.32/363.33 | 9.92/17.31 | 9.93/17.33 | 14.60% | 51.91%/25.81%/12.52% | 2.022 |
+| ultrachat | baseline | 5000 | 169.47 | 1.000x | 2999.21/3652.68 | 191.45/319.01 | 22.26/26.93 | 22.26/26.93 | — | -/-/- | — |
+| ultrachat | dspark_k3 | 5000 | 231.21 | 1.364x | 2200.76/2860.88 | 203.06/327.66 | 15.82/20.66 | 15.82/20.66 | 30.23% | 53.29%/25.70%/11.69% | 1.907 |
+| ultrachat | dspark_k7 | 5000 | 230.66 | 1.361x | 2205.89/2923.48 | 208.65/335.64 | 15.81/21.20 | 15.82/21.20 | 14.16% | 53.16%/25.40%/11.49% | 1.991 |

benchmarks/base/bs4/results.json

@@ -0,0 +1,560 @@
+[
+  {
+    "dataset": "math500",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 63527,
+    "request_throughput_rps": 2.575660280906157,
+    "output_throughput_tps": 327.24794133025085,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1549.881507652055,
+    "latency_p50_ms": 1541.6923305019736,
+    "latency_p95_ms": 1901.3991399056977,
+    "latency_p99_ms": 2073.3104983172957,
+    "ttft_mean_ms": 147.72230004466837,
+    "ttft_p50_ms": 128.53012699633837,
+    "ttft_p95_ms": 277.2130677534733,
+    "ttft_p99_ms": 387.54749131272547,
+    "itl_mean_ms": 11.11915976237464,
+    "itl_p50_ms": 11.026409515830519,
+    "itl_p95_ms": 13.801987814431913,
+    "itl_p99_ms": 15.665761921923757,
+    "tpot_mean_ms": 11.12106789862526,
+    "tpot_p50_ms": 11.028098413518146,
+    "tpot_p95_ms": 13.803411649602399,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63539,
+    "request_throughput_rps": 3.34003084405518,
+    "output_throughput_tps": 424.44443960084413,
+    "speedup_vs_baseline": 1.2970117944073019,
+    "latency_mean_ms": 1195.6849749402609,
+    "latency_p50_ms": 1169.5264599984512,
+    "latency_p95_ms": 1773.1451036757787,
+    "latency_p99_ms": 1944.9941020199901,
+    "ttft_mean_ms": 143.30231556599028,
+    "ttft_p50_ms": 116.45177099853754,
+    "ttft_p95_ms": 297.67805395094905,
+    "ttft_p99_ms": 373.04166853777133,
+    "itl_mean_ms": 8.352826905473885,
+    "itl_p50_ms": 8.158615838546549,
+    "itl_p95_ms": 12.354124409472469,
+    "itl_p99_ms": 14.635059152750648,
+    "tpot_mean_ms": 8.35462870098348,
+    "tpot_p50_ms": 8.163252110248358,
+    "tpot_p95_ms": 12.355534362218949,
+    "acceptance_rate": 0.415486407892723,
+    "acceptance_pos_1": 0.6332426800409706,
+    "acceptance_pos_2": 0.38554727510330944,
+    "acceptance_pos_3": 0.22766926853388902,
+    "mean_acceptance_length": 2.246459223678169
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63493,
+    "request_throughput_rps": 3.1055917217703812,
+    "output_throughput_tps": 394.36667038073364,
+    "speedup_vs_baseline": 1.2051005386852782,
+    "latency_mean_ms": 1286.8654769104323,
+    "latency_p50_ms": 1246.8089515023166,
+    "latency_p95_ms": 1958.8731202602503,
+    "latency_p99_ms": 2254.721420374407,
+    "ttft_mean_ms": 169.23283278750023,
+    "ttft_p50_ms": 150.04799248708878,
+    "ttft_p95_ms": 310.4628508415771,
+    "ttft_p99_ms": 383.799409790663,
+    "itl_mean_ms": 8.871197280140835,
+    "itl_p50_ms": 8.589971834756103,
+    "itl_p95_ms": 13.716522226358514,
+    "itl_p99_ms": 15.951512530517089,
+    "tpot_mean_ms": 8.87296793212422,
+    "tpot_p50_ms": 8.60067761416536,
+    "tpot_p95_ms": 13.71767588804694,
+    "acceptance_rate": 0.20277687384748888,
+    "acceptance_pos_1": 0.6178056188306758,
+    "acceptance_pos_2": 0.36526195899772207,
+    "acceptance_pos_3": 0.20611237661351556,
+    "mean_acceptance_length": 2.4194381169324224
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 1319,
+    "output_tokens": 168533,
+    "request_throughput_rps": 2.699786987185922,
+    "output_throughput_tps": 344.9607280602009,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1479.6602965273742,
+    "latency_p50_ms": 1454.6987209760118,
+    "latency_p95_ms": 1803.740513609955,
+    "latency_p99_ms": 1914.7201473527819,
+    "ttft_mean_ms": 168.7780613106763,
+    "ttft_p50_ms": 145.47203300753608,
+    "ttft_p95_ms": 336.32914681511363,
+    "ttft_p99_ms": 491.5234538837098,
+    "itl_mean_ms": 10.337075278453744,
+    "itl_p50_ms": 10.18635707075055,
+    "itl_p95_ms": 12.4034695235158,
+    "itl_p99_ms": 13.359812111263784,
+    "tpot_mean_ms": 10.338842536074479,
+    "tpot_p50_ms": 10.187574787391716,
+    "tpot_p95_ms": 12.40562485355891,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168415,
+    "request_throughput_rps": 2.95131608515978,
+    "output_throughput_tps": 376.8354044595787,
+    "speedup_vs_baseline": 1.0924008845256588,
+    "latency_mean_ms": 1352.7894092793806,
+    "latency_p50_ms": 1323.1997079856228,
+    "latency_p95_ms": 1901.4164583000816,
+    "latency_p99_ms": 2116.5834879834433,
+    "ttft_mean_ms": 146.35591101451402,
+    "ttft_p50_ms": 119.20796197955497,
+    "ttft_p95_ms": 300.50935930921696,
+    "ttft_p99_ms": 387.0458965114083,
+    "itl_mean_ms": 9.517948084655174,
+    "itl_p50_ms": 9.351653133988322,
+    "itl_p95_ms": 13.592088056107666,
+    "itl_p99_ms": 15.135234009802675,
+    "tpot_mean_ms": 9.519542839852642,
+    "tpot_p50_ms": 9.352777228419354,
+    "tpot_p95_ms": 13.593510024599716,
+    "acceptance_rate": 0.27039498963955716,
+    "acceptance_pos_1": 0.48590297454239884,
+    "acceptance_pos_2": 0.220014867325282,
+    "acceptance_pos_3": 0.10526712705099062,
+    "mean_acceptance_length": 1.8111849689186714
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168438,
+    "request_throughput_rps": 2.833061661782182,
+    "output_throughput_tps": 361.7856256158205,
+    "speedup_vs_baseline": 1.0487733709579932,
+    "latency_mean_ms": 1409.892868064269,
+    "latency_p50_ms": 1389.3212949915323,
+    "latency_p95_ms": 2024.2955645022444,
+    "latency_p99_ms": 2287.086103030015,
+    "ttft_mean_ms": 149.37952628049933,
+    "ttft_p50_ms": 121.9771110045258,
+    "ttft_p95_ms": 303.5228224063758,
+    "ttft_p99_ms": 355.9325602342128,
+    "itl_mean_ms": 9.94289976587715,
+    "itl_p50_ms": 9.82163717335926,
+    "itl_p95_ms": 14.449290831630137,
+    "itl_p99_ms": 16.439139549680032,
+    "tpot_mean_ms": 9.944535710893788,
+    "tpot_p50_ms": 9.823021669386277,
+    "tpot_p95_ms": 14.453224437911699,
+    "acceptance_rate": 0.1243881347729851,
+    "acceptance_pos_1": 0.47593713555839395,
+    "acceptance_pos_2": 0.21317265507858055,
+    "acceptance_pos_3": 0.10223801854834232,
+    "mean_acceptance_length": 1.870716943410896
+  },
+  {
+    "dataset": "humaneval",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 164,
+    "output_tokens": 11241,
+    "request_throughput_rps": 3.867290467605202,
+    "output_throughput_tps": 265.07446430701265,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1027.5289629021356,
+    "latency_p50_ms": 879.5409715094138,
+    "latency_p95_ms": 2018.1860155600586,
+    "latency_p99_ms": 2274.2641233184268,
+    "ttft_mean_ms": 140.52152795707892,
+    "ttft_p50_ms": 120.44445698847994,
+    "ttft_p95_ms": 278.58940106234513,
+    "ttft_p99_ms": 480.18000137846695,
+    "itl_mean_ms": 13.187347418172193,
+    "itl_p50_ms": 12.668324928583226,
+    "itl_p95_ms": 19.83492009233357,
+    "itl_p99_ms": 28.834143813500756,
+    "tpot_mean_ms": 13.194289122212655,
+    "tpot_p50_ms": 12.670714736622909,
+    "tpot_p95_ms": 19.844658022235045,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11202,
+    "request_throughput_rps": 5.440176459827299,
+    "output_throughput_tps": 371.59058965235,
+    "speedup_vs_baseline": 1.4018347283047567,
+    "latency_mean_ms": 731.6780699141573,
+    "latency_p50_ms": 653.8903180044144,
+    "latency_p95_ms": 1438.7055883620633,
+    "latency_p99_ms": 1758.196029517276,
+    "ttft_mean_ms": 153.84280301219493,
+    "ttft_p50_ms": 109.24666900245938,
+    "ttft_p95_ms": 371.5150598072796,
+    "ttft_p99_ms": 427.1234392697809,
+    "itl_mean_ms": 9.217930821861229,
+    "itl_p50_ms": 8.395432637747984,
+    "itl_p95_ms": 16.543497819817173,
+    "itl_p99_ms": 21.385043967787766,
+    "tpot_mean_ms": 9.224179054706173,
+    "tpot_p50_ms": 8.398416746314833,
+    "tpot_p95_ms": 16.54860166646416,
+    "acceptance_rate": 0.5723147241266732,
+    "acceptance_pos_1": 0.7548971596474046,
+    "acceptance_pos_2": 0.5506856023506367,
+    "acceptance_pos_3": 0.41136141038197843,
+    "mean_acceptance_length": 2.7169441723800194
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11135,
+    "request_throughput_rps": 5.728781174033297,
+    "output_throughput_tps": 388.9632827613461,
+    "speedup_vs_baseline": 1.467373644527463,
+    "latency_mean_ms": 695.9294298721943,
+    "latency_p50_ms": 634.961468007532,
+    "latency_p95_ms": 1258.2039341956258,
+    "latency_p99_ms": 1984.7797918543813,
+    "ttft_mean_ms": 170.66249135340757,
+    "ttft_p50_ms": 130.76092200935818,
+    "ttft_p95_ms": 440.41123243077857,
+    "ttft_p99_ms": 548.8665195452631,
+    "itl_mean_ms": 9.06925487325878,
+    "itl_p50_ms": 7.807209323622503,
+    "itl_p95_ms": 18.29704301704432,
+    "itl_p99_ms": 30.691704785244347,
+    "tpot_mean_ms": 9.07486906638103,
+    "tpot_p50_ms": 7.822465188736613,
+    "tpot_p95_ms": 18.303107731801564,
+    "acceptance_rate": 0.33277512988964747,
+    "acceptance_pos_1": 0.7348963029756538,
+    "acceptance_pos_2": 0.5404268109407875,
+    "acceptance_pos_3": 0.38352870453862337,
+    "mean_acceptance_length": 3.3294259092275325
+  },
+  {
+    "dataset": "mbpp",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 61587,
+    "request_throughput_rps": 2.8009211403112766,
+    "output_throughput_tps": 345.00066053670116,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1426.2509520059102,
+    "latency_p50_ms": 1449.8898644960718,
+    "latency_p95_ms": 1792.6750187427385,
+    "latency_p99_ms": 1927.6900081126948,
+    "ttft_mean_ms": 146.00965341442497,
+    "ttft_p50_ms": 129.28127399936784,
+    "ttft_p95_ms": 278.12606179941207,
+    "ttft_p99_ms": 302.88602881017135,
+    "itl_mean_ms": 10.539905415700249,
+    "itl_p50_ms": 10.33841663381217,
+    "itl_p95_ms": 12.799749751224665,
+    "itl_p99_ms": 13.565437609112568,
+    "tpot_mean_ms": 10.546513464177922,
+    "tpot_p50_ms": 10.339677704733752,
+    "tpot_p95_ms": 12.800697692052097,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61810,
+    "request_throughput_rps": 3.8673346513351055,
+    "output_throughput_tps": 478.07990959804573,
+    "speedup_vs_baseline": 1.385736215270775,
+    "latency_mean_ms": 1031.8113529087277,
+    "latency_p50_ms": 1022.938920505112,
+    "latency_p95_ms": 1450.6354179771733,
+    "latency_p99_ms": 1667.4116877623596,
+    "ttft_mean_ms": 149.58043881563935,
+    "ttft_p50_ms": 120.52947049960494,
+    "ttft_p95_ms": 290.24569394096017,
+    "ttft_p99_ms": 465.6294314624392,
+    "itl_mean_ms": 7.4866564872650105,
+    "itl_p50_ms": 7.0952005747625035,
+    "itl_p95_ms": 10.490002014466166,
+    "itl_p99_ms": 15.616462420633345,
+    "tpot_mean_ms": 7.490425515753587,
+    "tpot_p50_ms": 7.09647927951163,
+    "tpot_p95_ms": 10.49589030935033,
+    "acceptance_rate": 0.5115081479053122,
+    "acceptance_pos_1": 0.7031512518952587,
+    "acceptance_pos_2": 0.5011678891939516,
+    "acceptance_pos_3": 0.3302053026267262,
+    "mean_acceptance_length": 2.5345244437159367
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61424,
+    "request_throughput_rps": 3.9288605862280823,
+    "output_throughput_tps": 482.6526652969474,
+    "speedup_vs_baseline": 1.3989905542386718,
+    "latency_mean_ms": 1016.7640253802527,
+    "latency_p50_ms": 1009.4089240010362,
+    "latency_p95_ms": 1519.8205350112396,
+    "latency_p99_ms": 1755.0660978292578,
+    "ttft_mean_ms": 157.01642078207806,
+    "ttft_p50_ms": 127.09367199568078,
+    "ttft_p95_ms": 310.5101487963111,
+    "ttft_p99_ms": 555.6603015074506,
+    "itl_mean_ms": 7.464304052053599,
+    "itl_p50_ms": 6.92667360632959,
+    "itl_p95_ms": 11.035447346997291,
+    "itl_p99_ms": 15.66697385307634,
+    "tpot_mean_ms": 7.469533758903007,
+    "tpot_p50_ms": 6.928639874063014,
+    "tpot_p95_ms": 11.131861498385357,
+    "acceptance_rate": 0.27595293416337563,
+    "acceptance_pos_1": 0.6748067713025748,
+    "acceptance_pos_2": 0.46773199298212337,
+    "acceptance_pos_3": 0.29465598179145525,
+    "mean_acceptance_length": 2.9316705391436293
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 2.865123484091851,
+    "output_throughput_tps": 359.96695173258996,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1379.8839098995813,
+    "latency_p50_ms": 1349.3874825071543,
+    "latency_p95_ms": 1778.2662457597326,
+    "latency_p99_ms": 2064.43852098775,
+    "ttft_mean_ms": 230.34865414410888,
+    "ttft_p50_ms": 188.74082650290802,
+    "ttft_p95_ms": 469.9252384045379,
+    "ttft_p99_ms": 599.9244086013641,
+    "itl_mean_ms": 9.196491927825377,
+    "itl_p50_ms": 8.739159606336653,
+    "itl_p95_ms": 12.184589993809974,
+    "itl_p99_ms": 13.513825134104701,
+    "tpot_mean_ms": 9.201697317973778,
+    "tpot_p50_ms": 8.741404259885389,
+    "tpot_p95_ms": 12.185761682743273,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 3.1300295907514477,
+    "output_throughput_tps": 395.7139910107518,
+    "speedup_vs_baseline": 1.0993064477338947,
+    "latency_mean_ms": 1267.3618169241308,
+    "latency_p50_ms": 1240.8099970052717,
+    "latency_p95_ms": 1889.9596753937649,
+    "latency_p99_ms": 2197.152719998557,
+    "ttft_mean_ms": 149.93388268667331,
+    "ttft_p50_ms": 120.75363399344496,
+    "ttft_p95_ms": 296.08547244570207,
+    "ttft_p99_ms": 370.8032631434613,
+    "itl_mean_ms": 8.953320217924055,
+    "itl_p50_ms": 8.487721767728438,
+    "itl_p95_ms": 13.275213856385248,
+    "itl_p99_ms": 16.593942496504845,
+    "tpot_mean_ms": 8.957264486664604,
+    "tpot_p50_ms": 8.49002677173625,
+    "tpot_p95_ms": 13.297254559146252,
+    "acceptance_rate": 0.3200388161086851,
+    "acceptance_pos_1": 0.5434255215914604,
+    "acceptance_pos_2": 0.27666181465308104,
+    "acceptance_pos_3": 0.14002911208151383,
+    "mean_acceptance_length": 1.9601164483260554
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 2.8379656094158685,
+    "output_throughput_tps": 356.5549042529862,
+    "speedup_vs_baseline": 0.9905212201754052,
+    "latency_mean_ms": 1404.4557639432242,
+    "latency_p50_ms": 1343.6405550019117,
+    "latency_p95_ms": 2271.4736478112154,
+    "latency_p99_ms": 2877.03235346009,
+    "ttft_mean_ms": 182.31840012485918,
+    "ttft_p50_ms": 159.79043499100953,
+    "ttft_p95_ms": 363.3274522420834,
+    "ttft_p99_ms": 631.4187010040041,
+    "itl_mean_ms": 9.922806420825252,
+    "itl_p50_ms": 9.43615103145515,
+    "itl_p95_ms": 17.314218461555217,
+    "itl_p99_ms": 20.890300572353894,
+    "tpot_mean_ms": 9.927143236485232,
+    "tpot_p50_ms": 9.4376937125298,
+    "tpot_p95_ms": 17.32922917495226,
+    "acceptance_rate": 0.1459540058496301,
+    "acceptance_pos_1": 0.5190686471296668,
+    "acceptance_pos_2": 0.258129265355279,
+    "acceptance_pos_3": 0.12515054195102368,
+    "mean_acceptance_length": 2.0216780409474104
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 5000,
+    "output_tokens": 635560,
+    "request_throughput_rps": 1.3331967798759812,
+    "output_throughput_tps": 169.4653090835957,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 2999.209397312632,
+    "latency_p50_ms": 3004.6992159914225,
+    "latency_p95_ms": 3652.681129911798,
+    "latency_p99_ms": 3792.2580027181543,
+    "ttft_mean_ms": 191.445676819887,
+    "ttft_p50_ms": 180.53925450658426,
+    "ttft_p95_ms": 319.0085294045276,
+    "ttft_p99_ms": 436.0589518790949,
+    "itl_mean_ms": 22.255913944838188,
+    "itl_p50_ms": 22.191040874087605,
+    "itl_p95_ms": 26.93092076657395,
+    "itl_p99_ms": 27.96015746255569,
+    "tpot_mean_ms": 22.257870607914793,
+    "tpot_p50_ms": 22.192596555178742,
+    "tpot_p95_ms": 26.93375055238194,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636234,
+    "request_throughput_rps": 1.816992286110778,
+    "output_throughput_tps": 231.20645403228093,
+    "speedup_vs_baseline": 1.364329108314604,
+    "latency_mean_ms": 2200.762252031482,
+    "latency_p50_ms": 2194.020344497403,
+    "latency_p95_ms": 2860.881272106781,
+    "latency_p99_ms": 3141.828603268659,
+    "ttft_mean_ms": 203.0626388980716,
+    "ttft_p50_ms": 195.85809300770052,
+    "ttft_p95_ms": 327.66038095142005,
+    "ttft_p99_ms": 418.8016837404577,
+    "itl_mean_ms": 15.817088125396433,
+    "itl_p50_ms": 15.717495220445654,
+    "itl_p95_ms": 20.662014169011663,
+    "itl_p99_ms": 22.842756694141052,
+    "tpot_mean_ms": 15.818637421764908,
+    "tpot_p50_ms": 15.7196392047615,
+    "tpot_p95_ms": 20.663277460188848,
+    "acceptance_rate": 0.3022621517716569,
+    "acceptance_pos_1": 0.5328723151898772,
+    "acceptance_pos_2": 0.2570481426653975,
+    "acceptance_pos_3": 0.11686599745969595,
+    "mean_acceptance_length": 1.9067864553149707
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636210,
+    "request_throughput_rps": 1.8127834563773697,
+    "output_throughput_tps": 230.66219255636926,
+    "speedup_vs_baseline": 1.361117468841872,
+    "latency_mean_ms": 2205.888447939884,
+    "latency_p50_ms": 2192.8646555024898,
+    "latency_p95_ms": 2923.480463263696,
+    "latency_p99_ms": 3272.379539081885,
+    "ttft_mean_ms": 208.6473120003706,
+    "ttft_p50_ms": 199.3813320004847,
+    "ttft_p95_ms": 335.6416549038841,
+    "ttft_p99_ms": 448.6493355646964,
+    "itl_mean_ms": 15.814628500703911,
+    "itl_p50_ms": 15.656986732382371,
+    "itl_p95_ms": 21.199671716881205,
+    "itl_p99_ms": 23.549442959891635,
+    "tpot_mean_ms": 15.81632527800587,
+    "tpot_p50_ms": 15.658670244142634,
+    "tpot_p95_ms": 21.20066768934127,
+    "acceptance_rate": 0.14160791914766432,
+    "acceptance_pos_1": 0.5315988273762544,
+    "acceptance_pos_2": 0.25404655416494404,
+    "acceptance_pos_3": 0.11493529271745531,
+    "mean_acceptance_length": 1.9912554340336501
+  }
+]

benchmarks/base/bs4/run_config.json

@@ -0,0 +1,86 @@
+{
+  "args": {
+    "base_model": "google/t5gemma-2-1b-1b",
+    "dflash_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+    "dflare_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+    "dspark_model": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints/5",
+    "speculative_token_counts": [
+      3,
+      7
+    ],
+    "methods": [
+      "dspark"
+    ],
+    "datasets": [
+      "math500",
+      "gsm8k",
+      "humaneval",
+      "mbpp",
+      "mt_bench",
+      "ultrachat"
+    ],
+    "num_prompts": 0,
+    "ultrachat_num_prompts": 5000,
+    "warmup": 8,
+    "concurrency": 4,
+    "max_num_seqs": 4,
+    "max_num_batched_tokens": 4096,
+    "max_model_len": 2048,
+    "max_tokens": 128,
+    "gpu_memory_utilization": 0.85,
+    "host": "127.0.0.1",
+    "port": 8011,
+    "temperature": 0.0,
+    "top_p": 1.0,
+    "seed": 0,
+    "timeout_s": 180.0,
+    "server_timeout_s": 600.0,
+    "enforce_eager": false,
+    "include_baseline": true
+  },
+  "runs": [
+    {
+      "name": "baseline",
+      "checkpoint": null,
+      "num_speculative_tokens": null
+    },
+    {
+      "name": "dspark_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dspark_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    }
+  ],
+  "prompt_sets": {
+    "math500": {
+      "count": 500,
+      "sha256": "78a67699bc700588d0e23da6457ea3afc00d5635897d456a6245f1e964449c4a"
+    },
+    "gsm8k": {
+      "count": 1319,
+      "sha256": "d591bbef95c9d5b17f6f26999bde24def2f6fbf25d7e20953b9ac89c65ffbb8e"
+    },
+    "humaneval": {
+      "count": 164,
+      "sha256": "de1066c5b81dc0f99eab434315cee2c3c68c5a522b8c25d53c73f9c9042b44c8"
+    },
+    "mbpp": {
+      "count": 500,
+      "sha256": "c596be7f77834627fd0a05755691e30dbf4f6724425842de6d03348614e5fecd"
+    },
+    "mt_bench": {
+      "count": 160,
+      "sha256": "8761bfece58bd26e5b93c33e9aad480728ea9950a7d257aaff267db5c7b98010"
+    },
+    "ultrachat": {
+      "count": 5000,
+      "sha256": "5dac9e9dc0d63372c701dbebbb18f3f22a568e9b62cc0ff1b4fb88fc41bf5bf7"
+    }
+  },
+  "benchmark_fingerprint": "147953dd8a19af693b2d21afde185100031b98caa454b1e3dfbedd4cbd80175b",
+  "output_dir": "/mnt/d/Projects/dflash_t5/benchmark-results/t5gemma2_dspark_base_suite_bs4_fixed"
+}

benchmarks/large/bs1/comparison.csv

@@ -0,0 +1,43 @@
+dataset,run,method,num_speculative_tokens,requests,output_tokens,request_throughput_rps,output_throughput_tps,speedup_vs_baseline,latency_mean_ms,latency_p50_ms,latency_p95_ms,latency_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,acceptance_rate,acceptance_pos_1,acceptance_pos_2,acceptance_pos_3,mean_acceptance_length
+math500,baseline,baseline,,500,63642,0.8912881111809688,113.44671594355843,1.0,1121.8503840199992,1100.7910255029856,1314.5946230510162,1484.1045933025453,169.94078358601837,146.2457435009128,335.47060909950216,373.49712165247183,7.536181297069743,7.382753629891218,8.68439770515185,10.087323529295983,7.537555813759378,7.383992137781786,8.685839803156412,,,,,
+math500,dflash_k3,dflash,3,500,63731,0.5074555001107937,64.68129295512199,0.5701468959868523,1970.4961424658832,1974.9981015011144,2394.326527546944,2618.456352452631,176.97923488401284,151.66605199920014,348.84992924853566,401.735668064066,14.179071923655801,14.188702799193972,17.33207021927915,18.87767345473462,14.180381967095736,14.189951897641887,17.33332515668878,0.06323912607737021,0.16818250150330727,0.01894167167769092,0.0025932050511124475,1.1897173782321107
+math500,dflash_k7,dflash,7,500,63668,0.5823973894271881,74.16015398010043,0.6537003152827826,1716.9249164580651,1721.4581165007985,2122.7248207490447,2282.573311009855,138.95278630002576,141.47450450036558,232.83745254921087,255.96303990992965,12.488242557152846,12.48854895669593,15.423414866535039,16.76060266095041,12.489462153205425,12.489741562990455,15.424737617707956,0.027334900964792862,0.16893291649402992,0.019209763079588684,0.002561301743945158,1.19134430675355
+math500,dflare_k3,dflare,3,500,63731,0.7354717914893614,93.74470548681697,0.8263324742997098,1359.560892554051,1336.2468844989053,1919.7459352993976,2183.3393649194472,150.9933998281267,150.4645309996704,254.30402324909662,281.55977436035755,9.55804924402514,9.37427223622698,13.641124985851773,15.580885431589866,9.559225141953267,9.37540769291087,13.642259077954161,0.39181146804128403,0.6452053391595262,0.35083467039907146,0.1793943945652545,2.175434404123852
+math500,dflare_k7,dflare,7,500,63668,0.7652758346384362,97.4471636795199,0.8589685727703342,1306.6064661140117,1288.1172004999826,1831.9392699944729,2117.9882052937314,149.97463030007202,150.71226700456464,241.5094937001413,269.5821494147821,9.164062260873436,9.109615248048863,13.162089144880586,16.025888632277603,9.165267968224493,9.110725452762747,13.163240382294413,0.18547955359716675,0.6482415681752667,0.3530196021908331,0.17180743730181608,2.2983568751801675
+math500,dspark_k3,dspark,3,500,63731,1.0278341396966633,131.0097951140161,1.1548134648445494,972.8192880541756,951.3454184962029,1383.5792890022267,1658.223661600423,139.16257599797973,140.1768340037961,243.82559089899584,256.6295453281782,6.592793092238339,6.475244685019803,9.666808936191497,11.624004104496413,6.593887856670312,6.476409696829716,9.667887905099029,0.44804447383243945,0.6541032988024392,0.4218646682830064,0.26816545441187273,2.344133421497318
+math500,dspark_k7,dspark,7,500,63668,1.0718413712400758,136.4839928482263,1.2030669351074852,932.8721254997654,924.5265510035097,1415.9799854001901,1651.539169616298,142.46345183774247,146.44532399688615,239.27307030317024,256.7968881978595,6.260414953927999,6.1309666968776675,10.081072708306051,12.369931453538051,6.261500149276782,6.13199001179062,10.082133980709683,0.2283637249725866,0.6400113714819478,0.3961336961377574,0.24038500588880315,2.598546074808106
+gsm8k,baseline,baseline,,1319,168500,0.9045831802614913,115.55895820626328,1.0,1105.3625852046998,1089.5358249981655,1276.3373936999415,1319.7392648432285,164.77733511295938,141.7763389981701,332.90244539603,382.7308631595223,7.419506798811708,7.384766740153826,7.931435276391204,8.172192777608757,7.420847337932145,7.386314905524496,7.932668355122615,,,,,
+gsm8k,dflash_k3,dflash,3,1319,168418,0.5144087930838571,65.6828658935535,0.568392679486734,1943.8572523131418,1934.8288309993222,2365.305382902443,2508.7120192195284,178.53159811208383,159.649838002224,355.616971301788,411.2498142791445,13.929846219481163,13.876137322875465,16.931091559855826,17.821733611478884,13.931159616121034,13.87738181893644,16.93262100001677,0.06835341539008165,0.1835041448500868,0.019489077905896417,0.0020670234142617415,1.2050602461702449
+gsm8k,dflash_k7,dflash,7,1319,168457,0.5735945671017055,73.25702804416376,0.6339363834814603,1743.2748062024025,1739.7538450004504,2099.799393000285,2201.536506499797,152.56252339044147,159.50948600038828,246.41776080115957,261.92414222023217,12.548829301304638,12.529087370075011,15.037010089759487,15.820388422991211,12.550042645359513,12.530304661416404,15.038574778736802,0.02749077227835654,0.1718565178355114,0.018334307172683095,0.002037936980717981,1.1924354059484958
+gsm8k,dflare_k3,dflare,3,1319,168418,0.6339721362775607,80.94944597998045,0.7005034247149607,1577.243275963595,1558.0307409982197,2166.574809700978,2443.7027309426035,150.18097911596254,151.24057199864183,254.45422550219513,278.5744088418869,11.259462841455447,11.12902188976914,15.608968344093311,17.811628303623607,11.260660320716259,11.130129110223226,15.610202330715275,0.2788578880838151,0.5339278493601303,0.21577434619629957,0.08687146869501547,1.8365736642514454
+gsm8k,dflare_k7,dflare,7,1319,168457,0.6414136486780062,81.91858909427664,0.7088899931761125,1558.9424188529392,1528.7051639970741,2191.441884898086,2472.421863719936,152.77644680597203,155.94628499820828,255.01179830098403,283.681260881276,11.093297977502198,10.81332909192518,15.859892152744344,18.10012607823774,11.094531792060318,10.814917770089934,15.861106548793142,0.12798578221940327,0.5417164364111695,0.21953528205359357,0.0932478420590025,1.8959004755358229
+gsm8k,dspark_k3,dspark,3,1319,168418,0.8767773042557421,111.95229721618163,0.9687894296896996,1140.4352353411668,1110.0252579999506,1627.445946199441,1797.3359449025884,137.89601687177677,139.47895800083643,232.75966649744078,255.23558505970868,7.9109000767037285,7.7173600157683415,11.63812698740246,13.068145703495471,7.912022636927084,7.718454007901284,11.63953219130328,0.3119374812973321,0.5350344129088691,0.257809083166448,0.14296894781667932,1.9358124438919964
+gsm8k,dspark_k7,dspark,7,1319,168457,0.8698276010199105,111.09063547006147,0.9613329610654137,1149.4913456012362,1122.8163880005013,1690.7677308932762,1912.2929262730754,142.42033671721288,144.59181799611542,238.7138173042331,260.84847668505967,7.943716780175663,7.752916023642096,11.936723438570985,13.552413253823115,7.944849474277738,7.753948110208927,11.937846825916434,0.14634874499637113,0.5336472057746123,0.25006905994403145,0.13272720721586337,2.024441214974598
+humaneval,baseline,baseline,,164,11593,1.3248947198163699,93.65551516360473,1.0,754.6755473965566,647.226564502489,1367.1402201009187,1480.0008050996985,189.28278684156462,179.46013799883076,342.49747914982436,379.2299685735634,8.10896273002792,7.874335342868083,9.866437620078749,14.497655275340552,8.11360654911452,7.876261433408698,9.867678297241653,,,,,
+humaneval,dflash_k3,dflash,3,164,11403,0.8482418297936308,58.97866820205349,0.6297404706922488,1178.8075736159687,1028.3337114997266,2207.2047492016277,2550.0725987079936,162.98695540243034,142.10437850124435,323.0461064536938,404.1426902490639,15.05167389736459,15.00903958145443,20.248163725045735,21.988641405840404,15.056187308266047,15.011203893487512,20.267178071169052,0.0599447417200014,0.1643059490084986,0.01447906830343091,0.0010492078480747037,1.1798342251600042
+humaneval,dflash_k7,dflash,7,164,10873,0.9775628464767565,64.81122457159618,0.6920171701407963,1022.8616001402481,861.6272605004269,2083.6442281001837,2187.8588047894846,157.66290578664288,154.99908150104602,238.5054721988126,254.32537110971683,13.23754715845204,13.002521015674134,16.242605144630183,17.38454934863588,13.2414967604306,13.004417010839589,16.244994796315652,0.027065165688917075,0.17458379578246394,0.014095449500554939,0.0007769145394006659,1.1894561598224196
+humaneval,dflare_k3,dflare,3,164,11403,1.4279144513142106,99.28358834351184,1.060093344957588,700.2340810428306,654.7297254983278,1298.7420633009606,1572.5833712578244,163.544509548978,175.85190250065352,255.46592105038144,272.46247330847837,8.18464679387673,8.052460137035897,11.823310047215534,14.873981783527022,8.188209899387681,8.053541158448532,11.828740774528667,0.5862826458384184,0.7759336099585062,0.5643153526970954,0.4185989748596534,2.758847937515255
+humaneval,dflare_k7,dflare,7,164,10873,1.7259982556670077,114.43157947480107,1.2218349263778336,579.2890396647502,542.1822514981613,1124.121437451322,1337.6976848527559,153.88535212816083,160.26985099961166,249.8991386499256,257.3446220774349,7.2881835467627765,6.791492409145974,11.28842372509098,14.967644061684823,7.292224562039036,6.792956900041677,11.300806915223802,0.3260095527572731,0.7689969604863222,0.5525835866261398,0.40091185410334346,3.282066869300912
+humaneval,dspark_k3,dspark,3,164,11403,1.8838722899407694,130.98655928167435,1.3985995277785492,530.7444518413565,495.157746001496,905.6995584996912,1086.2262809240203,146.0434188045882,152.46622500490048,236.91825044916186,255.82899200926477,6.104835246645276,5.68536281673191,8.98480516036281,11.694451736160172,6.108340038664795,5.68948486237622,8.989236841355433,0.6043326693227091,0.7711653386454184,0.5893924302788844,0.4524402390438247,2.812998007968128
+humaneval,dspark_k7,dspark,7,164,10873,2.193269418244654,145.4110877108178,1.5526163884401514,455.86265411590676,409.3146830055048,852.2122731519629,1123.0342373249005,143.6314728172608,148.64660549210384,241.35335180908442,261.36555448989384,5.509904282925857,5.137632333503879,9.656399889036864,11.480592018691826,5.513652760595837,5.139975778089672,9.664061358919506,0.3497128270580728,0.7345245692405871,0.5520102105934908,0.3937460114869177,3.4479897894065092
+mbpp,baseline,baseline,,500,61467,0.9566788994866338,117.60836382948983,1.0,1045.163285173956,1051.2518215000455,1252.2429954937252,1313.4473929634257,170.204570235961,147.96629649936222,343.0072944531275,394.77327711974795,7.1850096596128905,7.149241377965513,7.642400291320152,7.887752700210706,7.1880918593224585,7.1504635433104955,7.643652335044196,,,,,
+mbpp,dflash_k3,dflash,3,500,61643,0.5463233439436157,67.3540197814326,0.5726975326268746,1830.2107265740342,1857.0996774942614,2291.14192195193,2455.683664069438,181.87522897598683,156.96376850246452,374.19471385073825,413.4018152903445,13.36878509883122,13.397268279545893,16.285712324037817,17.16560508802913,13.37178172838754,13.398526082703144,16.287178615374877,0.0651825815270455,0.18390939269673892,0.011013473930872876,0.0006248779535247023,1.1955477445811364
+mbpp,dflash_k7,dflash,7,500,61199,0.6112544218481429,74.81631872536899,0.636147942962957,1635.86529272798,1672.3464854994745,2016.2770743999317,2129.984438859737,152.66953481407472,159.74267549972865,244.47033859860304,260.42180475840723,12.100660442193382,12.01978580708575,14.406062782671242,15.092129389210932,12.10357721618688,12.02091915748017,14.40738571140975,0.028364658688842314,0.18660277646316453,0.011358258400378608,0.0005718567597412841,1.1985526108218962
+mbpp,dflare_k3,dflare,3,500,61643,0.8389172370608934,103.4267504882893,0.8794166258297648,1191.9062847682944,1208.0225355002767,1663.9658735984995,1887.62577743968,151.1433905701415,151.9770585000515,261.6679305512662,283.6371392303044,8.617537094703247,8.440096641743008,12.05171109251937,14.760888378997743,8.620249662427044,8.441339763779492,12.052845620474532,0.4696401115367576,0.6867328590415136,0.4438276913454773,0.278359784223282,2.408920334610273
+mbpp,dflare_k7,dflare,7,500,61199,0.943455812538664,115.47710454310739,0.9818783357153716,1059.8270873299334,1066.4042809985403,1511.5341901015197,1678.6425156925543,149.29563258778944,151.24238849966787,254.46768340189007,276.2093989884306,7.8246956861011725,7.491708066904327,11.13285018701698,19.920353018048132,7.827419804285574,7.492821476362439,11.13404580474156,0.24501954356323288,0.6746690203000882,0.42497793468667255,0.2500882612533098,2.71513680494263
+mbpp,dspark_k3,dspark,3,500,61643,1.1901763850957827,146.73208581291865,1.247633085225578,840.1123063238338,843.1062914969516,1144.961133646211,1246.0605967608715,133.81045833000098,131.7662695000763,238.02317654808573,263.0950807044428,5.951950751595078,5.73346276769729,8.067283917319386,14.154150555114027,5.954451005112758,5.734595043312457,8.068427130726949,0.5237361111111111,0.7102916666666667,0.510625,0.35029166666666667,2.5712083333333333
+mbpp,dspark_k7,dspark,7,500,61199,1.292868654843097,158.2445376154854,1.3455211216517766,773.3765653301089,777.3538839974208,1069.8221424441717,1181.5079208668612,144.87902384001063,148.97060450311983,240.50168554604164,261.4171255609835,5.4609269159301945,5.171082055140708,7.696279004271422,16.670999059686896,5.463527012393008,5.172086559067843,7.697418791287318,0.2885384335766015,0.6823742415345468,0.46951458211000197,0.3062732432961441,3.0197690350362105
+mt_bench,baseline,baseline,,160,20102,0.9303956789559812,116.89258711483208,1.0,1074.6847601811623,1056.2882739977795,1319.7490790018492,1393.648768519415,160.69353549364678,136.07312599924626,327.46918089833343,382.2414645081152,7.309607731920508,7.198581275588367,8.339637732306286,9.805146286736829,7.3133077895495875,7.1998062007842805,8.34087489329106,,,,,
+mt_bench,dflash_k3,dflash,3,160,20102,0.5241432036444518,65.85204174787981,0.5633551568431672,1907.7480007938448,1930.5097770011344,2463.4897702966555,2646.199334648336,175.24922163756855,140.2629744989099,376.67113924690057,423.97999386870646,13.833307178800341,13.946147039380268,17.445409657068915,18.298039104984053,13.837227284098185,13.947568574815339,17.446681594068878,0.060423199037646175,0.16210140211796722,0.017630006507720522,0.0015381884872507838,1.1812695971129386
+mt_bench,dflash_k7,dflash,7,160,20102,0.583168838359363,73.26787492937446,0.6267965893970513,1714.6447665063306,1721.8866950006486,2135.267027699956,2242.278299470854,139.6615898878281,138.0474095003592,237.62467689830373,245.49128495971672,12.686621501440076,12.566233303133897,15.546194449997866,16.16330862709806,12.6899270927419,12.567349094479857,15.547512893692872,0.02348325183475921,0.14881334188582424,0.01463642194880168,0.0008746865706455186,1.1643827628433145
+mt_bench,dflare_k3,dflare,3,160,20102,0.6809063012234507,85.5473654199613,0.7318459410597347,1468.511645206172,1454.6153324990883,2127.0678361986943,2281.048237952054,159.72944469986032,168.77598499922897,265.74251724632626,281.86643974164326,10.654948740516124,10.288291291322736,15.705922906309695,16.95837595336085,10.6584103940507,10.289786913365493,15.707090863808236,0.3230895107183767,0.5774177712322042,0.27245949926362295,0.1193912616593029,1.9692685321551302
+mt_bench,dflare_k7,dflare,7,160,20102,0.7093667936140545,89.12307053268577,0.7624356063326214,1409.5885398314294,1421.8769274993974,2124.1017438478584,2515.584632036989,146.91454585622523,142.44566849811235,259.6700968468212,287.97775861195987,10.234430485377915,10.088829999998364,15.341923362584064,19.70016743978345,10.238094617695932,10.089903972448514,15.343091349977053,0.14626535131870344,0.5742903160861688,0.2700825447956513,0.1106301590497282,2.023857459230924
+mt_bench,dspark_k3,dspark,3,160,20102,0.9638367060855711,121.09403416082594,1.035942801418763,1037.4081412996475,1001.3265734996821,1565.5059430991967,1781.7731402603386,141.7672610684349,145.46368399896892,240.4056666531687,261.25688084066496,7.313367654694055,6.893495897653395,11.711175241387847,16.168680668779416,7.316552068142107,6.894642385820525,11.712829905159229,0.3706431026018655,0.5900483904902167,0.3288449400378708,0.19303597727750893,2.1119293078055965
+mt_bench,dspark_k7,dspark,7,160,20102,0.9729636803231215,122.24072438659618,1.045752578531864,1027.6755360433526,1009.7822754978552,1596.3315117551247,2005.884019333607,141.24648624401743,139.84237199474592,255.56160285341318,265.59654699100065,7.283154806994812,6.8463791023933425,11.787724323593851,15.836048513204252,7.286351986039958,6.847493661481403,11.788784734661597,0.1742194735272419,0.5641138336446544,0.30238435336776176,0.17075046698165036,2.2195363146906932
+ultrachat,baseline,baseline,,5000,636089,0.4938431604479032,62.825640417229266,1.0,2024.8159037780163,1899.8980629985454,3256.729267303672,3405.7137782224895,166.95593686518114,163.98859200126026,289.8830948040995,336.2988253685762,14.712897944777563,13.827794673245338,23.864679124793426,24.76342741740255,14.714261969753975,13.829047236223541,23.866155100384233,,,,,
+ultrachat,dflash_k3,dflash,3,5000,635846,0.4093959318736204,52.06255313962281,0.82868320631309,2442.487533897996,2383.8188690006064,3377.258205899534,3705.6293327211833,168.35537181059328,170.54413749974628,252.9078775001381,299.23500963023804,18.0134018722681,17.54899170471861,24.705050003148795,27.191358835904875,18.01468435680956,17.550207216533355,24.706307322046488,0.07081888392688648,0.18361221407447353,0.02588322896495871,0.002961208741227203,1.2124566517806594
+ultrachat,dflash_k7,dflash,7,5000,636040,0.4019325528965641,51.12903618886613,0.8138243533900298,2487.8614642295834,2432.8269440011354,3443.123125650527,3724.6332480803176,172.7844437341555,173.04360800153518,258.7694342015311,299.8475944703024,18.332019165246155,17.85771458268332,25.186547632286487,27.352591655754214,18.33325328096626,17.85883874804062,25.18777078149849,0.03064899424451109,0.1849223980874338,0.025778901811864367,0.0031517369145264325,1.2145429597115776
+ultrachat,dflare_k3,dflare,3,5000,635846,0.4906630866625573,62.397232200408084,0.9931809972174402,2037.9430946188047,2007.9161120011122,2756.5027899490815,3044.030925800253,185.9597727056098,186.23321399718407,279.4342198045342,323.5009970532701,14.673263611031553,14.47747526376982,19.803011401590926,21.86518714472933,14.674543237540032,14.478983830686477,19.804314228371027,0.3113198824188073,0.572229825914399,0.26063458484348756,0.10109523649853532,1.933959647256422
+ultrachat,dflare_k7,dflare,7,5000,636040,0.5191022628219358,66.03396064905282,1.0510670517724432,1925.9773102255406,1903.7692794991017,2635.482599895113,2962.525904481372,180.70931879136043,182.9802385000221,272.8241580512986,313.5787184005314,13.822185940257995,13.62802470079417,18.901379390565385,21.488975903824134,13.823438931487352,13.629216417345067,18.902619697261734,0.14394009500546442,0.5769995228510126,0.26659988687461095,0.10386364282712356,2.007580665038251
+ultrachat,dspark_k3,dspark,3,5000,635846,0.6141321767148702,78.09869760708868,1.2431022921283417,1628.2056611145947,1592.831803496665,2287.4158730010095,2571.3276440004856,171.98177616532774,172.63183150498662,258.64595805287536,302.3406934656669,11.538623067112681,11.238307917321478,16.24375355278012,18.46682838556656,11.5398444087042,11.239705511848223,16.245154050035826,0.3240773825244086,0.5557454318356237,0.28176742365161084,0.13471929208599118,1.9722321475732256
+ultrachat,dspark_k7,dspark,7,5000,636040,0.6186701811110291,78.69979639877178,1.2526700225595981,1616.259658270469,1584.57970800373,2290.956548342365,2594.4233495666413,176.71817260388633,176.76159800612368,263.7752173024638,303.1083872308955,11.399967330431618,11.174435653507444,16.282335170467917,18.624601763757692,11.401195973910607,11.175635023563679,16.283561415453793,0.15448514765416219,0.5533425372792525,0.28025688169439844,0.13306681148704796,2.0813960335791357

benchmarks/large/bs1/comparison.md

@@ -0,0 +1,44 @@
+| Dataset | Run | Req | Output tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | TPOT mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| math500 | baseline | 500 | 113.45 | 1.000x | 1121.85/1314.59 | 169.94/335.47 | 7.54/8.68 | 7.54/8.69 | — | -/-/- | — |
+| math500 | dflash_k3 | 500 | 64.68 | 0.570x | 1970.50/2394.33 | 176.98/348.85 | 14.18/17.33 | 14.18/17.33 | 6.32% | 16.82%/1.89%/0.26% | 1.190 |
+| math500 | dflash_k7 | 500 | 74.16 | 0.654x | 1716.92/2122.72 | 138.95/232.84 | 12.49/15.42 | 12.49/15.42 | 2.73% | 16.89%/1.92%/0.26% | 1.191 |
+| math500 | dflare_k3 | 500 | 93.74 | 0.826x | 1359.56/1919.75 | 150.99/254.30 | 9.56/13.64 | 9.56/13.64 | 39.18% | 64.52%/35.08%/17.94% | 2.175 |
+| math500 | dflare_k7 | 500 | 97.45 | 0.859x | 1306.61/1831.94 | 149.97/241.51 | 9.16/13.16 | 9.17/13.16 | 18.55% | 64.82%/35.30%/17.18% | 2.298 |
+| math500 | dspark_k3 | 500 | 131.01 | 1.155x | 972.82/1383.58 | 139.16/243.83 | 6.59/9.67 | 6.59/9.67 | 44.80% | 65.41%/42.19%/26.82% | 2.344 |
+| math500 | dspark_k7 | 500 | 136.48 | 1.203x | 932.87/1415.98 | 142.46/239.27 | 6.26/10.08 | 6.26/10.08 | 22.84% | 64.00%/39.61%/24.04% | 2.599 |
+| gsm8k | baseline | 1319 | 115.56 | 1.000x | 1105.36/1276.34 | 164.78/332.90 | 7.42/7.93 | 7.42/7.93 | — | -/-/- | — |
+| gsm8k | dflash_k3 | 1319 | 65.68 | 0.568x | 1943.86/2365.31 | 178.53/355.62 | 13.93/16.93 | 13.93/16.93 | 6.84% | 18.35%/1.95%/0.21% | 1.205 |
+| gsm8k | dflash_k7 | 1319 | 73.26 | 0.634x | 1743.27/2099.80 | 152.56/246.42 | 12.55/15.04 | 12.55/15.04 | 2.75% | 17.19%/1.83%/0.20% | 1.192 |
+| gsm8k | dflare_k3 | 1319 | 80.95 | 0.701x | 1577.24/2166.57 | 150.18/254.45 | 11.26/15.61 | 11.26/15.61 | 27.89% | 53.39%/21.58%/8.69% | 1.837 |
+| gsm8k | dflare_k7 | 1319 | 81.92 | 0.709x | 1558.94/2191.44 | 152.78/255.01 | 11.09/15.86 | 11.09/15.86 | 12.80% | 54.17%/21.95%/9.32% | 1.896 |
+| gsm8k | dspark_k3 | 1319 | 111.95 | 0.969x | 1140.44/1627.45 | 137.90/232.76 | 7.91/11.64 | 7.91/11.64 | 31.19% | 53.50%/25.78%/14.30% | 1.936 |
+| gsm8k | dspark_k7 | 1319 | 111.09 | 0.961x | 1149.49/1690.77 | 142.42/238.71 | 7.94/11.94 | 7.94/11.94 | 14.63% | 53.36%/25.01%/13.27% | 2.024 |
+| humaneval | baseline | 164 | 93.66 | 1.000x | 754.68/1367.14 | 189.28/342.50 | 8.11/9.87 | 8.11/9.87 | — | -/-/- | — |
+| humaneval | dflash_k3 | 164 | 58.98 | 0.630x | 1178.81/2207.20 | 162.99/323.05 | 15.05/20.25 | 15.06/20.27 | 5.99% | 16.43%/1.45%/0.10% | 1.180 |
+| humaneval | dflash_k7 | 164 | 64.81 | 0.692x | 1022.86/2083.64 | 157.66/238.51 | 13.24/16.24 | 13.24/16.24 | 2.71% | 17.46%/1.41%/0.08% | 1.189 |
+| humaneval | dflare_k3 | 164 | 99.28 | 1.060x | 700.23/1298.74 | 163.54/255.47 | 8.18/11.82 | 8.19/11.83 | 58.63% | 77.59%/56.43%/41.86% | 2.759 |
+| humaneval | dflare_k7 | 164 | 114.43 | 1.222x | 579.29/1124.12 | 153.89/249.90 | 7.29/11.29 | 7.29/11.30 | 32.60% | 76.90%/55.26%/40.09% | 3.282 |
+| humaneval | dspark_k3 | 164 | 130.99 | 1.399x | 530.74/905.70 | 146.04/236.92 | 6.10/8.98 | 6.11/8.99 | 60.43% | 77.12%/58.94%/45.24% | 2.813 |
+| humaneval | dspark_k7 | 164 | 145.41 | 1.553x | 455.86/852.21 | 143.63/241.35 | 5.51/9.66 | 5.51/9.66 | 34.97% | 73.45%/55.20%/39.37% | 3.448 |
+| mbpp | baseline | 500 | 117.61 | 1.000x | 1045.16/1252.24 | 170.20/343.01 | 7.19/7.64 | 7.19/7.64 | — | -/-/- | — |
+| mbpp | dflash_k3 | 500 | 67.35 | 0.573x | 1830.21/2291.14 | 181.88/374.19 | 13.37/16.29 | 13.37/16.29 | 6.52% | 18.39%/1.10%/0.06% | 1.196 |
+| mbpp | dflash_k7 | 500 | 74.82 | 0.636x | 1635.87/2016.28 | 152.67/244.47 | 12.10/14.41 | 12.10/14.41 | 2.84% | 18.66%/1.14%/0.06% | 1.199 |
+| mbpp | dflare_k3 | 500 | 103.43 | 0.879x | 1191.91/1663.97 | 151.14/261.67 | 8.62/12.05 | 8.62/12.05 | 46.96% | 68.67%/44.38%/27.84% | 2.409 |
+| mbpp | dflare_k7 | 500 | 115.48 | 0.982x | 1059.83/1511.53 | 149.30/254.47 | 7.82/11.13 | 7.83/11.13 | 24.50% | 67.47%/42.50%/25.01% | 2.715 |
+| mbpp | dspark_k3 | 500 | 146.73 | 1.248x | 840.11/1144.96 | 133.81/238.02 | 5.95/8.07 | 5.95/8.07 | 52.37% | 71.03%/51.06%/35.03% | 2.571 |
+| mbpp | dspark_k7 | 500 | 158.24 | 1.346x | 773.38/1069.82 | 144.88/240.50 | 5.46/7.70 | 5.46/7.70 | 28.85% | 68.24%/46.95%/30.63% | 3.020 |
+| mt_bench | baseline | 160 | 116.89 | 1.000x | 1074.68/1319.75 | 160.69/327.47 | 7.31/8.34 | 7.31/8.34 | — | -/-/- | — |
+| mt_bench | dflash_k3 | 160 | 65.85 | 0.563x | 1907.75/2463.49 | 175.25/376.67 | 13.83/17.45 | 13.84/17.45 | 6.04% | 16.21%/1.76%/0.15% | 1.181 |
+| mt_bench | dflash_k7 | 160 | 73.27 | 0.627x | 1714.64/2135.27 | 139.66/237.62 | 12.69/15.55 | 12.69/15.55 | 2.35% | 14.88%/1.46%/0.09% | 1.164 |
+| mt_bench | dflare_k3 | 160 | 85.55 | 0.732x | 1468.51/2127.07 | 159.73/265.74 | 10.65/15.71 | 10.66/15.71 | 32.31% | 57.74%/27.25%/11.94% | 1.969 |
+| mt_bench | dflare_k7 | 160 | 89.12 | 0.762x | 1409.59/2124.10 | 146.91/259.67 | 10.23/15.34 | 10.24/15.34 | 14.63% | 57.43%/27.01%/11.06% | 2.024 |
+| mt_bench | dspark_k3 | 160 | 121.09 | 1.036x | 1037.41/1565.51 | 141.77/240.41 | 7.31/11.71 | 7.32/11.71 | 37.06% | 59.00%/32.88%/19.30% | 2.112 |
+| mt_bench | dspark_k7 | 160 | 122.24 | 1.046x | 1027.68/1596.33 | 141.25/255.56 | 7.28/11.79 | 7.29/11.79 | 17.42% | 56.41%/30.24%/17.08% | 2.220 |
+| ultrachat | baseline | 5000 | 62.83 | 1.000x | 2024.82/3256.73 | 166.96/289.88 | 14.71/23.86 | 14.71/23.87 | — | -/-/- | — |
+| ultrachat | dflash_k3 | 5000 | 52.06 | 0.829x | 2442.49/3377.26 | 168.36/252.91 | 18.01/24.71 | 18.01/24.71 | 7.08% | 18.36%/2.59%/0.30% | 1.212 |
+| ultrachat | dflash_k7 | 5000 | 51.13 | 0.814x | 2487.86/3443.12 | 172.78/258.77 | 18.33/25.19 | 18.33/25.19 | 3.06% | 18.49%/2.58%/0.32% | 1.215 |
+| ultrachat | dflare_k3 | 5000 | 62.40 | 0.993x | 2037.94/2756.50 | 185.96/279.43 | 14.67/19.80 | 14.67/19.80 | 31.13% | 57.22%/26.06%/10.11% | 1.934 |
+| ultrachat | dflare_k7 | 5000 | 66.03 | 1.051x | 1925.98/2635.48 | 180.71/272.82 | 13.82/18.90 | 13.82/18.90 | 14.39% | 57.70%/26.66%/10.39% | 2.008 |
+| ultrachat | dspark_k3 | 5000 | 78.10 | 1.243x | 1628.21/2287.42 | 171.98/258.65 | 11.54/16.24 | 11.54/16.25 | 32.41% | 55.57%/28.18%/13.47% | 1.972 |
+| ultrachat | dspark_k7 | 5000 | 78.70 | 1.253x | 1616.26/2290.96 | 176.72/263.78 | 11.40/16.28 | 11.40/16.28 | 15.45% | 55.33%/28.03%/13.31% | 2.081 |

benchmarks/large/bs1/results.json

@@ -0,0 +1,1304 @@
+[
+  {
+    "dataset": "math500",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 63642,
+    "request_throughput_rps": 0.8912881111809688,
+    "output_throughput_tps": 113.44671594355843,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1121.8503840199992,
+    "latency_p50_ms": 1100.7910255029856,
+    "latency_p95_ms": 1314.5946230510162,
+    "latency_p99_ms": 1484.1045933025453,
+    "ttft_mean_ms": 169.94078358601837,
+    "ttft_p50_ms": 146.2457435009128,
+    "ttft_p95_ms": 335.47060909950216,
+    "ttft_p99_ms": 373.49712165247183,
+    "itl_mean_ms": 7.536181297069743,
+    "itl_p50_ms": 7.382753629891218,
+    "itl_p95_ms": 8.68439770515185,
+    "itl_p99_ms": 10.087323529295983,
+    "tpot_mean_ms": 7.537555813759378,
+    "tpot_p50_ms": 7.383992137781786,
+    "tpot_p95_ms": 8.685839803156412,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 0.5074555001107937,
+    "output_throughput_tps": 64.68129295512199,
+    "speedup_vs_baseline": 0.5701468959868523,
+    "latency_mean_ms": 1970.4961424658832,
+    "latency_p50_ms": 1974.9981015011144,
+    "latency_p95_ms": 2394.326527546944,
+    "latency_p99_ms": 2618.456352452631,
+    "ttft_mean_ms": 176.97923488401284,
+    "ttft_p50_ms": 151.66605199920014,
+    "ttft_p95_ms": 348.84992924853566,
+    "ttft_p99_ms": 401.735668064066,
+    "itl_mean_ms": 14.179071923655801,
+    "itl_p50_ms": 14.188702799193972,
+    "itl_p95_ms": 17.33207021927915,
+    "itl_p99_ms": 18.87767345473462,
+    "tpot_mean_ms": 14.180381967095736,
+    "tpot_p50_ms": 14.189951897641887,
+    "tpot_p95_ms": 17.33332515668878,
+    "acceptance_rate": 0.06323912607737021,
+    "acceptance_pos_1": 0.16818250150330727,
+    "acceptance_pos_2": 0.01894167167769092,
+    "acceptance_pos_3": 0.0025932050511124475,
+    "mean_acceptance_length": 1.1897173782321107
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 0.5823973894271881,
+    "output_throughput_tps": 74.16015398010043,
+    "speedup_vs_baseline": 0.6537003152827826,
+    "latency_mean_ms": 1716.9249164580651,
+    "latency_p50_ms": 1721.4581165007985,
+    "latency_p95_ms": 2122.7248207490447,
+    "latency_p99_ms": 2282.573311009855,
+    "ttft_mean_ms": 138.95278630002576,
+    "ttft_p50_ms": 141.47450450036558,
+    "ttft_p95_ms": 232.83745254921087,
+    "ttft_p99_ms": 255.96303990992965,
+    "itl_mean_ms": 12.488242557152846,
+    "itl_p50_ms": 12.48854895669593,
+    "itl_p95_ms": 15.423414866535039,
+    "itl_p99_ms": 16.76060266095041,
+    "tpot_mean_ms": 12.489462153205425,
+    "tpot_p50_ms": 12.489741562990455,
+    "tpot_p95_ms": 15.424737617707956,
+    "acceptance_rate": 0.027334900964792862,
+    "acceptance_pos_1": 0.16893291649402992,
+    "acceptance_pos_2": 0.019209763079588684,
+    "acceptance_pos_3": 0.002561301743945158,
+    "mean_acceptance_length": 1.19134430675355
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 0.7354717914893614,
+    "output_throughput_tps": 93.74470548681697,
+    "speedup_vs_baseline": 0.8263324742997098,
+    "latency_mean_ms": 1359.560892554051,
+    "latency_p50_ms": 1336.2468844989053,
+    "latency_p95_ms": 1919.7459352993976,
+    "latency_p99_ms": 2183.3393649194472,
+    "ttft_mean_ms": 150.9933998281267,
+    "ttft_p50_ms": 150.4645309996704,
+    "ttft_p95_ms": 254.30402324909662,
+    "ttft_p99_ms": 281.55977436035755,
+    "itl_mean_ms": 9.55804924402514,
+    "itl_p50_ms": 9.37427223622698,
+    "itl_p95_ms": 13.641124985851773,
+    "itl_p99_ms": 15.580885431589866,
+    "tpot_mean_ms": 9.559225141953267,
+    "tpot_p50_ms": 9.37540769291087,
+    "tpot_p95_ms": 13.642259077954161,
+    "acceptance_rate": 0.39181146804128403,
+    "acceptance_pos_1": 0.6452053391595262,
+    "acceptance_pos_2": 0.35083467039907146,
+    "acceptance_pos_3": 0.1793943945652545,
+    "mean_acceptance_length": 2.175434404123852
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 0.7652758346384362,
+    "output_throughput_tps": 97.4471636795199,
+    "speedup_vs_baseline": 0.8589685727703342,
+    "latency_mean_ms": 1306.6064661140117,
+    "latency_p50_ms": 1288.1172004999826,
+    "latency_p95_ms": 1831.9392699944729,
+    "latency_p99_ms": 2117.9882052937314,
+    "ttft_mean_ms": 149.97463030007202,
+    "ttft_p50_ms": 150.71226700456464,
+    "ttft_p95_ms": 241.5094937001413,
+    "ttft_p99_ms": 269.5821494147821,
+    "itl_mean_ms": 9.164062260873436,
+    "itl_p50_ms": 9.109615248048863,
+    "itl_p95_ms": 13.162089144880586,
+    "itl_p99_ms": 16.025888632277603,
+    "tpot_mean_ms": 9.165267968224493,
+    "tpot_p50_ms": 9.110725452762747,
+    "tpot_p95_ms": 13.163240382294413,
+    "acceptance_rate": 0.18547955359716675,
+    "acceptance_pos_1": 0.6482415681752667,
+    "acceptance_pos_2": 0.3530196021908331,
+    "acceptance_pos_3": 0.17180743730181608,
+    "mean_acceptance_length": 2.2983568751801675
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63731,
+    "request_throughput_rps": 1.0278341396966633,
+    "output_throughput_tps": 131.0097951140161,
+    "speedup_vs_baseline": 1.1548134648445494,
+    "latency_mean_ms": 972.8192880541756,
+    "latency_p50_ms": 951.3454184962029,
+    "latency_p95_ms": 1383.5792890022267,
+    "latency_p99_ms": 1658.223661600423,
+    "ttft_mean_ms": 139.16257599797973,
+    "ttft_p50_ms": 140.1768340037961,
+    "ttft_p95_ms": 243.82559089899584,
+    "ttft_p99_ms": 256.6295453281782,
+    "itl_mean_ms": 6.592793092238339,
+    "itl_p50_ms": 6.475244685019803,
+    "itl_p95_ms": 9.666808936191497,
+    "itl_p99_ms": 11.624004104496413,
+    "tpot_mean_ms": 6.593887856670312,
+    "tpot_p50_ms": 6.476409696829716,
+    "tpot_p95_ms": 9.667887905099029,
+    "acceptance_rate": 0.44804447383243945,
+    "acceptance_pos_1": 0.6541032988024392,
+    "acceptance_pos_2": 0.4218646682830064,
+    "acceptance_pos_3": 0.26816545441187273,
+    "mean_acceptance_length": 2.344133421497318
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63668,
+    "request_throughput_rps": 1.0718413712400758,
+    "output_throughput_tps": 136.4839928482263,
+    "speedup_vs_baseline": 1.2030669351074852,
+    "latency_mean_ms": 932.8721254997654,
+    "latency_p50_ms": 924.5265510035097,
+    "latency_p95_ms": 1415.9799854001901,
+    "latency_p99_ms": 1651.539169616298,
+    "ttft_mean_ms": 142.46345183774247,
+    "ttft_p50_ms": 146.44532399688615,
+    "ttft_p95_ms": 239.27307030317024,
+    "ttft_p99_ms": 256.7968881978595,
+    "itl_mean_ms": 6.260414953927999,
+    "itl_p50_ms": 6.1309666968776675,
+    "itl_p95_ms": 10.081072708306051,
+    "itl_p99_ms": 12.369931453538051,
+    "tpot_mean_ms": 6.261500149276782,
+    "tpot_p50_ms": 6.13199001179062,
+    "tpot_p95_ms": 10.082133980709683,
+    "acceptance_rate": 0.2283637249725866,
+    "acceptance_pos_1": 0.6400113714819478,
+    "acceptance_pos_2": 0.3961336961377574,
+    "acceptance_pos_3": 0.24038500588880315,
+    "mean_acceptance_length": 2.598546074808106
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 1319,
+    "output_tokens": 168500,
+    "request_throughput_rps": 0.9045831802614913,
+    "output_throughput_tps": 115.55895820626328,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1105.3625852046998,
+    "latency_p50_ms": 1089.5358249981655,
+    "latency_p95_ms": 1276.3373936999415,
+    "latency_p99_ms": 1319.7392648432285,
+    "ttft_mean_ms": 164.77733511295938,
+    "ttft_p50_ms": 141.7763389981701,
+    "ttft_p95_ms": 332.90244539603,
+    "ttft_p99_ms": 382.7308631595223,
+    "itl_mean_ms": 7.419506798811708,
+    "itl_p50_ms": 7.384766740153826,
+    "itl_p95_ms": 7.931435276391204,
+    "itl_p99_ms": 8.172192777608757,
+    "tpot_mean_ms": 7.420847337932145,
+    "tpot_p50_ms": 7.386314905524496,
+    "tpot_p95_ms": 7.932668355122615,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 0.5144087930838571,
+    "output_throughput_tps": 65.6828658935535,
+    "speedup_vs_baseline": 0.568392679486734,
+    "latency_mean_ms": 1943.8572523131418,
+    "latency_p50_ms": 1934.8288309993222,
+    "latency_p95_ms": 2365.305382902443,
+    "latency_p99_ms": 2508.7120192195284,
+    "ttft_mean_ms": 178.53159811208383,
+    "ttft_p50_ms": 159.649838002224,
+    "ttft_p95_ms": 355.616971301788,
+    "ttft_p99_ms": 411.2498142791445,
+    "itl_mean_ms": 13.929846219481163,
+    "itl_p50_ms": 13.876137322875465,
+    "itl_p95_ms": 16.931091559855826,
+    "itl_p99_ms": 17.821733611478884,
+    "tpot_mean_ms": 13.931159616121034,
+    "tpot_p50_ms": 13.87738181893644,
+    "tpot_p95_ms": 16.93262100001677,
+    "acceptance_rate": 0.06835341539008165,
+    "acceptance_pos_1": 0.1835041448500868,
+    "acceptance_pos_2": 0.019489077905896417,
+    "acceptance_pos_3": 0.0020670234142617415,
+    "mean_acceptance_length": 1.2050602461702449
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.5735945671017055,
+    "output_throughput_tps": 73.25702804416376,
+    "speedup_vs_baseline": 0.6339363834814603,
+    "latency_mean_ms": 1743.2748062024025,
+    "latency_p50_ms": 1739.7538450004504,
+    "latency_p95_ms": 2099.799393000285,
+    "latency_p99_ms": 2201.536506499797,
+    "ttft_mean_ms": 152.56252339044147,
+    "ttft_p50_ms": 159.50948600038828,
+    "ttft_p95_ms": 246.41776080115957,
+    "ttft_p99_ms": 261.92414222023217,
+    "itl_mean_ms": 12.548829301304638,
+    "itl_p50_ms": 12.529087370075011,
+    "itl_p95_ms": 15.037010089759487,
+    "itl_p99_ms": 15.820388422991211,
+    "tpot_mean_ms": 12.550042645359513,
+    "tpot_p50_ms": 12.530304661416404,
+    "tpot_p95_ms": 15.038574778736802,
+    "acceptance_rate": 0.02749077227835654,
+    "acceptance_pos_1": 0.1718565178355114,
+    "acceptance_pos_2": 0.018334307172683095,
+    "acceptance_pos_3": 0.002037936980717981,
+    "mean_acceptance_length": 1.1924354059484958
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 0.6339721362775607,
+    "output_throughput_tps": 80.94944597998045,
+    "speedup_vs_baseline": 0.7005034247149607,
+    "latency_mean_ms": 1577.243275963595,
+    "latency_p50_ms": 1558.0307409982197,
+    "latency_p95_ms": 2166.574809700978,
+    "latency_p99_ms": 2443.7027309426035,
+    "ttft_mean_ms": 150.18097911596254,
+    "ttft_p50_ms": 151.24057199864183,
+    "ttft_p95_ms": 254.45422550219513,
+    "ttft_p99_ms": 278.5744088418869,
+    "itl_mean_ms": 11.259462841455447,
+    "itl_p50_ms": 11.12902188976914,
+    "itl_p95_ms": 15.608968344093311,
+    "itl_p99_ms": 17.811628303623607,
+    "tpot_mean_ms": 11.260660320716259,
+    "tpot_p50_ms": 11.130129110223226,
+    "tpot_p95_ms": 15.610202330715275,
+    "acceptance_rate": 0.2788578880838151,
+    "acceptance_pos_1": 0.5339278493601303,
+    "acceptance_pos_2": 0.21577434619629957,
+    "acceptance_pos_3": 0.08687146869501547,
+    "mean_acceptance_length": 1.8365736642514454
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.6414136486780062,
+    "output_throughput_tps": 81.91858909427664,
+    "speedup_vs_baseline": 0.7088899931761125,
+    "latency_mean_ms": 1558.9424188529392,
+    "latency_p50_ms": 1528.7051639970741,
+    "latency_p95_ms": 2191.441884898086,
+    "latency_p99_ms": 2472.421863719936,
+    "ttft_mean_ms": 152.77644680597203,
+    "ttft_p50_ms": 155.94628499820828,
+    "ttft_p95_ms": 255.01179830098403,
+    "ttft_p99_ms": 283.681260881276,
+    "itl_mean_ms": 11.093297977502198,
+    "itl_p50_ms": 10.81332909192518,
+    "itl_p95_ms": 15.859892152744344,
+    "itl_p99_ms": 18.10012607823774,
+    "tpot_mean_ms": 11.094531792060318,
+    "tpot_p50_ms": 10.814917770089934,
+    "tpot_p95_ms": 15.861106548793142,
+    "acceptance_rate": 0.12798578221940327,
+    "acceptance_pos_1": 0.5417164364111695,
+    "acceptance_pos_2": 0.21953528205359357,
+    "acceptance_pos_3": 0.0932478420590025,
+    "mean_acceptance_length": 1.8959004755358229
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168418,
+    "request_throughput_rps": 0.8767773042557421,
+    "output_throughput_tps": 111.95229721618163,
+    "speedup_vs_baseline": 0.9687894296896996,
+    "latency_mean_ms": 1140.4352353411668,
+    "latency_p50_ms": 1110.0252579999506,
+    "latency_p95_ms": 1627.445946199441,
+    "latency_p99_ms": 1797.3359449025884,
+    "ttft_mean_ms": 137.89601687177677,
+    "ttft_p50_ms": 139.47895800083643,
+    "ttft_p95_ms": 232.75966649744078,
+    "ttft_p99_ms": 255.23558505970868,
+    "itl_mean_ms": 7.9109000767037285,
+    "itl_p50_ms": 7.7173600157683415,
+    "itl_p95_ms": 11.63812698740246,
+    "itl_p99_ms": 13.068145703495471,
+    "tpot_mean_ms": 7.912022636927084,
+    "tpot_p50_ms": 7.718454007901284,
+    "tpot_p95_ms": 11.63953219130328,
+    "acceptance_rate": 0.3119374812973321,
+    "acceptance_pos_1": 0.5350344129088691,
+    "acceptance_pos_2": 0.257809083166448,
+    "acceptance_pos_3": 0.14296894781667932,
+    "mean_acceptance_length": 1.9358124438919964
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168457,
+    "request_throughput_rps": 0.8698276010199105,
+    "output_throughput_tps": 111.09063547006147,
+    "speedup_vs_baseline": 0.9613329610654137,
+    "latency_mean_ms": 1149.4913456012362,
+    "latency_p50_ms": 1122.8163880005013,
+    "latency_p95_ms": 1690.7677308932762,
+    "latency_p99_ms": 1912.2929262730754,
+    "ttft_mean_ms": 142.42033671721288,
+    "ttft_p50_ms": 144.59181799611542,
+    "ttft_p95_ms": 238.7138173042331,
+    "ttft_p99_ms": 260.84847668505967,
+    "itl_mean_ms": 7.943716780175663,
+    "itl_p50_ms": 7.752916023642096,
+    "itl_p95_ms": 11.936723438570985,
+    "itl_p99_ms": 13.552413253823115,
+    "tpot_mean_ms": 7.944849474277738,
+    "tpot_p50_ms": 7.753948110208927,
+    "tpot_p95_ms": 11.937846825916434,
+    "acceptance_rate": 0.14634874499637113,
+    "acceptance_pos_1": 0.5336472057746123,
+    "acceptance_pos_2": 0.25006905994403145,
+    "acceptance_pos_3": 0.13272720721586337,
+    "mean_acceptance_length": 2.024441214974598
+  },
+  {
+    "dataset": "humaneval",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 164,
+    "output_tokens": 11593,
+    "request_throughput_rps": 1.3248947198163699,
+    "output_throughput_tps": 93.65551516360473,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 754.6755473965566,
+    "latency_p50_ms": 647.226564502489,
+    "latency_p95_ms": 1367.1402201009187,
+    "latency_p99_ms": 1480.0008050996985,
+    "ttft_mean_ms": 189.28278684156462,
+    "ttft_p50_ms": 179.46013799883076,
+    "ttft_p95_ms": 342.49747914982436,
+    "ttft_p99_ms": 379.2299685735634,
+    "itl_mean_ms": 8.10896273002792,
+    "itl_p50_ms": 7.874335342868083,
+    "itl_p95_ms": 9.866437620078749,
+    "itl_p99_ms": 14.497655275340552,
+    "tpot_mean_ms": 8.11360654911452,
+    "tpot_p50_ms": 7.876261433408698,
+    "tpot_p95_ms": 9.867678297241653,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 0.8482418297936308,
+    "output_throughput_tps": 58.97866820205349,
+    "speedup_vs_baseline": 0.6297404706922488,
+    "latency_mean_ms": 1178.8075736159687,
+    "latency_p50_ms": 1028.3337114997266,
+    "latency_p95_ms": 2207.2047492016277,
+    "latency_p99_ms": 2550.0725987079936,
+    "ttft_mean_ms": 162.98695540243034,
+    "ttft_p50_ms": 142.10437850124435,
+    "ttft_p95_ms": 323.0461064536938,
+    "ttft_p99_ms": 404.1426902490639,
+    "itl_mean_ms": 15.05167389736459,
+    "itl_p50_ms": 15.00903958145443,
+    "itl_p95_ms": 20.248163725045735,
+    "itl_p99_ms": 21.988641405840404,
+    "tpot_mean_ms": 15.056187308266047,
+    "tpot_p50_ms": 15.011203893487512,
+    "tpot_p95_ms": 20.267178071169052,
+    "acceptance_rate": 0.0599447417200014,
+    "acceptance_pos_1": 0.1643059490084986,
+    "acceptance_pos_2": 0.01447906830343091,
+    "acceptance_pos_3": 0.0010492078480747037,
+    "mean_acceptance_length": 1.1798342251600042
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 0.9775628464767565,
+    "output_throughput_tps": 64.81122457159618,
+    "speedup_vs_baseline": 0.6920171701407963,
+    "latency_mean_ms": 1022.8616001402481,
+    "latency_p50_ms": 861.6272605004269,
+    "latency_p95_ms": 2083.6442281001837,
+    "latency_p99_ms": 2187.8588047894846,
+    "ttft_mean_ms": 157.66290578664288,
+    "ttft_p50_ms": 154.99908150104602,
+    "ttft_p95_ms": 238.5054721988126,
+    "ttft_p99_ms": 254.32537110971683,
+    "itl_mean_ms": 13.23754715845204,
+    "itl_p50_ms": 13.002521015674134,
+    "itl_p95_ms": 16.242605144630183,
+    "itl_p99_ms": 17.38454934863588,
+    "tpot_mean_ms": 13.2414967604306,
+    "tpot_p50_ms": 13.004417010839589,
+    "tpot_p95_ms": 16.244994796315652,
+    "acceptance_rate": 0.027065165688917075,
+    "acceptance_pos_1": 0.17458379578246394,
+    "acceptance_pos_2": 0.014095449500554939,
+    "acceptance_pos_3": 0.0007769145394006659,
+    "mean_acceptance_length": 1.1894561598224196
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 1.4279144513142106,
+    "output_throughput_tps": 99.28358834351184,
+    "speedup_vs_baseline": 1.060093344957588,
+    "latency_mean_ms": 700.2340810428306,
+    "latency_p50_ms": 654.7297254983278,
+    "latency_p95_ms": 1298.7420633009606,
+    "latency_p99_ms": 1572.5833712578244,
+    "ttft_mean_ms": 163.544509548978,
+    "ttft_p50_ms": 175.85190250065352,
+    "ttft_p95_ms": 255.46592105038144,
+    "ttft_p99_ms": 272.46247330847837,
+    "itl_mean_ms": 8.18464679387673,
+    "itl_p50_ms": 8.052460137035897,
+    "itl_p95_ms": 11.823310047215534,
+    "itl_p99_ms": 14.873981783527022,
+    "tpot_mean_ms": 8.188209899387681,
+    "tpot_p50_ms": 8.053541158448532,
+    "tpot_p95_ms": 11.828740774528667,
+    "acceptance_rate": 0.5862826458384184,
+    "acceptance_pos_1": 0.7759336099585062,
+    "acceptance_pos_2": 0.5643153526970954,
+    "acceptance_pos_3": 0.4185989748596534,
+    "mean_acceptance_length": 2.758847937515255
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 1.7259982556670077,
+    "output_throughput_tps": 114.43157947480107,
+    "speedup_vs_baseline": 1.2218349263778336,
+    "latency_mean_ms": 579.2890396647502,
+    "latency_p50_ms": 542.1822514981613,
+    "latency_p95_ms": 1124.121437451322,
+    "latency_p99_ms": 1337.6976848527559,
+    "ttft_mean_ms": 153.88535212816083,
+    "ttft_p50_ms": 160.26985099961166,
+    "ttft_p95_ms": 249.8991386499256,
+    "ttft_p99_ms": 257.3446220774349,
+    "itl_mean_ms": 7.2881835467627765,
+    "itl_p50_ms": 6.791492409145974,
+    "itl_p95_ms": 11.28842372509098,
+    "itl_p99_ms": 14.967644061684823,
+    "tpot_mean_ms": 7.292224562039036,
+    "tpot_p50_ms": 6.792956900041677,
+    "tpot_p95_ms": 11.300806915223802,
+    "acceptance_rate": 0.3260095527572731,
+    "acceptance_pos_1": 0.7689969604863222,
+    "acceptance_pos_2": 0.5525835866261398,
+    "acceptance_pos_3": 0.40091185410334346,
+    "mean_acceptance_length": 3.282066869300912
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11403,
+    "request_throughput_rps": 1.8838722899407694,
+    "output_throughput_tps": 130.98655928167435,
+    "speedup_vs_baseline": 1.3985995277785492,
+    "latency_mean_ms": 530.7444518413565,
+    "latency_p50_ms": 495.157746001496,
+    "latency_p95_ms": 905.6995584996912,
+    "latency_p99_ms": 1086.2262809240203,
+    "ttft_mean_ms": 146.0434188045882,
+    "ttft_p50_ms": 152.46622500490048,
+    "ttft_p95_ms": 236.91825044916186,
+    "ttft_p99_ms": 255.82899200926477,
+    "itl_mean_ms": 6.104835246645276,
+    "itl_p50_ms": 5.68536281673191,
+    "itl_p95_ms": 8.98480516036281,
+    "itl_p99_ms": 11.694451736160172,
+    "tpot_mean_ms": 6.108340038664795,
+    "tpot_p50_ms": 5.68948486237622,
+    "tpot_p95_ms": 8.989236841355433,
+    "acceptance_rate": 0.6043326693227091,
+    "acceptance_pos_1": 0.7711653386454184,
+    "acceptance_pos_2": 0.5893924302788844,
+    "acceptance_pos_3": 0.4524402390438247,
+    "mean_acceptance_length": 2.812998007968128
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 10873,
+    "request_throughput_rps": 2.193269418244654,
+    "output_throughput_tps": 145.4110877108178,
+    "speedup_vs_baseline": 1.5526163884401514,
+    "latency_mean_ms": 455.86265411590676,
+    "latency_p50_ms": 409.3146830055048,
+    "latency_p95_ms": 852.2122731519629,
+    "latency_p99_ms": 1123.0342373249005,
+    "ttft_mean_ms": 143.6314728172608,
+    "ttft_p50_ms": 148.64660549210384,
+    "ttft_p95_ms": 241.35335180908442,
+    "ttft_p99_ms": 261.36555448989384,
+    "itl_mean_ms": 5.509904282925857,
+    "itl_p50_ms": 5.137632333503879,
+    "itl_p95_ms": 9.656399889036864,
+    "itl_p99_ms": 11.480592018691826,
+    "tpot_mean_ms": 5.513652760595837,
+    "tpot_p50_ms": 5.139975778089672,
+    "tpot_p95_ms": 9.664061358919506,
+    "acceptance_rate": 0.3497128270580728,
+    "acceptance_pos_1": 0.7345245692405871,
+    "acceptance_pos_2": 0.5520102105934908,
+    "acceptance_pos_3": 0.3937460114869177,
+    "mean_acceptance_length": 3.4479897894065092
+  },
+  {
+    "dataset": "mbpp",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 61467,
+    "request_throughput_rps": 0.9566788994866338,
+    "output_throughput_tps": 117.60836382948983,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1045.163285173956,
+    "latency_p50_ms": 1051.2518215000455,
+    "latency_p95_ms": 1252.2429954937252,
+    "latency_p99_ms": 1313.4473929634257,
+    "ttft_mean_ms": 170.204570235961,
+    "ttft_p50_ms": 147.96629649936222,
+    "ttft_p95_ms": 343.0072944531275,
+    "ttft_p99_ms": 394.77327711974795,
+    "itl_mean_ms": 7.1850096596128905,
+    "itl_p50_ms": 7.149241377965513,
+    "itl_p95_ms": 7.642400291320152,
+    "itl_p99_ms": 7.887752700210706,
+    "tpot_mean_ms": 7.1880918593224585,
+    "tpot_p50_ms": 7.1504635433104955,
+    "tpot_p95_ms": 7.643652335044196,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 0.5463233439436157,
+    "output_throughput_tps": 67.3540197814326,
+    "speedup_vs_baseline": 0.5726975326268746,
+    "latency_mean_ms": 1830.2107265740342,
+    "latency_p50_ms": 1857.0996774942614,
+    "latency_p95_ms": 2291.14192195193,
+    "latency_p99_ms": 2455.683664069438,
+    "ttft_mean_ms": 181.87522897598683,
+    "ttft_p50_ms": 156.96376850246452,
+    "ttft_p95_ms": 374.19471385073825,
+    "ttft_p99_ms": 413.4018152903445,
+    "itl_mean_ms": 13.36878509883122,
+    "itl_p50_ms": 13.397268279545893,
+    "itl_p95_ms": 16.285712324037817,
+    "itl_p99_ms": 17.16560508802913,
+    "tpot_mean_ms": 13.37178172838754,
+    "tpot_p50_ms": 13.398526082703144,
+    "tpot_p95_ms": 16.287178615374877,
+    "acceptance_rate": 0.0651825815270455,
+    "acceptance_pos_1": 0.18390939269673892,
+    "acceptance_pos_2": 0.011013473930872876,
+    "acceptance_pos_3": 0.0006248779535247023,
+    "mean_acceptance_length": 1.1955477445811364
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 0.6112544218481429,
+    "output_throughput_tps": 74.81631872536899,
+    "speedup_vs_baseline": 0.636147942962957,
+    "latency_mean_ms": 1635.86529272798,
+    "latency_p50_ms": 1672.3464854994745,
+    "latency_p95_ms": 2016.2770743999317,
+    "latency_p99_ms": 2129.984438859737,
+    "ttft_mean_ms": 152.66953481407472,
+    "ttft_p50_ms": 159.74267549972865,
+    "ttft_p95_ms": 244.47033859860304,
+    "ttft_p99_ms": 260.42180475840723,
+    "itl_mean_ms": 12.100660442193382,
+    "itl_p50_ms": 12.01978580708575,
+    "itl_p95_ms": 14.406062782671242,
+    "itl_p99_ms": 15.092129389210932,
+    "tpot_mean_ms": 12.10357721618688,
+    "tpot_p50_ms": 12.02091915748017,
+    "tpot_p95_ms": 14.40738571140975,
+    "acceptance_rate": 0.028364658688842314,
+    "acceptance_pos_1": 0.18660277646316453,
+    "acceptance_pos_2": 0.011358258400378608,
+    "acceptance_pos_3": 0.0005718567597412841,
+    "mean_acceptance_length": 1.1985526108218962
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 0.8389172370608934,
+    "output_throughput_tps": 103.4267504882893,
+    "speedup_vs_baseline": 0.8794166258297648,
+    "latency_mean_ms": 1191.9062847682944,
+    "latency_p50_ms": 1208.0225355002767,
+    "latency_p95_ms": 1663.9658735984995,
+    "latency_p99_ms": 1887.62577743968,
+    "ttft_mean_ms": 151.1433905701415,
+    "ttft_p50_ms": 151.9770585000515,
+    "ttft_p95_ms": 261.6679305512662,
+    "ttft_p99_ms": 283.6371392303044,
+    "itl_mean_ms": 8.617537094703247,
+    "itl_p50_ms": 8.440096641743008,
+    "itl_p95_ms": 12.05171109251937,
+    "itl_p99_ms": 14.760888378997743,
+    "tpot_mean_ms": 8.620249662427044,
+    "tpot_p50_ms": 8.441339763779492,
+    "tpot_p95_ms": 12.052845620474532,
+    "acceptance_rate": 0.4696401115367576,
+    "acceptance_pos_1": 0.6867328590415136,
+    "acceptance_pos_2": 0.4438276913454773,
+    "acceptance_pos_3": 0.278359784223282,
+    "mean_acceptance_length": 2.408920334610273
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 0.943455812538664,
+    "output_throughput_tps": 115.47710454310739,
+    "speedup_vs_baseline": 0.9818783357153716,
+    "latency_mean_ms": 1059.8270873299334,
+    "latency_p50_ms": 1066.4042809985403,
+    "latency_p95_ms": 1511.5341901015197,
+    "latency_p99_ms": 1678.6425156925543,
+    "ttft_mean_ms": 149.29563258778944,
+    "ttft_p50_ms": 151.24238849966787,
+    "ttft_p95_ms": 254.46768340189007,
+    "ttft_p99_ms": 276.2093989884306,
+    "itl_mean_ms": 7.8246956861011725,
+    "itl_p50_ms": 7.491708066904327,
+    "itl_p95_ms": 11.13285018701698,
+    "itl_p99_ms": 19.920353018048132,
+    "tpot_mean_ms": 7.827419804285574,
+    "tpot_p50_ms": 7.492821476362439,
+    "tpot_p95_ms": 11.13404580474156,
+    "acceptance_rate": 0.24501954356323288,
+    "acceptance_pos_1": 0.6746690203000882,
+    "acceptance_pos_2": 0.42497793468667255,
+    "acceptance_pos_3": 0.2500882612533098,
+    "mean_acceptance_length": 2.71513680494263
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61643,
+    "request_throughput_rps": 1.1901763850957827,
+    "output_throughput_tps": 146.73208581291865,
+    "speedup_vs_baseline": 1.247633085225578,
+    "latency_mean_ms": 840.1123063238338,
+    "latency_p50_ms": 843.1062914969516,
+    "latency_p95_ms": 1144.961133646211,
+    "latency_p99_ms": 1246.0605967608715,
+    "ttft_mean_ms": 133.81045833000098,
+    "ttft_p50_ms": 131.7662695000763,
+    "ttft_p95_ms": 238.02317654808573,
+    "ttft_p99_ms": 263.0950807044428,
+    "itl_mean_ms": 5.951950751595078,
+    "itl_p50_ms": 5.73346276769729,
+    "itl_p95_ms": 8.067283917319386,
+    "itl_p99_ms": 14.154150555114027,
+    "tpot_mean_ms": 5.954451005112758,
+    "tpot_p50_ms": 5.734595043312457,
+    "tpot_p95_ms": 8.068427130726949,
+    "acceptance_rate": 0.5237361111111111,
+    "acceptance_pos_1": 0.7102916666666667,
+    "acceptance_pos_2": 0.510625,
+    "acceptance_pos_3": 0.35029166666666667,
+    "mean_acceptance_length": 2.5712083333333333
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61199,
+    "request_throughput_rps": 1.292868654843097,
+    "output_throughput_tps": 158.2445376154854,
+    "speedup_vs_baseline": 1.3455211216517766,
+    "latency_mean_ms": 773.3765653301089,
+    "latency_p50_ms": 777.3538839974208,
+    "latency_p95_ms": 1069.8221424441717,
+    "latency_p99_ms": 1181.5079208668612,
+    "ttft_mean_ms": 144.87902384001063,
+    "ttft_p50_ms": 148.97060450311983,
+    "ttft_p95_ms": 240.50168554604164,
+    "ttft_p99_ms": 261.4171255609835,
+    "itl_mean_ms": 5.4609269159301945,
+    "itl_p50_ms": 5.171082055140708,
+    "itl_p95_ms": 7.696279004271422,
+    "itl_p99_ms": 16.670999059686896,
+    "tpot_mean_ms": 5.463527012393008,
+    "tpot_p50_ms": 5.172086559067843,
+    "tpot_p95_ms": 7.697418791287318,
+    "acceptance_rate": 0.2885384335766015,
+    "acceptance_pos_1": 0.6823742415345468,
+    "acceptance_pos_2": 0.46951458211000197,
+    "acceptance_pos_3": 0.3062732432961441,
+    "mean_acceptance_length": 3.0197690350362105
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.9303956789559812,
+    "output_throughput_tps": 116.89258711483208,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1074.6847601811623,
+    "latency_p50_ms": 1056.2882739977795,
+    "latency_p95_ms": 1319.7490790018492,
+    "latency_p99_ms": 1393.648768519415,
+    "ttft_mean_ms": 160.69353549364678,
+    "ttft_p50_ms": 136.07312599924626,
+    "ttft_p95_ms": 327.46918089833343,
+    "ttft_p99_ms": 382.2414645081152,
+    "itl_mean_ms": 7.309607731920508,
+    "itl_p50_ms": 7.198581275588367,
+    "itl_p95_ms": 8.339637732306286,
+    "itl_p99_ms": 9.805146286736829,
+    "tpot_mean_ms": 7.3133077895495875,
+    "tpot_p50_ms": 7.1998062007842805,
+    "tpot_p95_ms": 8.34087489329106,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.5241432036444518,
+    "output_throughput_tps": 65.85204174787981,
+    "speedup_vs_baseline": 0.5633551568431672,
+    "latency_mean_ms": 1907.7480007938448,
+    "latency_p50_ms": 1930.5097770011344,
+    "latency_p95_ms": 2463.4897702966555,
+    "latency_p99_ms": 2646.199334648336,
+    "ttft_mean_ms": 175.24922163756855,
+    "ttft_p50_ms": 140.2629744989099,
+    "ttft_p95_ms": 376.67113924690057,
+    "ttft_p99_ms": 423.97999386870646,
+    "itl_mean_ms": 13.833307178800341,
+    "itl_p50_ms": 13.946147039380268,
+    "itl_p95_ms": 17.445409657068915,
+    "itl_p99_ms": 18.298039104984053,
+    "tpot_mean_ms": 13.837227284098185,
+    "tpot_p50_ms": 13.947568574815339,
+    "tpot_p95_ms": 17.446681594068878,
+    "acceptance_rate": 0.060423199037646175,
+    "acceptance_pos_1": 0.16210140211796722,
+    "acceptance_pos_2": 0.017630006507720522,
+    "acceptance_pos_3": 0.0015381884872507838,
+    "mean_acceptance_length": 1.1812695971129386
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.583168838359363,
+    "output_throughput_tps": 73.26787492937446,
+    "speedup_vs_baseline": 0.6267965893970513,
+    "latency_mean_ms": 1714.6447665063306,
+    "latency_p50_ms": 1721.8866950006486,
+    "latency_p95_ms": 2135.267027699956,
+    "latency_p99_ms": 2242.278299470854,
+    "ttft_mean_ms": 139.6615898878281,
+    "ttft_p50_ms": 138.0474095003592,
+    "ttft_p95_ms": 237.62467689830373,
+    "ttft_p99_ms": 245.49128495971672,
+    "itl_mean_ms": 12.686621501440076,
+    "itl_p50_ms": 12.566233303133897,
+    "itl_p95_ms": 15.546194449997866,
+    "itl_p99_ms": 16.16330862709806,
+    "tpot_mean_ms": 12.6899270927419,
+    "tpot_p50_ms": 12.567349094479857,
+    "tpot_p95_ms": 15.547512893692872,
+    "acceptance_rate": 0.02348325183475921,
+    "acceptance_pos_1": 0.14881334188582424,
+    "acceptance_pos_2": 0.01463642194880168,
+    "acceptance_pos_3": 0.0008746865706455186,
+    "mean_acceptance_length": 1.1643827628433145
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.6809063012234507,
+    "output_throughput_tps": 85.5473654199613,
+    "speedup_vs_baseline": 0.7318459410597347,
+    "latency_mean_ms": 1468.511645206172,
+    "latency_p50_ms": 1454.6153324990883,
+    "latency_p95_ms": 2127.0678361986943,
+    "latency_p99_ms": 2281.048237952054,
+    "ttft_mean_ms": 159.72944469986032,
+    "ttft_p50_ms": 168.77598499922897,
+    "ttft_p95_ms": 265.74251724632626,
+    "ttft_p99_ms": 281.86643974164326,
+    "itl_mean_ms": 10.654948740516124,
+    "itl_p50_ms": 10.288291291322736,
+    "itl_p95_ms": 15.705922906309695,
+    "itl_p99_ms": 16.95837595336085,
+    "tpot_mean_ms": 10.6584103940507,
+    "tpot_p50_ms": 10.289786913365493,
+    "tpot_p95_ms": 15.707090863808236,
+    "acceptance_rate": 0.3230895107183767,
+    "acceptance_pos_1": 0.5774177712322042,
+    "acceptance_pos_2": 0.27245949926362295,
+    "acceptance_pos_3": 0.1193912616593029,
+    "mean_acceptance_length": 1.9692685321551302
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.7093667936140545,
+    "output_throughput_tps": 89.12307053268577,
+    "speedup_vs_baseline": 0.7624356063326214,
+    "latency_mean_ms": 1409.5885398314294,
+    "latency_p50_ms": 1421.8769274993974,
+    "latency_p95_ms": 2124.1017438478584,
+    "latency_p99_ms": 2515.584632036989,
+    "ttft_mean_ms": 146.91454585622523,
+    "ttft_p50_ms": 142.44566849811235,
+    "ttft_p95_ms": 259.6700968468212,
+    "ttft_p99_ms": 287.97775861195987,
+    "itl_mean_ms": 10.234430485377915,
+    "itl_p50_ms": 10.088829999998364,
+    "itl_p95_ms": 15.341923362584064,
+    "itl_p99_ms": 19.70016743978345,
+    "tpot_mean_ms": 10.238094617695932,
+    "tpot_p50_ms": 10.089903972448514,
+    "tpot_p95_ms": 15.343091349977053,
+    "acceptance_rate": 0.14626535131870344,
+    "acceptance_pos_1": 0.5742903160861688,
+    "acceptance_pos_2": 0.2700825447956513,
+    "acceptance_pos_3": 0.1106301590497282,
+    "mean_acceptance_length": 2.023857459230924
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.9638367060855711,
+    "output_throughput_tps": 121.09403416082594,
+    "speedup_vs_baseline": 1.035942801418763,
+    "latency_mean_ms": 1037.4081412996475,
+    "latency_p50_ms": 1001.3265734996821,
+    "latency_p95_ms": 1565.5059430991967,
+    "latency_p99_ms": 1781.7731402603386,
+    "ttft_mean_ms": 141.7672610684349,
+    "ttft_p50_ms": 145.46368399896892,
+    "ttft_p95_ms": 240.4056666531687,
+    "ttft_p99_ms": 261.25688084066496,
+    "itl_mean_ms": 7.313367654694055,
+    "itl_p50_ms": 6.893495897653395,
+    "itl_p95_ms": 11.711175241387847,
+    "itl_p99_ms": 16.168680668779416,
+    "tpot_mean_ms": 7.316552068142107,
+    "tpot_p50_ms": 6.894642385820525,
+    "tpot_p95_ms": 11.712829905159229,
+    "acceptance_rate": 0.3706431026018655,
+    "acceptance_pos_1": 0.5900483904902167,
+    "acceptance_pos_2": 0.3288449400378708,
+    "acceptance_pos_3": 0.19303597727750893,
+    "mean_acceptance_length": 2.1119293078055965
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 0.9729636803231215,
+    "output_throughput_tps": 122.24072438659618,
+    "speedup_vs_baseline": 1.045752578531864,
+    "latency_mean_ms": 1027.6755360433526,
+    "latency_p50_ms": 1009.7822754978552,
+    "latency_p95_ms": 1596.3315117551247,
+    "latency_p99_ms": 2005.884019333607,
+    "ttft_mean_ms": 141.24648624401743,
+    "ttft_p50_ms": 139.84237199474592,
+    "ttft_p95_ms": 255.56160285341318,
+    "ttft_p99_ms": 265.59654699100065,
+    "itl_mean_ms": 7.283154806994812,
+    "itl_p50_ms": 6.8463791023933425,
+    "itl_p95_ms": 11.787724323593851,
+    "itl_p99_ms": 15.836048513204252,
+    "tpot_mean_ms": 7.286351986039958,
+    "tpot_p50_ms": 6.847493661481403,
+    "tpot_p95_ms": 11.788784734661597,
+    "acceptance_rate": 0.1742194735272419,
+    "acceptance_pos_1": 0.5641138336446544,
+    "acceptance_pos_2": 0.30238435336776176,
+    "acceptance_pos_3": 0.17075046698165036,
+    "mean_acceptance_length": 2.2195363146906932
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 5000,
+    "output_tokens": 636089,
+    "request_throughput_rps": 0.4938431604479032,
+    "output_throughput_tps": 62.825640417229266,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 2024.8159037780163,
+    "latency_p50_ms": 1899.8980629985454,
+    "latency_p95_ms": 3256.729267303672,
+    "latency_p99_ms": 3405.7137782224895,
+    "ttft_mean_ms": 166.95593686518114,
+    "ttft_p50_ms": 163.98859200126026,
+    "ttft_p95_ms": 289.8830948040995,
+    "ttft_p99_ms": 336.2988253685762,
+    "itl_mean_ms": 14.712897944777563,
+    "itl_p50_ms": 13.827794673245338,
+    "itl_p95_ms": 23.864679124793426,
+    "itl_p99_ms": 24.76342741740255,
+    "tpot_mean_ms": 14.714261969753975,
+    "tpot_p50_ms": 13.829047236223541,
+    "tpot_p95_ms": 23.866155100384233,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.4093959318736204,
+    "output_throughput_tps": 52.06255313962281,
+    "speedup_vs_baseline": 0.82868320631309,
+    "latency_mean_ms": 2442.487533897996,
+    "latency_p50_ms": 2383.8188690006064,
+    "latency_p95_ms": 3377.258205899534,
+    "latency_p99_ms": 3705.6293327211833,
+    "ttft_mean_ms": 168.35537181059328,
+    "ttft_p50_ms": 170.54413749974628,
+    "ttft_p95_ms": 252.9078775001381,
+    "ttft_p99_ms": 299.23500963023804,
+    "itl_mean_ms": 18.0134018722681,
+    "itl_p50_ms": 17.54899170471861,
+    "itl_p95_ms": 24.705050003148795,
+    "itl_p99_ms": 27.191358835904875,
+    "tpot_mean_ms": 18.01468435680956,
+    "tpot_p50_ms": 17.550207216533355,
+    "tpot_p95_ms": 24.706307322046488,
+    "acceptance_rate": 0.07081888392688648,
+    "acceptance_pos_1": 0.18361221407447353,
+    "acceptance_pos_2": 0.02588322896495871,
+    "acceptance_pos_3": 0.002961208741227203,
+    "mean_acceptance_length": 1.2124566517806594
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.4019325528965641,
+    "output_throughput_tps": 51.12903618886613,
+    "speedup_vs_baseline": 0.8138243533900298,
+    "latency_mean_ms": 2487.8614642295834,
+    "latency_p50_ms": 2432.8269440011354,
+    "latency_p95_ms": 3443.123125650527,
+    "latency_p99_ms": 3724.6332480803176,
+    "ttft_mean_ms": 172.7844437341555,
+    "ttft_p50_ms": 173.04360800153518,
+    "ttft_p95_ms": 258.7694342015311,
+    "ttft_p99_ms": 299.8475944703024,
+    "itl_mean_ms": 18.332019165246155,
+    "itl_p50_ms": 17.85771458268332,
+    "itl_p95_ms": 25.186547632286487,
+    "itl_p99_ms": 27.352591655754214,
+    "tpot_mean_ms": 18.33325328096626,
+    "tpot_p50_ms": 17.85883874804062,
+    "tpot_p95_ms": 25.18777078149849,
+    "acceptance_rate": 0.03064899424451109,
+    "acceptance_pos_1": 0.1849223980874338,
+    "acceptance_pos_2": 0.025778901811864367,
+    "acceptance_pos_3": 0.0031517369145264325,
+    "mean_acceptance_length": 1.2145429597115776
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.4906630866625573,
+    "output_throughput_tps": 62.397232200408084,
+    "speedup_vs_baseline": 0.9931809972174402,
+    "latency_mean_ms": 2037.9430946188047,
+    "latency_p50_ms": 2007.9161120011122,
+    "latency_p95_ms": 2756.5027899490815,
+    "latency_p99_ms": 3044.030925800253,
+    "ttft_mean_ms": 185.9597727056098,
+    "ttft_p50_ms": 186.23321399718407,
+    "ttft_p95_ms": 279.4342198045342,
+    "ttft_p99_ms": 323.5009970532701,
+    "itl_mean_ms": 14.673263611031553,
+    "itl_p50_ms": 14.47747526376982,
+    "itl_p95_ms": 19.803011401590926,
+    "itl_p99_ms": 21.86518714472933,
+    "tpot_mean_ms": 14.674543237540032,
+    "tpot_p50_ms": 14.478983830686477,
+    "tpot_p95_ms": 19.804314228371027,
+    "acceptance_rate": 0.3113198824188073,
+    "acceptance_pos_1": 0.572229825914399,
+    "acceptance_pos_2": 0.26063458484348756,
+    "acceptance_pos_3": 0.10109523649853532,
+    "mean_acceptance_length": 1.933959647256422
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.5191022628219358,
+    "output_throughput_tps": 66.03396064905282,
+    "speedup_vs_baseline": 1.0510670517724432,
+    "latency_mean_ms": 1925.9773102255406,
+    "latency_p50_ms": 1903.7692794991017,
+    "latency_p95_ms": 2635.482599895113,
+    "latency_p99_ms": 2962.525904481372,
+    "ttft_mean_ms": 180.70931879136043,
+    "ttft_p50_ms": 182.9802385000221,
+    "ttft_p95_ms": 272.8241580512986,
+    "ttft_p99_ms": 313.5787184005314,
+    "itl_mean_ms": 13.822185940257995,
+    "itl_p50_ms": 13.62802470079417,
+    "itl_p95_ms": 18.901379390565385,
+    "itl_p99_ms": 21.488975903824134,
+    "tpot_mean_ms": 13.823438931487352,
+    "tpot_p50_ms": 13.629216417345067,
+    "tpot_p95_ms": 18.902619697261734,
+    "acceptance_rate": 0.14394009500546442,
+    "acceptance_pos_1": 0.5769995228510126,
+    "acceptance_pos_2": 0.26659988687461095,
+    "acceptance_pos_3": 0.10386364282712356,
+    "mean_acceptance_length": 2.007580665038251
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 635846,
+    "request_throughput_rps": 0.6141321767148702,
+    "output_throughput_tps": 78.09869760708868,
+    "speedup_vs_baseline": 1.2431022921283417,
+    "latency_mean_ms": 1628.2056611145947,
+    "latency_p50_ms": 1592.831803496665,
+    "latency_p95_ms": 2287.4158730010095,
+    "latency_p99_ms": 2571.3276440004856,
+    "ttft_mean_ms": 171.98177616532774,
+    "ttft_p50_ms": 172.63183150498662,
+    "ttft_p95_ms": 258.64595805287536,
+    "ttft_p99_ms": 302.3406934656669,
+    "itl_mean_ms": 11.538623067112681,
+    "itl_p50_ms": 11.238307917321478,
+    "itl_p95_ms": 16.24375355278012,
+    "itl_p99_ms": 18.46682838556656,
+    "tpot_mean_ms": 11.5398444087042,
+    "tpot_p50_ms": 11.239705511848223,
+    "tpot_p95_ms": 16.245154050035826,
+    "acceptance_rate": 0.3240773825244086,
+    "acceptance_pos_1": 0.5557454318356237,
+    "acceptance_pos_2": 0.28176742365161084,
+    "acceptance_pos_3": 0.13471929208599118,
+    "mean_acceptance_length": 1.9722321475732256
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636040,
+    "request_throughput_rps": 0.6186701811110291,
+    "output_throughput_tps": 78.69979639877178,
+    "speedup_vs_baseline": 1.2526700225595981,
+    "latency_mean_ms": 1616.259658270469,
+    "latency_p50_ms": 1584.57970800373,
+    "latency_p95_ms": 2290.956548342365,
+    "latency_p99_ms": 2594.4233495666413,
+    "ttft_mean_ms": 176.71817260388633,
+    "ttft_p50_ms": 176.76159800612368,
+    "ttft_p95_ms": 263.7752173024638,
+    "ttft_p99_ms": 303.1083872308955,
+    "itl_mean_ms": 11.399967330431618,
+    "itl_p50_ms": 11.174435653507444,
+    "itl_p95_ms": 16.282335170467917,
+    "itl_p99_ms": 18.624601763757692,
+    "tpot_mean_ms": 11.401195973910607,
+    "tpot_p50_ms": 11.175635023563679,
+    "tpot_p95_ms": 16.283561415453793,
+    "acceptance_rate": 0.15448514765416219,
+    "acceptance_pos_1": 0.5533425372792525,
+    "acceptance_pos_2": 0.28025688169439844,
+    "acceptance_pos_3": 0.13306681148704796,
+    "mean_acceptance_length": 2.0813960335791357
+  }
+]

benchmarks/large/bs1/run_config.json

@@ -0,0 +1,108 @@
+{
+  "args": {
+    "base_model": "google/t5gemma-2-1b-1b",
+    "dflash_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+    "dflare_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+    "dspark_model": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+    "speculative_token_counts": [
+      3,
+      7
+    ],
+    "methods": [
+      "dflash",
+      "dflare",
+      "dspark"
+    ],
+    "datasets": [
+      "math500",
+      "gsm8k",
+      "humaneval",
+      "mbpp",
+      "mt_bench",
+      "ultrachat"
+    ],
+    "num_prompts": 0,
+    "ultrachat_num_prompts": 5000,
+    "warmup": 8,
+    "concurrency": 1,
+    "max_num_seqs": 1,
+    "max_num_batched_tokens": 4096,
+    "max_model_len": 2048,
+    "max_tokens": 128,
+    "gpu_memory_utilization": 0.85,
+    "host": "127.0.0.1",
+    "port": 8011,
+    "temperature": 0.0,
+    "top_p": 1.0,
+    "seed": 0,
+    "timeout_s": 180.0,
+    "server_timeout_s": 600.0,
+    "enforce_eager": false,
+    "include_baseline": true
+  },
+  "runs": [
+    {
+      "name": "baseline",
+      "checkpoint": null,
+      "num_speculative_tokens": null
+    },
+    {
+      "name": "dflash_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflash_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dflare_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflare_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dspark_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dspark_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    }
+  ],
+  "prompt_sets": {
+    "math500": {
+      "count": 500,
+      "sha256": "78a67699bc700588d0e23da6457ea3afc00d5635897d456a6245f1e964449c4a"
+    },
+    "gsm8k": {
+      "count": 1319,
+      "sha256": "d591bbef95c9d5b17f6f26999bde24def2f6fbf25d7e20953b9ac89c65ffbb8e"
+    },
+    "humaneval": {
+      "count": 164,
+      "sha256": "de1066c5b81dc0f99eab434315cee2c3c68c5a522b8c25d53c73f9c9042b44c8"
+    },
+    "mbpp": {
+      "count": 500,
+      "sha256": "c596be7f77834627fd0a05755691e30dbf4f6724425842de6d03348614e5fecd"
+    },
+    "mt_bench": {
+      "count": 160,
+      "sha256": "8761bfece58bd26e5b93c33e9aad480728ea9950a7d257aaff267db5c7b98010"
+    },
+    "ultrachat": {
+      "count": 5000,
+      "sha256": "5dac9e9dc0d63372c701dbebbb18f3f22a568e9b62cc0ff1b4fb88fc41bf5bf7"
+    }
+  },
+  "benchmark_fingerprint": "436353e1f7cd491994a7f381976fb001a63b77a4fea4411f0b645d604defc27e",
+  "output_dir": "/mnt/d/Projects/dflash_t5/benchmark-results/t5gemma2_speculators_suite_bs1_fixed"
+}

benchmarks/large/bs4/comparison.csv

@@ -0,0 +1,43 @@
+dataset,run,method,num_speculative_tokens,requests,output_tokens,request_throughput_rps,output_throughput_tps,speedup_vs_baseline,latency_mean_ms,latency_p50_ms,latency_p95_ms,latency_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,acceptance_rate,acceptance_pos_1,acceptance_pos_2,acceptance_pos_3,mean_acceptance_length
+math500,baseline,baseline,,500,63527,2.0994279500650657,266.7407187675668,1.0,1901.9527513722423,1894.7456854948541,2310.2332794354875,2623.0439431988634,211.01336648262804,200.41962849791162,379.0829671488606,495.6532387045445,13.40169340837536,13.301524035513445,16.309320616836498,19.31773654766166,13.403480814341385,13.303263444846724,16.311841278199893,,,,,
+math500,dflash_k3,dflash,3,500,63537,1.4572988369276767,185.18479240374757,0.694250181447229,2740.524441106001,2728.2846120000104,3396.0575988000182,3908.6018861800358,232.72850664999817,208.99405350002098,422.7843856499816,635.657571979995,19.894394063291394,19.884664330708496,24.709495220472647,27.384374742046813,19.89611259958302,19.88607477952736,24.71110046456739,0.06285151149944762,0.16639047905995782,0.019716028924374813,0.002448026514010244,1.1885545344983428
+math500,dflash_k7,dflash,7,500,63468,1.484521663636696,188.43924189538765,0.7064509789358043,2690.2326844700183,2693.1952875002025,3411.2693664001654,3642.2530803891277,223.1485508320311,201.97454650042346,410.6845789998714,603.7728062999304,19.605501851902595,19.610514098427146,24.830368157877164,27.14365097669306,19.607220136180974,19.612673795280877,24.8317818901585,0.02742245654217219,0.1686266354080012,0.020192089540951374,0.0026090902215836044,1.1919571957952053
+math500,dflare_k3,dflare,3,500,63495,1.7148645616582274,217.7706506849783,0.8164132258889946,2328.9442384679533,2273.6253690000012,3331.7489405495494,3864.866585709678,279.50968451999506,263.16054300059477,506.23001639960404,776.1183005902055,16.27184360810761,15.999782673234437,23.205805455507797,27.393199366532944,16.273605324281704,16.000888346459,23.207303030310158,0.3935291759720552,0.6488919429651263,0.35275725820305787,0.17893832674798144,2.1805875279161655
+math500,dflare_k7,dflare,7,500,63622,1.7760438758773052,225.9909269421318,0.8472307039820806,2249.2024364179815,2185.3041670019593,3338.340588499702,4000.3402635519888,263.0280147520243,252.19799500155204,475.0075314999774,595.4660286509533,15.760428864173193,15.345385657476402,24.247727375981995,29.25197200714273,15.762062569323362,15.347322909449966,24.248864364582953,0.18693377898042707,0.6466625620223824,0.3548223534098729,0.17529245590525516,2.3085364528629895
+math500,dspark_k3,dspark,3,500,63636,2.2498034619335257,286.33698620720367,1.0734656018405375,1774.1428698560267,1733.4975750000012,2512.932257099783,2941.1046552174953,237.00501048202568,212.508828000864,434.44543489968055,581.4404782277412,12.18272455456093,11.912186838580821,17.877204798435372,22.473681951484796,12.18441765344298,11.913581936995776,17.87863581495454,0.4499175952574226,0.6567411999114456,0.42247804590067156,0.27053353996015056,2.3497527857722678
+math500,dspark_k7,dspark,7,500,63621,2.316449613127783,294.7496816736054,1.1050044516467135,1725.2525323180525,1675.7616530012456,2646.2700289983313,3227.1719797111646,247.76108907210437,221.10979600074643,461.0275230510521,670.7492681724033,11.723149998862587,11.28927224801989,18.88152986101506,22.771947493768472,11.72477114787681,11.290766421259073,18.882649400401576,0.22890383017299992,0.6397510474718301,0.3964121547410812,0.24219989423585406,2.6023268112109994
+gsm8k,baseline,baseline,,1319,168502,2.196360834095345,280.5846802628763,1.0,1819.2423071760666,1798.3721370110288,2167.7596873778384,2310.5863884446435,217.04894016683772,191.49014502181672,398.45379750477116,646.2297889695037,12.638841870584082,12.576321425240883,15.217476899920795,16.315808897562086,12.640636354922657,12.58000173247566,15.218959284321164,,,,,
+gsm8k,dflash_k3,dflash,3,1319,168356,1.4790952868779552,188.79042162064067,0.6728465055318247,2702.3829456360854,2683.6698380000144,3317.1016436999594,3597.6961572799746,235.6068064935536,221.15605599992705,424.62412700004944,526.131548720104,19.477842147847973,19.355762133858388,24.183653737845873,26.07432056251997,19.479491279517458,19.357246826771867,24.18510518785972,0.0680806946304376,0.18367244085417656,0.018568116693305686,0.002001526343830547,1.2042420838913128
+gsm8k,dflash_k7,dflash,7,1319,168367,1.4764153266187332,188.46066663898122,0.6716712632436481,2705.295749132689,2679.983797999739,3477.261450399509,4079.732661799678,301.1807209363258,224.2310860001453,834.4522548001792,1980.7203211395963,18.978770912763007,19.034192118114312,23.592571637004227,25.352459187087952,18.980617877378993,19.03604929134238,23.594471511024295,0.027380290750817878,0.17135507958901572,0.018069627919570206,0.001987944081056817,1.1916620352557252
+gsm8k,dflare_k3,dflare,3,1319,168415,1.574435488473932,201.0299869532504,0.7164681505950642,2536.437202144023,2473.5267859996384,3534.031745998982,4064.8575092010615,258.1910506959681,238.57910900005663,484.66444300011057,598.7782853190582,17.977036765777857,17.445515787392107,25.81660302204809,30.753264190233132,17.978638165785107,17.446966267718064,25.818113115744246,0.2794753137068277,0.536425000273502,0.21512575623311125,0.08687518461386984,1.838425941120483
+gsm8k,dflare_k7,dflare,7,1319,168400,1.2094482721851378,154.41325931461503,0.5503267647041427,3304.2710242100056,3064.5563209982356,5885.097530001435,7240.322633720641,1205.8129131272765,677.296675999969,4089.5798977991944,5644.379561060411,16.556715234828605,16.74034803936521,27.050037795280637,35.51621275700378,16.558478447758667,16.743846314973666,27.058657145679476,0.12825457741085033,0.5421524916756024,0.21891754613691516,0.09374118178226762,1.8977820418759523
+gsm8k,dspark_k3,dspark,3,1319,168413,2.0975172075679485,267.81589497963677,0.954492222200882,1904.8592331402729,1849.926056998811,2717.593596396909,3128.5920352824137,232.41419941998183,211.30282399826683,436.5587686021171,608.768046818439,13.20091798661006,12.809261669308327,19.22371915434054,22.673929599071368,13.202611622505504,12.810728181104036,19.225053650402195,0.31145141140312554,0.5333057414177645,0.2575245453082247,0.14352394748338737,1.9343542342093767
+gsm8k,dspark_k7,dspark,7,1319,168420,2.0069879067415446,256.2675536417065,0.9133340900922055,1991.4858750174596,1940.2353619989299,2946.936268898934,3409.506298801133,240.2313964965955,217.0051639986923,457.4054478005564,621.9962791216665,13.815387803215751,13.491749685057384,21.069445844088154,24.68408968393126,13.817049546266604,13.493896850398981,21.07079229370886,0.1463064478504878,0.5319253673998655,0.2490394774757468,0.13306838920372682,2.0241451349534145
+humaneval,baseline,baseline,,164,11229,2.83651766504482,194.21498085846514,1.0,1403.542471638946,1171.1717744910857,2734.2781511018984,3048.235554893327,218.20821551103487,209.54365949728526,420.2904565172503,467.85162955609854,17.843567054063506,17.024532066210227,29.900638077389125,38.85421750062092,17.849692097307134,17.028038263717008,29.90535920137036,,,,,
+humaneval,dflash_k3,dflash,3,164,11292,2.1388107405014773,147.26494440087,0.7582573895686753,1856.8446467317099,1529.8299214999815,3609.9989535499276,4076.3032318901037,250.89464046951855,233.70097149984304,497.8459723500346,638.7824368799534,24.549073372278432,22.60172811232828,42.012251182637584,51.31418752062339,24.555336995750512,22.603792561644838,42.01500447592228,0.059506172839506176,0.16412698412698412,0.013544973544973546,0.0008465608465608466,1.1785185185185185
+humaneval,dflash_k7,dflash,7,164,11052,2.1057562192267367,141.90742521276766,0.7306718801274306,1877.2794746829145,1504.3869174996871,3978.088593899155,4819.554965540591,343.89938129273037,254.9212224994335,982.0552947493524,1265.2674350103007,22.96417710011784,22.33989236389506,33.208631611878445,43.04405845282055,22.970011297649936,22.34099295422598,33.23066878742338,0.02684898226160627,0.17398888040989863,0.013081870707511174,0.0008721247138340783,1.1879428758312438
+humaneval,dflare_k3,dflare,3,164,11277,2.8294040460667382,194.55603309447932,1.0017560552461333,1401.6537050121503,1262.2241380004198,2758.0062766009178,3165.725670439934,307.73021359141796,282.3090209994916,627.4767150008302,760.1846476996616,17.775962665717213,16.24701220755945,33.04292447846012,49.24142361158533,17.781789938797445,16.250892179999738,33.04544840657803,0.5873682476943346,0.7734683794466403,0.5642292490118577,0.4244071146245059,2.762104743083004
+humaneval,dflare_k7,dflare,7,164,11047,3.0665775841960174,206.5639181256915,1.0635838554402026,1296.2304299939337,1225.3228209992812,2684.5432901496665,3137.9315633195806,395.1852547253976,324.23020049827755,956.4745560996016,1385.3407927481385,16.226717747252554,13.57385855245207,32.95489575833981,75.28418661784087,16.23359719581637,13.576510465236616,32.96468897523092,0.33047302832995556,0.7790522185330516,0.5644431029278599,0.4044672502263809,3.3133111983096892
+humaneval,dspark_k3,dspark,3,164,11346,3.393797033660498,234.79281185312203,1.2089325489480554,1172.9745371464626,1033.1892974991206,2107.933460299682,3156.528969858848,265.29695418294733,228.4132034983486,502.8849432006609,699.91048091968,14.77773082701616,13.023184389006781,29.633302174848463,45.321116141877226,14.783925134182368,13.039449575903792,29.64001845995613,0.5991372158619546,0.7653061224489796,0.587356893977103,0.44474863115978097,2.7974116475858635
+humaneval,dspark_k7,dspark,7,164,11058,3.1234019762512726,210.6010917889425,1.08437099372071,1276.6294960669388,1098.64930799813,2616.217871799927,3188.0119062716053,564.2550010548891,336.2399385005119,1681.4955957521306,2938.6766085621284,12.486812538079404,11.807729195672675,25.278697270681064,43.8447163140126,12.494207252966246,11.83652314342243,25.281766504248417,0.35094729280775794,0.740100565681961,0.5502828409805154,0.39849151477058453,3.4566310496543053
+mbpp,baseline,baseline,,500,61587,2.23460184339787,275.2448474586892,1.0,1788.0222868800047,1819.359868997708,2216.346881000209,2362.3936853421037,221.07102080830373,204.21197451651096,398.6740397042012,655.3067808030755,13.201886877023522,12.736825452678412,15.664009772456069,17.321856500478695,13.205948736768402,12.738972996026057,15.67098670536386,,,,,
+mbpp,dflash_k3,dflash,3,500,61808,1.5365610042290248,189.94352509877513,0.690089303587357,2595.6573488480017,2617.603001500015,3320.9663893999277,3516.577585000016,234.1106791739985,218.79009800011318,439.53066730014143,544.9614302001191,19.14316073130201,18.9773965472434,23.96081993622043,25.695518167638486,19.146981896371145,18.978798267717032,23.96214876692921,0.06462320419065401,0.1824529329391629,0.01106659405632488,0.0003500855764742493,1.1938696125719621
+mbpp,dflash_k7,dflash,7,500,61377,1.387242813763742,170.2896043607544,0.6186840768611036,2875.578809605966,2855.830152499948,4074.322934499559,4764.013334749542,634.0442261159515,392.3641729988958,1885.916530999566,2897.4566687989864,18.201249949891572,18.463196299214943,24.69457239645406,30.63269140497326,18.205769807911185,18.464951303145412,24.695845573618083,0.028501713363674443,0.1877840964993408,0.011255632735787795,0.0004525866309844743,1.199511993545721
+mbpp,dflare_k3,dflare,3,500,61640,1.9801835796759275,244.11703170244834,0.8869086340992713,2016.50626224402,2021.2945930006754,2791.478140551044,3086.5239088800263,280.0661946479813,256.2331505005204,557.8820319009537,769.5084362501256,14.264965995399256,14.109097578741029,19.439541319448956,23.81797452189929,14.269046870264877,14.11056187795813,19.634308889327635,0.47048386886358,0.6888967163711792,0.44499236820476695,0.2775625220147939,2.41145160659074
+mbpp,dflare_k7,dflare,7,500,61475,1.503343533132173,184.83608739860065,0.6715333242572042,2657.845550606042,2483.5515449995,4488.94517174831,5523.38156365822,1190.47147037393,932.4905619996571,3259.4635773521077,4279.743841777054,12.451201825429687,12.228331582672416,22.859363721256052,36.967242600596585,12.456394792296264,12.230752885830222,22.860772319684717,0.24657171832973235,0.6712818025486132,0.42739979716918736,0.2544644825609595,2.7260020283081268
+mbpp,dspark_k3,dspark,3,500,61981,2.507209609073491,310.7987175599681,1.1291717916958104,1592.6791832899762,1603.4026140005153,2178.7681314988727,2552.1267430388243,240.7260946480019,224.12168300070334,455.65863744814123,601.2736888401558,11.346454396871485,10.845360450802424,15.682580235835754,23.395322236705972,11.350096012527779,10.847383575174785,15.684506253152122,0.5228963898667771,0.7125284738041002,0.5082211638020294,0.3479395319942017,2.568689169600331
+mbpp,dspark_k7,dspark,7,500,61632,2.2269171937357544,274.498720968644,0.997289226312739,1794.3305799321024,1694.5109794996824,3171.435100350026,4215.3635372184735,643.143013108056,362.4557425009698,2119.0080873015154,3187.935962892621,10.101565495090881,9.322213830710384,14.88066484486098,19.67402437656759,10.105878412845934,9.323771708668072,14.882234868113521,0.28831800034716193,0.6831591737545565,0.47018226002430136,0.3054678007290401,3.0182260024301337
+mt_bench,baseline,baseline,,160,20102,2.461309935234548,309.2328269880305,1.0,1607.6186366983165,1548.2021450006869,2284.110534451611,2534.602298826794,320.49049263077904,293.138627501321,629.1677434302974,759.140422597702,11.007379475699068,9.985256696983395,15.183904158686165,26.580996584276846,11.014195983013959,9.988513000022554,15.18576166809785,,,,,
+mt_bench,dflash_k3,dflash,3,160,20228,1.5473009629136512,195.61752423635835,0.632589774318915,2573.229441274998,2591.1725164999098,3166.3267970500783,3608.086452589978,221.35167090626453,203.15486550020978,448.1615023500803,696.169536259867,18.676105907776027,18.64073133858227,23.103904855904904,25.284028265118394,18.68020851087627,18.641842641731216,23.105400170866087,0.060748671490479834,0.16453909053473734,0.01705982704864992,0.0006470968880522384,1.1822460144714395
+mt_bench,dflash_k7,dflash,7,160,20102,1.4356652700068615,180.37339536048705,0.5832931681844656,2753.9001531249596,2798.7496705000012,3791.8684454991308,4289.662929459572,381.8320665374472,295.44119050024165,1026.5876399000251,1367.1522384703346,21.709309200685574,18.78574032282878,25.575840727953157,59.525236419324514,21.713930513288513,18.787575688973476,25.57992160196576,0.024768061794215188,0.15474581251836614,0.016044666470761092,0.0012342051131354686,1.1733764325595064
+mt_bench,dflare_k3,dflare,3,160,20228,1.7821635622658862,225.31002835946467,0.7286096710818666,2231.2470228500047,2197.780867999427,3333.7097092000463,3700.461037129698,248.96705823127832,215.34929249992274,520.0022110006103,662.6540237599145,15.77862160127276,15.232473944878599,23.602753981101593,27.57565280062838,15.782222927801888,15.23403437795065,23.60411872952008,0.3229694606887589,0.5750487329434698,0.27573099415204677,0.11812865497076024,1.9689083820662767
+mt_bench,dflare_k7,dflare,7,160,20102,1.0658824478410331,133.9148060406278,0.4330549487419434,3752.528322143644,3464.8722160018224,6717.973978601185,7876.092720710149,1934.0097326686418,1588.4214640009304,4820.7839886515985,6494.002312830161,14.317043586959333,13.67209011023508,29.82729992639314,39.377268431499964,14.323458562974045,13.673784755900577,29.829240646451908,0.14578666207134242,0.5689585846401287,0.26940088459991957,0.11540008041817451,2.020506634499397
+mt_bench,dspark_k3,dspark,3,160,20228,2.2379435718508462,282.9320160712432,0.9149481923605564,1771.671463362577,1775.1809370001865,2604.8437047013063,3118.5169013906484,252.11589151258522,227.79171349975513,503.8048422973585,627.0037127103565,12.360936149822532,11.83748445670406,19.148658472828007,25.009972698654565,12.364664553850584,11.838727370086703,19.150851622824714,0.36540854417254254,0.5802571547075902,0.3253836582330983,0.19058481957693904,2.0962256325176276
+mt_bench,dspark_k7,dspark,7,160,20102,1.784863821370322,224.24582835741384,0.7251682511898833,2225.1736813125035,2117.7182369992806,3657.939555099983,4345.6770745698295,838.8437911749861,589.9439045006147,2441.8948530525054,2773.190703780456,11.426459321042842,10.71108703149011,22.870212017662954,25.594833969671022,11.431539335517765,10.713255098423582,23.0010987298119,0.17215237263282687,0.5529591613889495,0.2997379340467351,0.16739462764795807,2.205066608429788
+ultrachat,baseline,baseline,,5000,635569,1.2230023684575178,155.4604784636352,1.0,3269.5651201336645,3276.8055605120026,3934.875892131822,4087.044628756121,231.24218063543086,218.61417350010015,386.9901933081565,553.9115706225857,24.08620830909222,24.040817992012823,28.90904858166638,29.91989293644905,24.087996070253364,24.042644452736308,28.910471471294663,,,,,
+ultrachat,dflash_k3,dflash,3,5000,636175,1.0622074332559788,135.14996277032446,0.8693525461002507,3763.952480385199,3769.2712649998157,4515.89870830021,4805.0974385998825,257.6036252754003,246.59032999988995,405.558623099887,540.4809878199559,27.764084011389272,27.68626928346698,33.282436218505644,35.5445708588995,27.765764208304795,27.687906570868,33.28350510590754,0.07093823412968715,0.1840484911746139,0.02574222849873457,0.0030239827157130055,1.2128147023890614
+ultrachat,dflash_k7,dflash,7,5000,636064,1.0536069317356047,134.03228788549515,0.8621630990081349,3794.9240800698,3797.6578120005797,4543.265116648946,4838.748624158453,255.93729505000047,246.4688734989977,399.62789619985415,533.8578004602641,28.02829548006744,27.958994283470407,33.55774176338532,35.83546665536011,28.030316558857468,27.961111295281434,33.55949413779617,0.030697806232490035,0.1853871675887231,0.025839366107025186,0.0030756573256390638,1.2148846436274303
+ultrachat,dflare_k3,dflare,3,5000,636228,1.267241365085764,161.25088784515708,1.0372468259376697,3155.8181194270032,3158.2490724995296,4013.4220384479713,4389.7799070399205,282.9019767206311,270.00340999984473,456.05508385024234,595.7626636705755,22.751411578135976,22.677721858262814,29.19536946929717,32.0268499520443,22.753117775499057,22.679262984259076,29.196810057103683,0.31057058500760176,0.5710008489248474,0.2600387645252867,0.10067214157267132,1.9317117550228053
+ultrachat,dflare_k7,dflare,7,5000,635796,1.2415631421695674,157.87617590776844,1.0155389811481785,3221.0815614180046,3164.543153499835,4123.292088050039,4616.265497698114,352.63900740198807,287.9059334991325,612.3089999979129,1196.3210188611995,22.7251370702789,22.450031862195097,29.62520190592225,33.215207786221775,22.72683404563422,22.451526275598177,29.626681563398265,0.1436652803641537,0.5763152990672803,0.26589619110489227,0.10326562455582894,2.005656962549076
+ultrachat,dspark_k3,dspark,3,5000,636215,1.5004265390600509,190.91877410961806,1.2280855944636573,2665.1578789277846,2665.9706919999735,3441.952741001842,3755.308294759525,264.3723180442066,253.26144649989146,429.1125219519018,559.1942446103716,19.008812154313944,18.913669216526902,24.80109081614243,27.068464386929747,19.010452079968594,18.915089492132775,24.80313036299977,0.32401200895393567,0.5557804733415505,0.28182543551086175,0.1344301180093948,1.9720360268618071
+ultrachat,dspark_k7,dspark,7,5000,636157,1.4902242443540412,189.60331692310675,1.2196239121151176,2683.3432791144196,2638.018617002672,3469.7974141512536,3857.7436888694515,287.66377756739894,259.45143649914826,433.6856615469515,568.5856753819111,18.969870524426348,18.7173821692899,25.069323533594044,28.087401625863414,18.971464480602076,18.718672405515193,25.071671481768842,0.15400452166690387,0.5521141942543566,0.27944601414362474,0.13266262718646393,2.078031651668327

benchmarks/large/bs4/comparison.md

@@ -0,0 +1,44 @@
+| Dataset | Run | Req | Output tok/s | Speedup | Latency mean/p95 ms | TTFT mean/p95 ms | ITL mean/p95 ms | TPOT mean/p95 ms | Accept rate | Accept p1/p2/p3 | Accept len |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| math500 | baseline | 500 | 266.74 | 1.000x | 1901.95/2310.23 | 211.01/379.08 | 13.40/16.31 | 13.40/16.31 | — | -/-/- | — |
+| math500 | dflash_k3 | 500 | 185.18 | 0.694x | 2740.52/3396.06 | 232.73/422.78 | 19.89/24.71 | 19.90/24.71 | 6.29% | 16.64%/1.97%/0.24% | 1.189 |
+| math500 | dflash_k7 | 500 | 188.44 | 0.706x | 2690.23/3411.27 | 223.15/410.68 | 19.61/24.83 | 19.61/24.83 | 2.74% | 16.86%/2.02%/0.26% | 1.192 |
+| math500 | dflare_k3 | 500 | 217.77 | 0.816x | 2328.94/3331.75 | 279.51/506.23 | 16.27/23.21 | 16.27/23.21 | 39.35% | 64.89%/35.28%/17.89% | 2.181 |
+| math500 | dflare_k7 | 500 | 225.99 | 0.847x | 2249.20/3338.34 | 263.03/475.01 | 15.76/24.25 | 15.76/24.25 | 18.69% | 64.67%/35.48%/17.53% | 2.309 |
+| math500 | dspark_k3 | 500 | 286.34 | 1.073x | 1774.14/2512.93 | 237.01/434.45 | 12.18/17.88 | 12.18/17.88 | 44.99% | 65.67%/42.25%/27.05% | 2.350 |
+| math500 | dspark_k7 | 500 | 294.75 | 1.105x | 1725.25/2646.27 | 247.76/461.03 | 11.72/18.88 | 11.72/18.88 | 22.89% | 63.98%/39.64%/24.22% | 2.602 |
+| gsm8k | baseline | 1319 | 280.58 | 1.000x | 1819.24/2167.76 | 217.05/398.45 | 12.64/15.22 | 12.64/15.22 | — | -/-/- | — |
+| gsm8k | dflash_k3 | 1319 | 188.79 | 0.673x | 2702.38/3317.10 | 235.61/424.62 | 19.48/24.18 | 19.48/24.19 | 6.81% | 18.37%/1.86%/0.20% | 1.204 |
+| gsm8k | dflash_k7 | 1319 | 188.46 | 0.672x | 2705.30/3477.26 | 301.18/834.45 | 18.98/23.59 | 18.98/23.59 | 2.74% | 17.14%/1.81%/0.20% | 1.192 |
+| gsm8k | dflare_k3 | 1319 | 201.03 | 0.716x | 2536.44/3534.03 | 258.19/484.66 | 17.98/25.82 | 17.98/25.82 | 27.95% | 53.64%/21.51%/8.69% | 1.838 |
+| gsm8k | dflare_k7 | 1319 | 154.41 | 0.550x | 3304.27/5885.10 | 1205.81/4089.58 | 16.56/27.05 | 16.56/27.06 | 12.83% | 54.22%/21.89%/9.37% | 1.898 |
+| gsm8k | dspark_k3 | 1319 | 267.82 | 0.954x | 1904.86/2717.59 | 232.41/436.56 | 13.20/19.22 | 13.20/19.23 | 31.15% | 53.33%/25.75%/14.35% | 1.934 |
+| gsm8k | dspark_k7 | 1319 | 256.27 | 0.913x | 1991.49/2946.94 | 240.23/457.41 | 13.82/21.07 | 13.82/21.07 | 14.63% | 53.19%/24.90%/13.31% | 2.024 |
+| humaneval | baseline | 164 | 194.21 | 1.000x | 1403.54/2734.28 | 218.21/420.29 | 17.84/29.90 | 17.85/29.91 | — | -/-/- | — |
+| humaneval | dflash_k3 | 164 | 147.26 | 0.758x | 1856.84/3610.00 | 250.89/497.85 | 24.55/42.01 | 24.56/42.02 | 5.95% | 16.41%/1.35%/0.08% | 1.179 |
+| humaneval | dflash_k7 | 164 | 141.91 | 0.731x | 1877.28/3978.09 | 343.90/982.06 | 22.96/33.21 | 22.97/33.23 | 2.68% | 17.40%/1.31%/0.09% | 1.188 |
+| humaneval | dflare_k3 | 164 | 194.56 | 1.002x | 1401.65/2758.01 | 307.73/627.48 | 17.78/33.04 | 17.78/33.05 | 58.74% | 77.35%/56.42%/42.44% | 2.762 |
+| humaneval | dflare_k7 | 164 | 206.56 | 1.064x | 1296.23/2684.54 | 395.19/956.47 | 16.23/32.95 | 16.23/32.96 | 33.05% | 77.91%/56.44%/40.45% | 3.313 |
+| humaneval | dspark_k3 | 164 | 234.79 | 1.209x | 1172.97/2107.93 | 265.30/502.88 | 14.78/29.63 | 14.78/29.64 | 59.91% | 76.53%/58.74%/44.47% | 2.797 |
+| humaneval | dspark_k7 | 164 | 210.60 | 1.084x | 1276.63/2616.22 | 564.26/1681.50 | 12.49/25.28 | 12.49/25.28 | 35.09% | 74.01%/55.03%/39.85% | 3.457 |
+| mbpp | baseline | 500 | 275.24 | 1.000x | 1788.02/2216.35 | 221.07/398.67 | 13.20/15.66 | 13.21/15.67 | — | -/-/- | — |
+| mbpp | dflash_k3 | 500 | 189.94 | 0.690x | 2595.66/3320.97 | 234.11/439.53 | 19.14/23.96 | 19.15/23.96 | 6.46% | 18.25%/1.11%/0.04% | 1.194 |
+| mbpp | dflash_k7 | 500 | 170.29 | 0.619x | 2875.58/4074.32 | 634.04/1885.92 | 18.20/24.69 | 18.21/24.70 | 2.85% | 18.78%/1.13%/0.05% | 1.200 |
+| mbpp | dflare_k3 | 500 | 244.12 | 0.887x | 2016.51/2791.48 | 280.07/557.88 | 14.26/19.44 | 14.27/19.63 | 47.05% | 68.89%/44.50%/27.76% | 2.411 |
+| mbpp | dflare_k7 | 500 | 184.84 | 0.672x | 2657.85/4488.95 | 1190.47/3259.46 | 12.45/22.86 | 12.46/22.86 | 24.66% | 67.13%/42.74%/25.45% | 2.726 |
+| mbpp | dspark_k3 | 500 | 310.80 | 1.129x | 1592.68/2178.77 | 240.73/455.66 | 11.35/15.68 | 11.35/15.68 | 52.29% | 71.25%/50.82%/34.79% | 2.569 |
+| mbpp | dspark_k7 | 500 | 274.50 | 0.997x | 1794.33/3171.44 | 643.14/2119.01 | 10.10/14.88 | 10.11/14.88 | 28.83% | 68.32%/47.02%/30.55% | 3.018 |
+| mt_bench | baseline | 160 | 309.23 | 1.000x | 1607.62/2284.11 | 320.49/629.17 | 11.01/15.18 | 11.01/15.19 | — | -/-/- | — |
+| mt_bench | dflash_k3 | 160 | 195.62 | 0.633x | 2573.23/3166.33 | 221.35/448.16 | 18.68/23.10 | 18.68/23.11 | 6.07% | 16.45%/1.71%/0.06% | 1.182 |
+| mt_bench | dflash_k7 | 160 | 180.37 | 0.583x | 2753.90/3791.87 | 381.83/1026.59 | 21.71/25.58 | 21.71/25.58 | 2.48% | 15.47%/1.60%/0.12% | 1.173 |
+| mt_bench | dflare_k3 | 160 | 225.31 | 0.729x | 2231.25/3333.71 | 248.97/520.00 | 15.78/23.60 | 15.78/23.60 | 32.30% | 57.50%/27.57%/11.81% | 1.969 |
+| mt_bench | dflare_k7 | 160 | 133.91 | 0.433x | 3752.53/6717.97 | 1934.01/4820.78 | 14.32/29.83 | 14.32/29.83 | 14.58% | 56.90%/26.94%/11.54% | 2.021 |
+| mt_bench | dspark_k3 | 160 | 282.93 | 0.915x | 1771.67/2604.84 | 252.12/503.80 | 12.36/19.15 | 12.36/19.15 | 36.54% | 58.03%/32.54%/19.06% | 2.096 |
+| mt_bench | dspark_k7 | 160 | 224.25 | 0.725x | 2225.17/3657.94 | 838.84/2441.89 | 11.43/22.87 | 11.43/23.00 | 17.22% | 55.30%/29.97%/16.74% | 2.205 |
+| ultrachat | baseline | 5000 | 155.46 | 1.000x | 3269.57/3934.88 | 231.24/386.99 | 24.09/28.91 | 24.09/28.91 | — | -/-/- | — |
+| ultrachat | dflash_k3 | 5000 | 135.15 | 0.869x | 3763.95/4515.90 | 257.60/405.56 | 27.76/33.28 | 27.77/33.28 | 7.09% | 18.40%/2.57%/0.30% | 1.213 |
+| ultrachat | dflash_k7 | 5000 | 134.03 | 0.862x | 3794.92/4543.27 | 255.94/399.63 | 28.03/33.56 | 28.03/33.56 | 3.07% | 18.54%/2.58%/0.31% | 1.215 |
+| ultrachat | dflare_k3 | 5000 | 161.25 | 1.037x | 3155.82/4013.42 | 282.90/456.06 | 22.75/29.20 | 22.75/29.20 | 31.06% | 57.10%/26.00%/10.07% | 1.932 |
+| ultrachat | dflare_k7 | 5000 | 157.88 | 1.016x | 3221.08/4123.29 | 352.64/612.31 | 22.73/29.63 | 22.73/29.63 | 14.37% | 57.63%/26.59%/10.33% | 2.006 |
+| ultrachat | dspark_k3 | 5000 | 190.92 | 1.228x | 2665.16/3441.95 | 264.37/429.11 | 19.01/24.80 | 19.01/24.80 | 32.40% | 55.58%/28.18%/13.44% | 1.972 |
+| ultrachat | dspark_k7 | 5000 | 189.60 | 1.220x | 2683.34/3469.80 | 287.66/433.69 | 18.97/25.07 | 18.97/25.07 | 15.40% | 55.21%/27.94%/13.27% | 2.078 |

benchmarks/large/bs4/results.json

@@ -0,0 +1,1304 @@
+[
+  {
+    "dataset": "math500",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 63527,
+    "request_throughput_rps": 2.0994279500650657,
+    "output_throughput_tps": 266.7407187675668,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1901.9527513722423,
+    "latency_p50_ms": 1894.7456854948541,
+    "latency_p95_ms": 2310.2332794354875,
+    "latency_p99_ms": 2623.0439431988634,
+    "ttft_mean_ms": 211.01336648262804,
+    "ttft_p50_ms": 200.41962849791162,
+    "ttft_p95_ms": 379.0829671488606,
+    "ttft_p99_ms": 495.6532387045445,
+    "itl_mean_ms": 13.40169340837536,
+    "itl_p50_ms": 13.301524035513445,
+    "itl_p95_ms": 16.309320616836498,
+    "itl_p99_ms": 19.31773654766166,
+    "tpot_mean_ms": 13.403480814341385,
+    "tpot_p50_ms": 13.303263444846724,
+    "tpot_p95_ms": 16.311841278199893,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63537,
+    "request_throughput_rps": 1.4572988369276767,
+    "output_throughput_tps": 185.18479240374757,
+    "speedup_vs_baseline": 0.694250181447229,
+    "latency_mean_ms": 2740.524441106001,
+    "latency_p50_ms": 2728.2846120000104,
+    "latency_p95_ms": 3396.0575988000182,
+    "latency_p99_ms": 3908.6018861800358,
+    "ttft_mean_ms": 232.72850664999817,
+    "ttft_p50_ms": 208.99405350002098,
+    "ttft_p95_ms": 422.7843856499816,
+    "ttft_p99_ms": 635.657571979995,
+    "itl_mean_ms": 19.894394063291394,
+    "itl_p50_ms": 19.884664330708496,
+    "itl_p95_ms": 24.709495220472647,
+    "itl_p99_ms": 27.384374742046813,
+    "tpot_mean_ms": 19.89611259958302,
+    "tpot_p50_ms": 19.88607477952736,
+    "tpot_p95_ms": 24.71110046456739,
+    "acceptance_rate": 0.06285151149944762,
+    "acceptance_pos_1": 0.16639047905995782,
+    "acceptance_pos_2": 0.019716028924374813,
+    "acceptance_pos_3": 0.002448026514010244,
+    "mean_acceptance_length": 1.1885545344983428
+  },
+  {
+    "dataset": "math500",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63468,
+    "request_throughput_rps": 1.484521663636696,
+    "output_throughput_tps": 188.43924189538765,
+    "speedup_vs_baseline": 0.7064509789358043,
+    "latency_mean_ms": 2690.2326844700183,
+    "latency_p50_ms": 2693.1952875002025,
+    "latency_p95_ms": 3411.2693664001654,
+    "latency_p99_ms": 3642.2530803891277,
+    "ttft_mean_ms": 223.1485508320311,
+    "ttft_p50_ms": 201.97454650042346,
+    "ttft_p95_ms": 410.6845789998714,
+    "ttft_p99_ms": 603.7728062999304,
+    "itl_mean_ms": 19.605501851902595,
+    "itl_p50_ms": 19.610514098427146,
+    "itl_p95_ms": 24.830368157877164,
+    "itl_p99_ms": 27.14365097669306,
+    "tpot_mean_ms": 19.607220136180974,
+    "tpot_p50_ms": 19.612673795280877,
+    "tpot_p95_ms": 24.8317818901585,
+    "acceptance_rate": 0.02742245654217219,
+    "acceptance_pos_1": 0.1686266354080012,
+    "acceptance_pos_2": 0.020192089540951374,
+    "acceptance_pos_3": 0.0026090902215836044,
+    "mean_acceptance_length": 1.1919571957952053
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63495,
+    "request_throughput_rps": 1.7148645616582274,
+    "output_throughput_tps": 217.7706506849783,
+    "speedup_vs_baseline": 0.8164132258889946,
+    "latency_mean_ms": 2328.9442384679533,
+    "latency_p50_ms": 2273.6253690000012,
+    "latency_p95_ms": 3331.7489405495494,
+    "latency_p99_ms": 3864.866585709678,
+    "ttft_mean_ms": 279.50968451999506,
+    "ttft_p50_ms": 263.16054300059477,
+    "ttft_p95_ms": 506.23001639960404,
+    "ttft_p99_ms": 776.1183005902055,
+    "itl_mean_ms": 16.27184360810761,
+    "itl_p50_ms": 15.999782673234437,
+    "itl_p95_ms": 23.205805455507797,
+    "itl_p99_ms": 27.393199366532944,
+    "tpot_mean_ms": 16.273605324281704,
+    "tpot_p50_ms": 16.000888346459,
+    "tpot_p95_ms": 23.207303030310158,
+    "acceptance_rate": 0.3935291759720552,
+    "acceptance_pos_1": 0.6488919429651263,
+    "acceptance_pos_2": 0.35275725820305787,
+    "acceptance_pos_3": 0.17893832674798144,
+    "mean_acceptance_length": 2.1805875279161655
+  },
+  {
+    "dataset": "math500",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63622,
+    "request_throughput_rps": 1.7760438758773052,
+    "output_throughput_tps": 225.9909269421318,
+    "speedup_vs_baseline": 0.8472307039820806,
+    "latency_mean_ms": 2249.2024364179815,
+    "latency_p50_ms": 2185.3041670019593,
+    "latency_p95_ms": 3338.340588499702,
+    "latency_p99_ms": 4000.3402635519888,
+    "ttft_mean_ms": 263.0280147520243,
+    "ttft_p50_ms": 252.19799500155204,
+    "ttft_p95_ms": 475.0075314999774,
+    "ttft_p99_ms": 595.4660286509533,
+    "itl_mean_ms": 15.760428864173193,
+    "itl_p50_ms": 15.345385657476402,
+    "itl_p95_ms": 24.247727375981995,
+    "itl_p99_ms": 29.25197200714273,
+    "tpot_mean_ms": 15.762062569323362,
+    "tpot_p50_ms": 15.347322909449966,
+    "tpot_p95_ms": 24.248864364582953,
+    "acceptance_rate": 0.18693377898042707,
+    "acceptance_pos_1": 0.6466625620223824,
+    "acceptance_pos_2": 0.3548223534098729,
+    "acceptance_pos_3": 0.17529245590525516,
+    "mean_acceptance_length": 2.3085364528629895
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 63636,
+    "request_throughput_rps": 2.2498034619335257,
+    "output_throughput_tps": 286.33698620720367,
+    "speedup_vs_baseline": 1.0734656018405375,
+    "latency_mean_ms": 1774.1428698560267,
+    "latency_p50_ms": 1733.4975750000012,
+    "latency_p95_ms": 2512.932257099783,
+    "latency_p99_ms": 2941.1046552174953,
+    "ttft_mean_ms": 237.00501048202568,
+    "ttft_p50_ms": 212.508828000864,
+    "ttft_p95_ms": 434.44543489968055,
+    "ttft_p99_ms": 581.4404782277412,
+    "itl_mean_ms": 12.18272455456093,
+    "itl_p50_ms": 11.912186838580821,
+    "itl_p95_ms": 17.877204798435372,
+    "itl_p99_ms": 22.473681951484796,
+    "tpot_mean_ms": 12.18441765344298,
+    "tpot_p50_ms": 11.913581936995776,
+    "tpot_p95_ms": 17.87863581495454,
+    "acceptance_rate": 0.4499175952574226,
+    "acceptance_pos_1": 0.6567411999114456,
+    "acceptance_pos_2": 0.42247804590067156,
+    "acceptance_pos_3": 0.27053353996015056,
+    "mean_acceptance_length": 2.3497527857722678
+  },
+  {
+    "dataset": "math500",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 63621,
+    "request_throughput_rps": 2.316449613127783,
+    "output_throughput_tps": 294.7496816736054,
+    "speedup_vs_baseline": 1.1050044516467135,
+    "latency_mean_ms": 1725.2525323180525,
+    "latency_p50_ms": 1675.7616530012456,
+    "latency_p95_ms": 2646.2700289983313,
+    "latency_p99_ms": 3227.1719797111646,
+    "ttft_mean_ms": 247.76108907210437,
+    "ttft_p50_ms": 221.10979600074643,
+    "ttft_p95_ms": 461.0275230510521,
+    "ttft_p99_ms": 670.7492681724033,
+    "itl_mean_ms": 11.723149998862587,
+    "itl_p50_ms": 11.28927224801989,
+    "itl_p95_ms": 18.88152986101506,
+    "itl_p99_ms": 22.771947493768472,
+    "tpot_mean_ms": 11.72477114787681,
+    "tpot_p50_ms": 11.290766421259073,
+    "tpot_p95_ms": 18.882649400401576,
+    "acceptance_rate": 0.22890383017299992,
+    "acceptance_pos_1": 0.6397510474718301,
+    "acceptance_pos_2": 0.3964121547410812,
+    "acceptance_pos_3": 0.24219989423585406,
+    "mean_acceptance_length": 2.6023268112109994
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 1319,
+    "output_tokens": 168502,
+    "request_throughput_rps": 2.196360834095345,
+    "output_throughput_tps": 280.5846802628763,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1819.2423071760666,
+    "latency_p50_ms": 1798.3721370110288,
+    "latency_p95_ms": 2167.7596873778384,
+    "latency_p99_ms": 2310.5863884446435,
+    "ttft_mean_ms": 217.04894016683772,
+    "ttft_p50_ms": 191.49014502181672,
+    "ttft_p95_ms": 398.45379750477116,
+    "ttft_p99_ms": 646.2297889695037,
+    "itl_mean_ms": 12.638841870584082,
+    "itl_p50_ms": 12.576321425240883,
+    "itl_p95_ms": 15.217476899920795,
+    "itl_p99_ms": 16.315808897562086,
+    "tpot_mean_ms": 12.640636354922657,
+    "tpot_p50_ms": 12.58000173247566,
+    "tpot_p95_ms": 15.218959284321164,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168356,
+    "request_throughput_rps": 1.4790952868779552,
+    "output_throughput_tps": 188.79042162064067,
+    "speedup_vs_baseline": 0.6728465055318247,
+    "latency_mean_ms": 2702.3829456360854,
+    "latency_p50_ms": 2683.6698380000144,
+    "latency_p95_ms": 3317.1016436999594,
+    "latency_p99_ms": 3597.6961572799746,
+    "ttft_mean_ms": 235.6068064935536,
+    "ttft_p50_ms": 221.15605599992705,
+    "ttft_p95_ms": 424.62412700004944,
+    "ttft_p99_ms": 526.131548720104,
+    "itl_mean_ms": 19.477842147847973,
+    "itl_p50_ms": 19.355762133858388,
+    "itl_p95_ms": 24.183653737845873,
+    "itl_p99_ms": 26.07432056251997,
+    "tpot_mean_ms": 19.479491279517458,
+    "tpot_p50_ms": 19.357246826771867,
+    "tpot_p95_ms": 24.18510518785972,
+    "acceptance_rate": 0.0680806946304376,
+    "acceptance_pos_1": 0.18367244085417656,
+    "acceptance_pos_2": 0.018568116693305686,
+    "acceptance_pos_3": 0.002001526343830547,
+    "mean_acceptance_length": 1.2042420838913128
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168367,
+    "request_throughput_rps": 1.4764153266187332,
+    "output_throughput_tps": 188.46066663898122,
+    "speedup_vs_baseline": 0.6716712632436481,
+    "latency_mean_ms": 2705.295749132689,
+    "latency_p50_ms": 2679.983797999739,
+    "latency_p95_ms": 3477.261450399509,
+    "latency_p99_ms": 4079.732661799678,
+    "ttft_mean_ms": 301.1807209363258,
+    "ttft_p50_ms": 224.2310860001453,
+    "ttft_p95_ms": 834.4522548001792,
+    "ttft_p99_ms": 1980.7203211395963,
+    "itl_mean_ms": 18.978770912763007,
+    "itl_p50_ms": 19.034192118114312,
+    "itl_p95_ms": 23.592571637004227,
+    "itl_p99_ms": 25.352459187087952,
+    "tpot_mean_ms": 18.980617877378993,
+    "tpot_p50_ms": 19.03604929134238,
+    "tpot_p95_ms": 23.594471511024295,
+    "acceptance_rate": 0.027380290750817878,
+    "acceptance_pos_1": 0.17135507958901572,
+    "acceptance_pos_2": 0.018069627919570206,
+    "acceptance_pos_3": 0.001987944081056817,
+    "mean_acceptance_length": 1.1916620352557252
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168415,
+    "request_throughput_rps": 1.574435488473932,
+    "output_throughput_tps": 201.0299869532504,
+    "speedup_vs_baseline": 0.7164681505950642,
+    "latency_mean_ms": 2536.437202144023,
+    "latency_p50_ms": 2473.5267859996384,
+    "latency_p95_ms": 3534.031745998982,
+    "latency_p99_ms": 4064.8575092010615,
+    "ttft_mean_ms": 258.1910506959681,
+    "ttft_p50_ms": 238.57910900005663,
+    "ttft_p95_ms": 484.66444300011057,
+    "ttft_p99_ms": 598.7782853190582,
+    "itl_mean_ms": 17.977036765777857,
+    "itl_p50_ms": 17.445515787392107,
+    "itl_p95_ms": 25.81660302204809,
+    "itl_p99_ms": 30.753264190233132,
+    "tpot_mean_ms": 17.978638165785107,
+    "tpot_p50_ms": 17.446966267718064,
+    "tpot_p95_ms": 25.818113115744246,
+    "acceptance_rate": 0.2794753137068277,
+    "acceptance_pos_1": 0.536425000273502,
+    "acceptance_pos_2": 0.21512575623311125,
+    "acceptance_pos_3": 0.08687518461386984,
+    "mean_acceptance_length": 1.838425941120483
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168400,
+    "request_throughput_rps": 1.2094482721851378,
+    "output_throughput_tps": 154.41325931461503,
+    "speedup_vs_baseline": 0.5503267647041427,
+    "latency_mean_ms": 3304.2710242100056,
+    "latency_p50_ms": 3064.5563209982356,
+    "latency_p95_ms": 5885.097530001435,
+    "latency_p99_ms": 7240.322633720641,
+    "ttft_mean_ms": 1205.8129131272765,
+    "ttft_p50_ms": 677.296675999969,
+    "ttft_p95_ms": 4089.5798977991944,
+    "ttft_p99_ms": 5644.379561060411,
+    "itl_mean_ms": 16.556715234828605,
+    "itl_p50_ms": 16.74034803936521,
+    "itl_p95_ms": 27.050037795280637,
+    "itl_p99_ms": 35.51621275700378,
+    "tpot_mean_ms": 16.558478447758667,
+    "tpot_p50_ms": 16.743846314973666,
+    "tpot_p95_ms": 27.058657145679476,
+    "acceptance_rate": 0.12825457741085033,
+    "acceptance_pos_1": 0.5421524916756024,
+    "acceptance_pos_2": 0.21891754613691516,
+    "acceptance_pos_3": 0.09374118178226762,
+    "mean_acceptance_length": 1.8977820418759523
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 1319,
+    "output_tokens": 168413,
+    "request_throughput_rps": 2.0975172075679485,
+    "output_throughput_tps": 267.81589497963677,
+    "speedup_vs_baseline": 0.954492222200882,
+    "latency_mean_ms": 1904.8592331402729,
+    "latency_p50_ms": 1849.926056998811,
+    "latency_p95_ms": 2717.593596396909,
+    "latency_p99_ms": 3128.5920352824137,
+    "ttft_mean_ms": 232.41419941998183,
+    "ttft_p50_ms": 211.30282399826683,
+    "ttft_p95_ms": 436.5587686021171,
+    "ttft_p99_ms": 608.768046818439,
+    "itl_mean_ms": 13.20091798661006,
+    "itl_p50_ms": 12.809261669308327,
+    "itl_p95_ms": 19.22371915434054,
+    "itl_p99_ms": 22.673929599071368,
+    "tpot_mean_ms": 13.202611622505504,
+    "tpot_p50_ms": 12.810728181104036,
+    "tpot_p95_ms": 19.225053650402195,
+    "acceptance_rate": 0.31145141140312554,
+    "acceptance_pos_1": 0.5333057414177645,
+    "acceptance_pos_2": 0.2575245453082247,
+    "acceptance_pos_3": 0.14352394748338737,
+    "mean_acceptance_length": 1.9343542342093767
+  },
+  {
+    "dataset": "gsm8k",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 1319,
+    "output_tokens": 168420,
+    "request_throughput_rps": 2.0069879067415446,
+    "output_throughput_tps": 256.2675536417065,
+    "speedup_vs_baseline": 0.9133340900922055,
+    "latency_mean_ms": 1991.4858750174596,
+    "latency_p50_ms": 1940.2353619989299,
+    "latency_p95_ms": 2946.936268898934,
+    "latency_p99_ms": 3409.506298801133,
+    "ttft_mean_ms": 240.2313964965955,
+    "ttft_p50_ms": 217.0051639986923,
+    "ttft_p95_ms": 457.4054478005564,
+    "ttft_p99_ms": 621.9962791216665,
+    "itl_mean_ms": 13.815387803215751,
+    "itl_p50_ms": 13.491749685057384,
+    "itl_p95_ms": 21.069445844088154,
+    "itl_p99_ms": 24.68408968393126,
+    "tpot_mean_ms": 13.817049546266604,
+    "tpot_p50_ms": 13.493896850398981,
+    "tpot_p95_ms": 21.07079229370886,
+    "acceptance_rate": 0.1463064478504878,
+    "acceptance_pos_1": 0.5319253673998655,
+    "acceptance_pos_2": 0.2490394774757468,
+    "acceptance_pos_3": 0.13306838920372682,
+    "mean_acceptance_length": 2.0241451349534145
+  },
+  {
+    "dataset": "humaneval",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 164,
+    "output_tokens": 11229,
+    "request_throughput_rps": 2.83651766504482,
+    "output_throughput_tps": 194.21498085846514,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1403.542471638946,
+    "latency_p50_ms": 1171.1717744910857,
+    "latency_p95_ms": 2734.2781511018984,
+    "latency_p99_ms": 3048.235554893327,
+    "ttft_mean_ms": 218.20821551103487,
+    "ttft_p50_ms": 209.54365949728526,
+    "ttft_p95_ms": 420.2904565172503,
+    "ttft_p99_ms": 467.85162955609854,
+    "itl_mean_ms": 17.843567054063506,
+    "itl_p50_ms": 17.024532066210227,
+    "itl_p95_ms": 29.900638077389125,
+    "itl_p99_ms": 38.85421750062092,
+    "tpot_mean_ms": 17.849692097307134,
+    "tpot_p50_ms": 17.028038263717008,
+    "tpot_p95_ms": 29.90535920137036,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11292,
+    "request_throughput_rps": 2.1388107405014773,
+    "output_throughput_tps": 147.26494440087,
+    "speedup_vs_baseline": 0.7582573895686753,
+    "latency_mean_ms": 1856.8446467317099,
+    "latency_p50_ms": 1529.8299214999815,
+    "latency_p95_ms": 3609.9989535499276,
+    "latency_p99_ms": 4076.3032318901037,
+    "ttft_mean_ms": 250.89464046951855,
+    "ttft_p50_ms": 233.70097149984304,
+    "ttft_p95_ms": 497.8459723500346,
+    "ttft_p99_ms": 638.7824368799534,
+    "itl_mean_ms": 24.549073372278432,
+    "itl_p50_ms": 22.60172811232828,
+    "itl_p95_ms": 42.012251182637584,
+    "itl_p99_ms": 51.31418752062339,
+    "tpot_mean_ms": 24.555336995750512,
+    "tpot_p50_ms": 22.603792561644838,
+    "tpot_p95_ms": 42.01500447592228,
+    "acceptance_rate": 0.059506172839506176,
+    "acceptance_pos_1": 0.16412698412698412,
+    "acceptance_pos_2": 0.013544973544973546,
+    "acceptance_pos_3": 0.0008465608465608466,
+    "mean_acceptance_length": 1.1785185185185185
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11052,
+    "request_throughput_rps": 2.1057562192267367,
+    "output_throughput_tps": 141.90742521276766,
+    "speedup_vs_baseline": 0.7306718801274306,
+    "latency_mean_ms": 1877.2794746829145,
+    "latency_p50_ms": 1504.3869174996871,
+    "latency_p95_ms": 3978.088593899155,
+    "latency_p99_ms": 4819.554965540591,
+    "ttft_mean_ms": 343.89938129273037,
+    "ttft_p50_ms": 254.9212224994335,
+    "ttft_p95_ms": 982.0552947493524,
+    "ttft_p99_ms": 1265.2674350103007,
+    "itl_mean_ms": 22.96417710011784,
+    "itl_p50_ms": 22.33989236389506,
+    "itl_p95_ms": 33.208631611878445,
+    "itl_p99_ms": 43.04405845282055,
+    "tpot_mean_ms": 22.970011297649936,
+    "tpot_p50_ms": 22.34099295422598,
+    "tpot_p95_ms": 33.23066878742338,
+    "acceptance_rate": 0.02684898226160627,
+    "acceptance_pos_1": 0.17398888040989863,
+    "acceptance_pos_2": 0.013081870707511174,
+    "acceptance_pos_3": 0.0008721247138340783,
+    "mean_acceptance_length": 1.1879428758312438
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11277,
+    "request_throughput_rps": 2.8294040460667382,
+    "output_throughput_tps": 194.55603309447932,
+    "speedup_vs_baseline": 1.0017560552461333,
+    "latency_mean_ms": 1401.6537050121503,
+    "latency_p50_ms": 1262.2241380004198,
+    "latency_p95_ms": 2758.0062766009178,
+    "latency_p99_ms": 3165.725670439934,
+    "ttft_mean_ms": 307.73021359141796,
+    "ttft_p50_ms": 282.3090209994916,
+    "ttft_p95_ms": 627.4767150008302,
+    "ttft_p99_ms": 760.1846476996616,
+    "itl_mean_ms": 17.775962665717213,
+    "itl_p50_ms": 16.24701220755945,
+    "itl_p95_ms": 33.04292447846012,
+    "itl_p99_ms": 49.24142361158533,
+    "tpot_mean_ms": 17.781789938797445,
+    "tpot_p50_ms": 16.250892179999738,
+    "tpot_p95_ms": 33.04544840657803,
+    "acceptance_rate": 0.5873682476943346,
+    "acceptance_pos_1": 0.7734683794466403,
+    "acceptance_pos_2": 0.5642292490118577,
+    "acceptance_pos_3": 0.4244071146245059,
+    "mean_acceptance_length": 2.762104743083004
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11047,
+    "request_throughput_rps": 3.0665775841960174,
+    "output_throughput_tps": 206.5639181256915,
+    "speedup_vs_baseline": 1.0635838554402026,
+    "latency_mean_ms": 1296.2304299939337,
+    "latency_p50_ms": 1225.3228209992812,
+    "latency_p95_ms": 2684.5432901496665,
+    "latency_p99_ms": 3137.9315633195806,
+    "ttft_mean_ms": 395.1852547253976,
+    "ttft_p50_ms": 324.23020049827755,
+    "ttft_p95_ms": 956.4745560996016,
+    "ttft_p99_ms": 1385.3407927481385,
+    "itl_mean_ms": 16.226717747252554,
+    "itl_p50_ms": 13.57385855245207,
+    "itl_p95_ms": 32.95489575833981,
+    "itl_p99_ms": 75.28418661784087,
+    "tpot_mean_ms": 16.23359719581637,
+    "tpot_p50_ms": 13.576510465236616,
+    "tpot_p95_ms": 32.96468897523092,
+    "acceptance_rate": 0.33047302832995556,
+    "acceptance_pos_1": 0.7790522185330516,
+    "acceptance_pos_2": 0.5644431029278599,
+    "acceptance_pos_3": 0.4044672502263809,
+    "mean_acceptance_length": 3.3133111983096892
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 164,
+    "output_tokens": 11346,
+    "request_throughput_rps": 3.393797033660498,
+    "output_throughput_tps": 234.79281185312203,
+    "speedup_vs_baseline": 1.2089325489480554,
+    "latency_mean_ms": 1172.9745371464626,
+    "latency_p50_ms": 1033.1892974991206,
+    "latency_p95_ms": 2107.933460299682,
+    "latency_p99_ms": 3156.528969858848,
+    "ttft_mean_ms": 265.29695418294733,
+    "ttft_p50_ms": 228.4132034983486,
+    "ttft_p95_ms": 502.8849432006609,
+    "ttft_p99_ms": 699.91048091968,
+    "itl_mean_ms": 14.77773082701616,
+    "itl_p50_ms": 13.023184389006781,
+    "itl_p95_ms": 29.633302174848463,
+    "itl_p99_ms": 45.321116141877226,
+    "tpot_mean_ms": 14.783925134182368,
+    "tpot_p50_ms": 13.039449575903792,
+    "tpot_p95_ms": 29.64001845995613,
+    "acceptance_rate": 0.5991372158619546,
+    "acceptance_pos_1": 0.7653061224489796,
+    "acceptance_pos_2": 0.587356893977103,
+    "acceptance_pos_3": 0.44474863115978097,
+    "mean_acceptance_length": 2.7974116475858635
+  },
+  {
+    "dataset": "humaneval",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 164,
+    "output_tokens": 11058,
+    "request_throughput_rps": 3.1234019762512726,
+    "output_throughput_tps": 210.6010917889425,
+    "speedup_vs_baseline": 1.08437099372071,
+    "latency_mean_ms": 1276.6294960669388,
+    "latency_p50_ms": 1098.64930799813,
+    "latency_p95_ms": 2616.217871799927,
+    "latency_p99_ms": 3188.0119062716053,
+    "ttft_mean_ms": 564.2550010548891,
+    "ttft_p50_ms": 336.2399385005119,
+    "ttft_p95_ms": 1681.4955957521306,
+    "ttft_p99_ms": 2938.6766085621284,
+    "itl_mean_ms": 12.486812538079404,
+    "itl_p50_ms": 11.807729195672675,
+    "itl_p95_ms": 25.278697270681064,
+    "itl_p99_ms": 43.8447163140126,
+    "tpot_mean_ms": 12.494207252966246,
+    "tpot_p50_ms": 11.83652314342243,
+    "tpot_p95_ms": 25.281766504248417,
+    "acceptance_rate": 0.35094729280775794,
+    "acceptance_pos_1": 0.740100565681961,
+    "acceptance_pos_2": 0.5502828409805154,
+    "acceptance_pos_3": 0.39849151477058453,
+    "mean_acceptance_length": 3.4566310496543053
+  },
+  {
+    "dataset": "mbpp",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 500,
+    "output_tokens": 61587,
+    "request_throughput_rps": 2.23460184339787,
+    "output_throughput_tps": 275.2448474586892,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1788.0222868800047,
+    "latency_p50_ms": 1819.359868997708,
+    "latency_p95_ms": 2216.346881000209,
+    "latency_p99_ms": 2362.3936853421037,
+    "ttft_mean_ms": 221.07102080830373,
+    "ttft_p50_ms": 204.21197451651096,
+    "ttft_p95_ms": 398.6740397042012,
+    "ttft_p99_ms": 655.3067808030755,
+    "itl_mean_ms": 13.201886877023522,
+    "itl_p50_ms": 12.736825452678412,
+    "itl_p95_ms": 15.664009772456069,
+    "itl_p99_ms": 17.321856500478695,
+    "tpot_mean_ms": 13.205948736768402,
+    "tpot_p50_ms": 12.738972996026057,
+    "tpot_p95_ms": 15.67098670536386,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61808,
+    "request_throughput_rps": 1.5365610042290248,
+    "output_throughput_tps": 189.94352509877513,
+    "speedup_vs_baseline": 0.690089303587357,
+    "latency_mean_ms": 2595.6573488480017,
+    "latency_p50_ms": 2617.603001500015,
+    "latency_p95_ms": 3320.9663893999277,
+    "latency_p99_ms": 3516.577585000016,
+    "ttft_mean_ms": 234.1106791739985,
+    "ttft_p50_ms": 218.79009800011318,
+    "ttft_p95_ms": 439.53066730014143,
+    "ttft_p99_ms": 544.9614302001191,
+    "itl_mean_ms": 19.14316073130201,
+    "itl_p50_ms": 18.9773965472434,
+    "itl_p95_ms": 23.96081993622043,
+    "itl_p99_ms": 25.695518167638486,
+    "tpot_mean_ms": 19.146981896371145,
+    "tpot_p50_ms": 18.978798267717032,
+    "tpot_p95_ms": 23.96214876692921,
+    "acceptance_rate": 0.06462320419065401,
+    "acceptance_pos_1": 0.1824529329391629,
+    "acceptance_pos_2": 0.01106659405632488,
+    "acceptance_pos_3": 0.0003500855764742493,
+    "mean_acceptance_length": 1.1938696125719621
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61377,
+    "request_throughput_rps": 1.387242813763742,
+    "output_throughput_tps": 170.2896043607544,
+    "speedup_vs_baseline": 0.6186840768611036,
+    "latency_mean_ms": 2875.578809605966,
+    "latency_p50_ms": 2855.830152499948,
+    "latency_p95_ms": 4074.322934499559,
+    "latency_p99_ms": 4764.013334749542,
+    "ttft_mean_ms": 634.0442261159515,
+    "ttft_p50_ms": 392.3641729988958,
+    "ttft_p95_ms": 1885.916530999566,
+    "ttft_p99_ms": 2897.4566687989864,
+    "itl_mean_ms": 18.201249949891572,
+    "itl_p50_ms": 18.463196299214943,
+    "itl_p95_ms": 24.69457239645406,
+    "itl_p99_ms": 30.63269140497326,
+    "tpot_mean_ms": 18.205769807911185,
+    "tpot_p50_ms": 18.464951303145412,
+    "tpot_p95_ms": 24.695845573618083,
+    "acceptance_rate": 0.028501713363674443,
+    "acceptance_pos_1": 0.1877840964993408,
+    "acceptance_pos_2": 0.011255632735787795,
+    "acceptance_pos_3": 0.0004525866309844743,
+    "mean_acceptance_length": 1.199511993545721
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61640,
+    "request_throughput_rps": 1.9801835796759275,
+    "output_throughput_tps": 244.11703170244834,
+    "speedup_vs_baseline": 0.8869086340992713,
+    "latency_mean_ms": 2016.50626224402,
+    "latency_p50_ms": 2021.2945930006754,
+    "latency_p95_ms": 2791.478140551044,
+    "latency_p99_ms": 3086.5239088800263,
+    "ttft_mean_ms": 280.0661946479813,
+    "ttft_p50_ms": 256.2331505005204,
+    "ttft_p95_ms": 557.8820319009537,
+    "ttft_p99_ms": 769.5084362501256,
+    "itl_mean_ms": 14.264965995399256,
+    "itl_p50_ms": 14.109097578741029,
+    "itl_p95_ms": 19.439541319448956,
+    "itl_p99_ms": 23.81797452189929,
+    "tpot_mean_ms": 14.269046870264877,
+    "tpot_p50_ms": 14.11056187795813,
+    "tpot_p95_ms": 19.634308889327635,
+    "acceptance_rate": 0.47048386886358,
+    "acceptance_pos_1": 0.6888967163711792,
+    "acceptance_pos_2": 0.44499236820476695,
+    "acceptance_pos_3": 0.2775625220147939,
+    "mean_acceptance_length": 2.41145160659074
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61475,
+    "request_throughput_rps": 1.503343533132173,
+    "output_throughput_tps": 184.83608739860065,
+    "speedup_vs_baseline": 0.6715333242572042,
+    "latency_mean_ms": 2657.845550606042,
+    "latency_p50_ms": 2483.5515449995,
+    "latency_p95_ms": 4488.94517174831,
+    "latency_p99_ms": 5523.38156365822,
+    "ttft_mean_ms": 1190.47147037393,
+    "ttft_p50_ms": 932.4905619996571,
+    "ttft_p95_ms": 3259.4635773521077,
+    "ttft_p99_ms": 4279.743841777054,
+    "itl_mean_ms": 12.451201825429687,
+    "itl_p50_ms": 12.228331582672416,
+    "itl_p95_ms": 22.859363721256052,
+    "itl_p99_ms": 36.967242600596585,
+    "tpot_mean_ms": 12.456394792296264,
+    "tpot_p50_ms": 12.230752885830222,
+    "tpot_p95_ms": 22.860772319684717,
+    "acceptance_rate": 0.24657171832973235,
+    "acceptance_pos_1": 0.6712818025486132,
+    "acceptance_pos_2": 0.42739979716918736,
+    "acceptance_pos_3": 0.2544644825609595,
+    "mean_acceptance_length": 2.7260020283081268
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 500,
+    "output_tokens": 61981,
+    "request_throughput_rps": 2.507209609073491,
+    "output_throughput_tps": 310.7987175599681,
+    "speedup_vs_baseline": 1.1291717916958104,
+    "latency_mean_ms": 1592.6791832899762,
+    "latency_p50_ms": 1603.4026140005153,
+    "latency_p95_ms": 2178.7681314988727,
+    "latency_p99_ms": 2552.1267430388243,
+    "ttft_mean_ms": 240.7260946480019,
+    "ttft_p50_ms": 224.12168300070334,
+    "ttft_p95_ms": 455.65863744814123,
+    "ttft_p99_ms": 601.2736888401558,
+    "itl_mean_ms": 11.346454396871485,
+    "itl_p50_ms": 10.845360450802424,
+    "itl_p95_ms": 15.682580235835754,
+    "itl_p99_ms": 23.395322236705972,
+    "tpot_mean_ms": 11.350096012527779,
+    "tpot_p50_ms": 10.847383575174785,
+    "tpot_p95_ms": 15.684506253152122,
+    "acceptance_rate": 0.5228963898667771,
+    "acceptance_pos_1": 0.7125284738041002,
+    "acceptance_pos_2": 0.5082211638020294,
+    "acceptance_pos_3": 0.3479395319942017,
+    "mean_acceptance_length": 2.568689169600331
+  },
+  {
+    "dataset": "mbpp",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 500,
+    "output_tokens": 61632,
+    "request_throughput_rps": 2.2269171937357544,
+    "output_throughput_tps": 274.498720968644,
+    "speedup_vs_baseline": 0.997289226312739,
+    "latency_mean_ms": 1794.3305799321024,
+    "latency_p50_ms": 1694.5109794996824,
+    "latency_p95_ms": 3171.435100350026,
+    "latency_p99_ms": 4215.3635372184735,
+    "ttft_mean_ms": 643.143013108056,
+    "ttft_p50_ms": 362.4557425009698,
+    "ttft_p95_ms": 2119.0080873015154,
+    "ttft_p99_ms": 3187.935962892621,
+    "itl_mean_ms": 10.101565495090881,
+    "itl_p50_ms": 9.322213830710384,
+    "itl_p95_ms": 14.88066484486098,
+    "itl_p99_ms": 19.67402437656759,
+    "tpot_mean_ms": 10.105878412845934,
+    "tpot_p50_ms": 9.323771708668072,
+    "tpot_p95_ms": 14.882234868113521,
+    "acceptance_rate": 0.28831800034716193,
+    "acceptance_pos_1": 0.6831591737545565,
+    "acceptance_pos_2": 0.47018226002430136,
+    "acceptance_pos_3": 0.3054678007290401,
+    "mean_acceptance_length": 3.0182260024301337
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 2.461309935234548,
+    "output_throughput_tps": 309.2328269880305,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 1607.6186366983165,
+    "latency_p50_ms": 1548.2021450006869,
+    "latency_p95_ms": 2284.110534451611,
+    "latency_p99_ms": 2534.602298826794,
+    "ttft_mean_ms": 320.49049263077904,
+    "ttft_p50_ms": 293.138627501321,
+    "ttft_p95_ms": 629.1677434302974,
+    "ttft_p99_ms": 759.140422597702,
+    "itl_mean_ms": 11.007379475699068,
+    "itl_p50_ms": 9.985256696983395,
+    "itl_p95_ms": 15.183904158686165,
+    "itl_p99_ms": 26.580996584276846,
+    "tpot_mean_ms": 11.014195983013959,
+    "tpot_p50_ms": 9.988513000022554,
+    "tpot_p95_ms": 15.18576166809785,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 1.5473009629136512,
+    "output_throughput_tps": 195.61752423635835,
+    "speedup_vs_baseline": 0.632589774318915,
+    "latency_mean_ms": 2573.229441274998,
+    "latency_p50_ms": 2591.1725164999098,
+    "latency_p95_ms": 3166.3267970500783,
+    "latency_p99_ms": 3608.086452589978,
+    "ttft_mean_ms": 221.35167090626453,
+    "ttft_p50_ms": 203.15486550020978,
+    "ttft_p95_ms": 448.1615023500803,
+    "ttft_p99_ms": 696.169536259867,
+    "itl_mean_ms": 18.676105907776027,
+    "itl_p50_ms": 18.64073133858227,
+    "itl_p95_ms": 23.103904855904904,
+    "itl_p99_ms": 25.284028265118394,
+    "tpot_mean_ms": 18.68020851087627,
+    "tpot_p50_ms": 18.641842641731216,
+    "tpot_p95_ms": 23.105400170866087,
+    "acceptance_rate": 0.060748671490479834,
+    "acceptance_pos_1": 0.16453909053473734,
+    "acceptance_pos_2": 0.01705982704864992,
+    "acceptance_pos_3": 0.0006470968880522384,
+    "mean_acceptance_length": 1.1822460144714395
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.4356652700068615,
+    "output_throughput_tps": 180.37339536048705,
+    "speedup_vs_baseline": 0.5832931681844656,
+    "latency_mean_ms": 2753.9001531249596,
+    "latency_p50_ms": 2798.7496705000012,
+    "latency_p95_ms": 3791.8684454991308,
+    "latency_p99_ms": 4289.662929459572,
+    "ttft_mean_ms": 381.8320665374472,
+    "ttft_p50_ms": 295.44119050024165,
+    "ttft_p95_ms": 1026.5876399000251,
+    "ttft_p99_ms": 1367.1522384703346,
+    "itl_mean_ms": 21.709309200685574,
+    "itl_p50_ms": 18.78574032282878,
+    "itl_p95_ms": 25.575840727953157,
+    "itl_p99_ms": 59.525236419324514,
+    "tpot_mean_ms": 21.713930513288513,
+    "tpot_p50_ms": 18.787575688973476,
+    "tpot_p95_ms": 25.57992160196576,
+    "acceptance_rate": 0.024768061794215188,
+    "acceptance_pos_1": 0.15474581251836614,
+    "acceptance_pos_2": 0.016044666470761092,
+    "acceptance_pos_3": 0.0012342051131354686,
+    "mean_acceptance_length": 1.1733764325595064
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 1.7821635622658862,
+    "output_throughput_tps": 225.31002835946467,
+    "speedup_vs_baseline": 0.7286096710818666,
+    "latency_mean_ms": 2231.2470228500047,
+    "latency_p50_ms": 2197.780867999427,
+    "latency_p95_ms": 3333.7097092000463,
+    "latency_p99_ms": 3700.461037129698,
+    "ttft_mean_ms": 248.96705823127832,
+    "ttft_p50_ms": 215.34929249992274,
+    "ttft_p95_ms": 520.0022110006103,
+    "ttft_p99_ms": 662.6540237599145,
+    "itl_mean_ms": 15.77862160127276,
+    "itl_p50_ms": 15.232473944878599,
+    "itl_p95_ms": 23.602753981101593,
+    "itl_p99_ms": 27.57565280062838,
+    "tpot_mean_ms": 15.782222927801888,
+    "tpot_p50_ms": 15.23403437795065,
+    "tpot_p95_ms": 23.60411872952008,
+    "acceptance_rate": 0.3229694606887589,
+    "acceptance_pos_1": 0.5750487329434698,
+    "acceptance_pos_2": 0.27573099415204677,
+    "acceptance_pos_3": 0.11812865497076024,
+    "mean_acceptance_length": 1.9689083820662767
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.0658824478410331,
+    "output_throughput_tps": 133.9148060406278,
+    "speedup_vs_baseline": 0.4330549487419434,
+    "latency_mean_ms": 3752.528322143644,
+    "latency_p50_ms": 3464.8722160018224,
+    "latency_p95_ms": 6717.973978601185,
+    "latency_p99_ms": 7876.092720710149,
+    "ttft_mean_ms": 1934.0097326686418,
+    "ttft_p50_ms": 1588.4214640009304,
+    "ttft_p95_ms": 4820.7839886515985,
+    "ttft_p99_ms": 6494.002312830161,
+    "itl_mean_ms": 14.317043586959333,
+    "itl_p50_ms": 13.67209011023508,
+    "itl_p95_ms": 29.82729992639314,
+    "itl_p99_ms": 39.377268431499964,
+    "tpot_mean_ms": 14.323458562974045,
+    "tpot_p50_ms": 13.673784755900577,
+    "tpot_p95_ms": 29.829240646451908,
+    "acceptance_rate": 0.14578666207134242,
+    "acceptance_pos_1": 0.5689585846401287,
+    "acceptance_pos_2": 0.26940088459991957,
+    "acceptance_pos_3": 0.11540008041817451,
+    "mean_acceptance_length": 2.020506634499397
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 160,
+    "output_tokens": 20228,
+    "request_throughput_rps": 2.2379435718508462,
+    "output_throughput_tps": 282.9320160712432,
+    "speedup_vs_baseline": 0.9149481923605564,
+    "latency_mean_ms": 1771.671463362577,
+    "latency_p50_ms": 1775.1809370001865,
+    "latency_p95_ms": 2604.8437047013063,
+    "latency_p99_ms": 3118.5169013906484,
+    "ttft_mean_ms": 252.11589151258522,
+    "ttft_p50_ms": 227.79171349975513,
+    "ttft_p95_ms": 503.8048422973585,
+    "ttft_p99_ms": 627.0037127103565,
+    "itl_mean_ms": 12.360936149822532,
+    "itl_p50_ms": 11.83748445670406,
+    "itl_p95_ms": 19.148658472828007,
+    "itl_p99_ms": 25.009972698654565,
+    "tpot_mean_ms": 12.364664553850584,
+    "tpot_p50_ms": 11.838727370086703,
+    "tpot_p95_ms": 19.150851622824714,
+    "acceptance_rate": 0.36540854417254254,
+    "acceptance_pos_1": 0.5802571547075902,
+    "acceptance_pos_2": 0.3253836582330983,
+    "acceptance_pos_3": 0.19058481957693904,
+    "mean_acceptance_length": 2.0962256325176276
+  },
+  {
+    "dataset": "mt_bench",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 160,
+    "output_tokens": 20102,
+    "request_throughput_rps": 1.784863821370322,
+    "output_throughput_tps": 224.24582835741384,
+    "speedup_vs_baseline": 0.7251682511898833,
+    "latency_mean_ms": 2225.1736813125035,
+    "latency_p50_ms": 2117.7182369992806,
+    "latency_p95_ms": 3657.939555099983,
+    "latency_p99_ms": 4345.6770745698295,
+    "ttft_mean_ms": 838.8437911749861,
+    "ttft_p50_ms": 589.9439045006147,
+    "ttft_p95_ms": 2441.8948530525054,
+    "ttft_p99_ms": 2773.190703780456,
+    "itl_mean_ms": 11.426459321042842,
+    "itl_p50_ms": 10.71108703149011,
+    "itl_p95_ms": 22.870212017662954,
+    "itl_p99_ms": 25.594833969671022,
+    "tpot_mean_ms": 11.431539335517765,
+    "tpot_p50_ms": 10.713255098423582,
+    "tpot_p95_ms": 23.0010987298119,
+    "acceptance_rate": 0.17215237263282687,
+    "acceptance_pos_1": 0.5529591613889495,
+    "acceptance_pos_2": 0.2997379340467351,
+    "acceptance_pos_3": 0.16739462764795807,
+    "mean_acceptance_length": 2.205066608429788
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "baseline",
+    "method": "baseline",
+    "num_speculative_tokens": null,
+    "requests": 5000,
+    "output_tokens": 635569,
+    "request_throughput_rps": 1.2230023684575178,
+    "output_throughput_tps": 155.4604784636352,
+    "speedup_vs_baseline": 1.0,
+    "latency_mean_ms": 3269.5651201336645,
+    "latency_p50_ms": 3276.8055605120026,
+    "latency_p95_ms": 3934.875892131822,
+    "latency_p99_ms": 4087.044628756121,
+    "ttft_mean_ms": 231.24218063543086,
+    "ttft_p50_ms": 218.61417350010015,
+    "ttft_p95_ms": 386.9901933081565,
+    "ttft_p99_ms": 553.9115706225857,
+    "itl_mean_ms": 24.08620830909222,
+    "itl_p50_ms": 24.040817992012823,
+    "itl_p95_ms": 28.90904858166638,
+    "itl_p99_ms": 29.91989293644905,
+    "tpot_mean_ms": 24.087996070253364,
+    "tpot_p50_ms": 24.042644452736308,
+    "tpot_p95_ms": 28.910471471294663,
+    "acceptance_rate": null,
+    "acceptance_pos_1": null,
+    "acceptance_pos_2": null,
+    "acceptance_pos_3": null,
+    "mean_acceptance_length": null
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k3",
+    "method": "dflash",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636175,
+    "request_throughput_rps": 1.0622074332559788,
+    "output_throughput_tps": 135.14996277032446,
+    "speedup_vs_baseline": 0.8693525461002507,
+    "latency_mean_ms": 3763.952480385199,
+    "latency_p50_ms": 3769.2712649998157,
+    "latency_p95_ms": 4515.89870830021,
+    "latency_p99_ms": 4805.0974385998825,
+    "ttft_mean_ms": 257.6036252754003,
+    "ttft_p50_ms": 246.59032999988995,
+    "ttft_p95_ms": 405.558623099887,
+    "ttft_p99_ms": 540.4809878199559,
+    "itl_mean_ms": 27.764084011389272,
+    "itl_p50_ms": 27.68626928346698,
+    "itl_p95_ms": 33.282436218505644,
+    "itl_p99_ms": 35.5445708588995,
+    "tpot_mean_ms": 27.765764208304795,
+    "tpot_p50_ms": 27.687906570868,
+    "tpot_p95_ms": 33.28350510590754,
+    "acceptance_rate": 0.07093823412968715,
+    "acceptance_pos_1": 0.1840484911746139,
+    "acceptance_pos_2": 0.02574222849873457,
+    "acceptance_pos_3": 0.0030239827157130055,
+    "mean_acceptance_length": 1.2128147023890614
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflash_k7",
+    "method": "dflash",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636064,
+    "request_throughput_rps": 1.0536069317356047,
+    "output_throughput_tps": 134.03228788549515,
+    "speedup_vs_baseline": 0.8621630990081349,
+    "latency_mean_ms": 3794.9240800698,
+    "latency_p50_ms": 3797.6578120005797,
+    "latency_p95_ms": 4543.265116648946,
+    "latency_p99_ms": 4838.748624158453,
+    "ttft_mean_ms": 255.93729505000047,
+    "ttft_p50_ms": 246.4688734989977,
+    "ttft_p95_ms": 399.62789619985415,
+    "ttft_p99_ms": 533.8578004602641,
+    "itl_mean_ms": 28.02829548006744,
+    "itl_p50_ms": 27.958994283470407,
+    "itl_p95_ms": 33.55774176338532,
+    "itl_p99_ms": 35.83546665536011,
+    "tpot_mean_ms": 28.030316558857468,
+    "tpot_p50_ms": 27.961111295281434,
+    "tpot_p95_ms": 33.55949413779617,
+    "acceptance_rate": 0.030697806232490035,
+    "acceptance_pos_1": 0.1853871675887231,
+    "acceptance_pos_2": 0.025839366107025186,
+    "acceptance_pos_3": 0.0030756573256390638,
+    "mean_acceptance_length": 1.2148846436274303
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k3",
+    "method": "dflare",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636228,
+    "request_throughput_rps": 1.267241365085764,
+    "output_throughput_tps": 161.25088784515708,
+    "speedup_vs_baseline": 1.0372468259376697,
+    "latency_mean_ms": 3155.8181194270032,
+    "latency_p50_ms": 3158.2490724995296,
+    "latency_p95_ms": 4013.4220384479713,
+    "latency_p99_ms": 4389.7799070399205,
+    "ttft_mean_ms": 282.9019767206311,
+    "ttft_p50_ms": 270.00340999984473,
+    "ttft_p95_ms": 456.05508385024234,
+    "ttft_p99_ms": 595.7626636705755,
+    "itl_mean_ms": 22.751411578135976,
+    "itl_p50_ms": 22.677721858262814,
+    "itl_p95_ms": 29.19536946929717,
+    "itl_p99_ms": 32.0268499520443,
+    "tpot_mean_ms": 22.753117775499057,
+    "tpot_p50_ms": 22.679262984259076,
+    "tpot_p95_ms": 29.196810057103683,
+    "acceptance_rate": 0.31057058500760176,
+    "acceptance_pos_1": 0.5710008489248474,
+    "acceptance_pos_2": 0.2600387645252867,
+    "acceptance_pos_3": 0.10067214157267132,
+    "mean_acceptance_length": 1.9317117550228053
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dflare_k7",
+    "method": "dflare",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 635796,
+    "request_throughput_rps": 1.2415631421695674,
+    "output_throughput_tps": 157.87617590776844,
+    "speedup_vs_baseline": 1.0155389811481785,
+    "latency_mean_ms": 3221.0815614180046,
+    "latency_p50_ms": 3164.543153499835,
+    "latency_p95_ms": 4123.292088050039,
+    "latency_p99_ms": 4616.265497698114,
+    "ttft_mean_ms": 352.63900740198807,
+    "ttft_p50_ms": 287.9059334991325,
+    "ttft_p95_ms": 612.3089999979129,
+    "ttft_p99_ms": 1196.3210188611995,
+    "itl_mean_ms": 22.7251370702789,
+    "itl_p50_ms": 22.450031862195097,
+    "itl_p95_ms": 29.62520190592225,
+    "itl_p99_ms": 33.215207786221775,
+    "tpot_mean_ms": 22.72683404563422,
+    "tpot_p50_ms": 22.451526275598177,
+    "tpot_p95_ms": 29.626681563398265,
+    "acceptance_rate": 0.1436652803641537,
+    "acceptance_pos_1": 0.5763152990672803,
+    "acceptance_pos_2": 0.26589619110489227,
+    "acceptance_pos_3": 0.10326562455582894,
+    "mean_acceptance_length": 2.005656962549076
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k3",
+    "method": "dspark",
+    "num_speculative_tokens": 3,
+    "requests": 5000,
+    "output_tokens": 636215,
+    "request_throughput_rps": 1.5004265390600509,
+    "output_throughput_tps": 190.91877410961806,
+    "speedup_vs_baseline": 1.2280855944636573,
+    "latency_mean_ms": 2665.1578789277846,
+    "latency_p50_ms": 2665.9706919999735,
+    "latency_p95_ms": 3441.952741001842,
+    "latency_p99_ms": 3755.308294759525,
+    "ttft_mean_ms": 264.3723180442066,
+    "ttft_p50_ms": 253.26144649989146,
+    "ttft_p95_ms": 429.1125219519018,
+    "ttft_p99_ms": 559.1942446103716,
+    "itl_mean_ms": 19.008812154313944,
+    "itl_p50_ms": 18.913669216526902,
+    "itl_p95_ms": 24.80109081614243,
+    "itl_p99_ms": 27.068464386929747,
+    "tpot_mean_ms": 19.010452079968594,
+    "tpot_p50_ms": 18.915089492132775,
+    "tpot_p95_ms": 24.80313036299977,
+    "acceptance_rate": 0.32401200895393567,
+    "acceptance_pos_1": 0.5557804733415505,
+    "acceptance_pos_2": 0.28182543551086175,
+    "acceptance_pos_3": 0.1344301180093948,
+    "mean_acceptance_length": 1.9720360268618071
+  },
+  {
+    "dataset": "ultrachat",
+    "run": "dspark_k7",
+    "method": "dspark",
+    "num_speculative_tokens": 7,
+    "requests": 5000,
+    "output_tokens": 636157,
+    "request_throughput_rps": 1.4902242443540412,
+    "output_throughput_tps": 189.60331692310675,
+    "speedup_vs_baseline": 1.2196239121151176,
+    "latency_mean_ms": 2683.3432791144196,
+    "latency_p50_ms": 2638.018617002672,
+    "latency_p95_ms": 3469.7974141512536,
+    "latency_p99_ms": 3857.7436888694515,
+    "ttft_mean_ms": 287.66377756739894,
+    "ttft_p50_ms": 259.45143649914826,
+    "ttft_p95_ms": 433.6856615469515,
+    "ttft_p99_ms": 568.5856753819111,
+    "itl_mean_ms": 18.969870524426348,
+    "itl_p50_ms": 18.7173821692899,
+    "itl_p95_ms": 25.069323533594044,
+    "itl_p99_ms": 28.087401625863414,
+    "tpot_mean_ms": 18.971464480602076,
+    "tpot_p50_ms": 18.718672405515193,
+    "tpot_p95_ms": 25.071671481768842,
+    "acceptance_rate": 0.15400452166690387,
+    "acceptance_pos_1": 0.5521141942543566,
+    "acceptance_pos_2": 0.27944601414362474,
+    "acceptance_pos_3": 0.13266262718646393,
+    "mean_acceptance_length": 2.078031651668327
+  }
+]

benchmarks/large/bs4/run_config.json

@@ -0,0 +1,108 @@
+{
+  "args": {
+    "base_model": "google/t5gemma-2-1b-1b",
+    "dflash_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+    "dflare_model": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+    "dspark_model": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+    "speculative_token_counts": [
+      3,
+      7
+    ],
+    "methods": [
+      "dflash",
+      "dflare",
+      "dspark"
+    ],
+    "datasets": [
+      "math500",
+      "gsm8k",
+      "humaneval",
+      "mbpp",
+      "mt_bench",
+      "ultrachat"
+    ],
+    "num_prompts": 0,
+    "ultrachat_num_prompts": 5000,
+    "warmup": 8,
+    "concurrency": 4,
+    "max_num_seqs": 4,
+    "max_num_batched_tokens": 4096,
+    "max_model_len": 2048,
+    "max_tokens": 128,
+    "gpu_memory_utilization": 0.85,
+    "host": "127.0.0.1",
+    "port": 8011,
+    "temperature": 0.0,
+    "top_p": 1.0,
+    "seed": 0,
+    "timeout_s": 180.0,
+    "server_timeout_s": 600.0,
+    "enforce_eager": false,
+    "include_baseline": true
+  },
+  "runs": [
+    {
+      "name": "baseline",
+      "checkpoint": null,
+      "num_speculative_tokens": null
+    },
+    {
+      "name": "dflash_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflash_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflash_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dflare_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dflare_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dflare_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    },
+    {
+      "name": "dspark_k3",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 3
+    },
+    {
+      "name": "dspark_k7",
+      "checkpoint": "/mnt/d/Projects/dflash_t5/dflash-output/dspark_t5gemma2_mixture_200k/checkpoints/5",
+      "num_speculative_tokens": 7
+    }
+  ],
+  "prompt_sets": {
+    "math500": {
+      "count": 500,
+      "sha256": "78a67699bc700588d0e23da6457ea3afc00d5635897d456a6245f1e964449c4a"
+    },
+    "gsm8k": {
+      "count": 1319,
+      "sha256": "d591bbef95c9d5b17f6f26999bde24def2f6fbf25d7e20953b9ac89c65ffbb8e"
+    },
+    "humaneval": {
+      "count": 164,
+      "sha256": "de1066c5b81dc0f99eab434315cee2c3c68c5a522b8c25d53c73f9c9042b44c8"
+    },
+    "mbpp": {
+      "count": 500,
+      "sha256": "c596be7f77834627fd0a05755691e30dbf4f6724425842de6d03348614e5fecd"
+    },
+    "mt_bench": {
+      "count": 160,
+      "sha256": "8761bfece58bd26e5b93c33e9aad480728ea9950a7d257aaff267db5c7b98010"
+    },
+    "ultrachat": {
+      "count": 5000,
+      "sha256": "5dac9e9dc0d63372c701dbebbb18f3f22a568e9b62cc0ff1b4fb88fc41bf5bf7"
+    }
+  },
+  "benchmark_fingerprint": "f9d631623c44c4276ee1b29d5e2e5ce03c216e6d5c95ffda6333d549323e0c89",
+  "output_dir": "/mnt/d/Projects/dflash_t5/benchmark-results/t5gemma2_speculators_suite_bs4_fixed"
+}

config.json

@@ -0,0 +1,89 @@
+{
+  "architectures": [
+    "DSparkDraftModel"
+  ],
+  "auto_map": {
+    "": "config.DSparkSpeculatorConfig"
+  },
+  "aux_hidden_state_layer_ids": [
+    2,
+    7,
+    13,
+    18,
+    21
+  ],
+  "block_size": 8,
+  "confidence_head_with_markov": true,
+  "draft_vocab_size": 32000,
+  "dtype": "bfloat16",
+  "enable_confidence_head": true,
+  "markov_head_type": "vanilla",
+  "markov_rank": 256,
+  "mask_token_id": 4,
+  "sliding_window_non_causal": false,
+  "speculators_config": {
+    "algorithm": "dspark",
+    "default_proposal_method": "greedy",
+    "proposal_methods": [
+      {
+        "accept_tolerance": 0.0,
+        "proposal_type": "greedy",
+        "speculative_tokens": 7,
+        "verifier_accept_k": 1
+      }
+    ],
+    "verifier": {
+      "architectures": [
+        "T5Gemma2ForConditionalGeneration"
+      ],
+      "name_or_path": "google/t5gemma-2-1b-1b"
+    }
+  },
+  "speculators_model_type": "dspark",
+  "speculators_version": "0.7.0.dev72",
+  "target_hidden_size": null,
+  "tie_word_embeddings": false,
+  "transformer_layer_config": {
+    "architectures": [
+      "T5Gemma2ForConditionalGeneration"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 1152,
+    "initializer_range": 0.02,
+    "intermediate_size": 6912,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 32768,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "num_attention_heads": 4,
+    "num_hidden_layers": 3,
+    "num_key_value_heads": 1,
+    "pad_token_id": null,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "factor": 8.0,
+      "rope_theta": 1000000,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000,
+    "sliding_window": 2048,
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "verifier_final_logit_softcapping": null,
+    "verifier_hidden_states_are_normalized": true,
+    "verifier_is_encoder_decoder": true,
+    "verifier_model_type": "t5gemma2",
+    "vocab_size": 262144
+  },
+  "transformers_version": "5.12.1"
+}

config.py

@@ -0,0 +1,55 @@
+from typing import Literal
+
+from pydantic import Field
+
+from speculators import SpeculatorModelConfig
+from speculators.models.dflash.config import DFlashSpeculatorConfig
+
+__all__ = [
+    "DSparkSpeculatorConfig",
+]
+
+
+@SpeculatorModelConfig.register("dspark")
+class DSparkSpeculatorConfig(DFlashSpeculatorConfig):
+    """DFlash config plus a Markov logit-bias head and a confidence head.
+
+    The Markov head lets each draft position condition on previously sampled
+    tokens within the block; the confidence head predicts the per-position
+    acceptance probability. All DFlash fields are inherited unchanged.
+    """
+
+    speculators_model_type: Literal["dspark"] = "dspark"  # type: ignore[assignment]
+    architectures: list[str] = Field(
+        default_factory=lambda: ["DSparkSpeculator"],
+        description="Model architectures that can load these weights",
+    )
+
+    # Sequential (Markov) head.
+    markov_rank: int = Field(
+        default=256,
+        description=(
+            "Low-rank dimension of the Markov logit-bias factorization B = W1 @ W2. "
+            "Set to 0 to disable the sequential head (pure DFlash drafting)."
+        ),
+    )
+    markov_head_type: Literal["vanilla", "gated", "rnn"] = Field(
+        default="vanilla",
+        description=(
+            "Sequential head variant: 'vanilla' (first-order Markov bias), 'gated' "
+            "(hidden-gated bias), or 'rnn' (recurrent state over the block)."
+        ),
+    )
+
+    # Confidence head.
+    enable_confidence_head: bool = Field(
+        default=True,
+        description="Whether to attach the per-position acceptance-probability head.",
+    )
+    confidence_head_with_markov: bool = Field(
+        default=True,
+        description=(
+            "Concatenate the Markov previous-token embedding with the backbone "
+            "hidden state as the confidence-head input."
+        ),
+    )

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d3ca97ba114c0701b95711bd1cf303a4c636556d448a138cccd763459e08b06
+size 1003149506

optimizer_state_dict.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e276c2a8a6434eb39edc2d22a7aa542bd238fc72115cc1a2adac3f4cc2b2e76b
+size 649875573

scheduler_state_dict.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c2768020750de480fd72c1d5a77315c1d840eb9814df19d857f400239b6598b
+size 1531

tensorboard/dspark_base_t5gemma2_mixture_200k/events.out.tfevents.1784193657.DESKTOP-OU3A33R.751.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec8d73aeaba83a2f72db20b24ca2459d7aac9d639e7129dd033296f48dc024dd
+size 224270442

tensorboard/dspark_t5gemma2_mixture_200k/events.out.tfevents.1783692701.DESKTOP-OU3A33R.282721.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1beb2290cd15cef2c17a04877ec075baac6c75567878d72eadc1ff29c608da9
+size 224270442

train_command.txt

@@ -0,0 +1,9 @@
+# Timestamp: 2026-07-16T09:20:35.403196+00:00
+# Git SHA: 4bd884b204230bbbec44dbb6538eb82593c4016f
+# World size: 1
+# speculators: 0.7.0.dev72
+# vllm: not installed
+# transformers: 5.12.1
+# torch: 2.12.0
+# compressed-tensors: not installed
+scripts/train_t5gemma_online.py --verifier-name-or-path google/t5gemma-2-1b-1b --data-path /home/pc/dflash-output/t5gemma2_mixture_200k --save-path ../dflash-output/dspark_base_t5gemma2_mixture_200k/checkpoints --draft-vocab-size 32000 --speculator-type dspark --draft-arch llama --draft-hidden-act silu --draft-attn-impl eager --block-size 8 --max-anchors 256 --num-layers 3 --target-layer-ids 2 7 13 18 21 --markov-rank 256 --markov-head-type vanilla --enable-confidence-head --confidence-head-with-markov --confidence-head-alpha 1.0 --total-seq-len 2048 --epochs 6 --lr 6e-4 --loss-fn '{"ce": 0.1, "tv": 0.9}' --logger tensorboard --log-dir /home/pc/dflash-output/tensorboard/t5gemma2_mixture_200k --run-name dspark_base_t5gemma2_mixture_200k --on-missing raise --num-workers 0 --prefetch-factor 1

training/data_config.json

@@ -0,0 +1,46 @@
+{
+  "model": "google/t5gemma-2-1b-1b",
+  "mixture": "t5gemma2_200k",
+  "samples": 200000,
+  "seed": 42,
+  "sources": [
+    {
+      "source": "ultrachat",
+      "converter": "auto",
+      "requested": 60000,
+      "kept": 60000
+    },
+    {
+      "source": "sharegpt",
+      "converter": "auto",
+      "requested": 20000,
+      "kept": 20000
+    },
+    {
+      "source": "hf:ise-uiuc/Magicoder-Evol-Instruct-110K::train",
+      "converter": "instruction_response",
+      "requested": 50000,
+      "kept": 50000
+    },
+    {
+      "source": "hf:Salesforce/xlam-function-calling-60k::train",
+      "converter": "xlam_function_calling",
+      "requested": 40000,
+      "kept": 40000
+    },
+    {
+      "source": "hf:glaiveai/glaive-function-calling-v2::train",
+      "converter": "glaive_function_calling",
+      "requested": 20000,
+      "kept": 20000
+    },
+    {
+      "source": "hf:TIGER-Lab/MathInstruct::train",
+      "converter": "instruction_output",
+      "requested": 10000,
+      "kept": 10000
+    }
+  ],
+  "decoder_start_token_id": 2,
+  "eos_token_id": 1
+}

training/train.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+set -euo pipefail
+
+MODEL="${MODEL:-google/t5gemma-2-1b-1b}"
+MIXTURE="${MIXTURE:-t5gemma2_200k}"
+DATASET="${DATASET:-}"
+DATA_DIR="${DATA_DIR:-$HOME/dflash-output/t5gemma2_mixture_200k}"
+OUTPUT_DIR="${OUTPUT_DIR:-../dflash-output/dspark_base_t5gemma2_mixture_200k}"
+LOG_DIR="${LOG_DIR:-$HOME/dflash-output/tensorboard/t5gemma2_mixture_200k}"
+RUN_NAME="${RUN_NAME:-dspark_base_t5gemma2_mixture_200k}"
+MAX_SAMPLES="${MAX_SAMPLES:-200000}"
+SEED="${SEED:-42}"
+T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE="${T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE:-32}"
+TARGET_LAYER_IDS=(2 7 13 18 21)
+
+if [[ ! -f "$DATA_DIR/dataset_info.json" ]]; then
+    PREPARE_ARGS=(
+        scripts/prepare_t5gemma_data.py
+        --model "$MODEL"
+        --output "$DATA_DIR"
+        --max-samples "$MAX_SAMPLES"
+        --seed "$SEED"
+        --encoder-seq-length 2048
+        --decoder-seq-length 1024
+    )
+    if [[ -n "$MIXTURE" ]]; then
+        PREPARE_ARGS+=(--mixture "$MIXTURE")
+    else
+        PREPARE_ARGS+=(--data "$DATASET")
+    fi
+    python "${PREPARE_ARGS[@]}"
+else
+    echo "Prepared dataset already exists at $DATA_DIR; reusing it."
+fi
+
+PYTORCH_ALLOC_CONF=expandable_segments:True \
+TORCHDYNAMO_DISABLE=1 \
+T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE="$T5GEMMA_ONLINE_EXTRACTOR_BATCH_SIZE" \
+python scripts/train_t5gemma_online.py \
+    --verifier-name-or-path "$MODEL" \
+    --data-path "$DATA_DIR" \
+    --save-path "$OUTPUT_DIR/checkpoints" \
+    --draft-vocab-size 32000 \
+    --speculator-type dspark \
+    --draft-arch llama \
+    --draft-hidden-act silu \
+    --draft-attn-impl eager \
+    --block-size 8 \
+    --max-anchors 256 \
+    --num-layers 3 \
+    --target-layer-ids "${TARGET_LAYER_IDS[@]}" \
+    --markov-rank 256 \
+    --markov-head-type vanilla \
+    --enable-confidence-head \
+    --confidence-head-with-markov \
+    --confidence-head-alpha 1.0 \
+    --total-seq-len 2048 \
+    --epochs 6 \
+    --lr 6e-4 \
+    --loss-fn '{"ce": 0.1, "tv": 0.9}' \
+    --logger tensorboard \
+    --log-dir "$LOG_DIR" \
+    --run-name "$RUN_NAME" \
+    --on-missing raise \
+    --num-workers 0 \
+    --prefetch-factor 1

training_state.json

@@ -0,0 +1 @@
+{"epoch": 5, "local_step": 0, "global_step": 146486}

val_metrics.json

@@ -0,0 +1 @@
+{"confidence_loss_epoch": 0.28316470049183373, "confidence_abs_error_epoch": 0.18955371990510891, "confidence_pred_mean_epoch": 0.41305110830444164, "confidence_cumprod_bias_epoch": -0.0008320625755997413, "loss_epoch": 0.6960643153767632, "ce_loss_epoch": 1.539941188752784, "tv_loss_epoch": 0.2876654428823311, "accept_rate_epoch": 0.4054859633838376, "accept_len_epoch": 2.418000488148435, "full_acc_epoch": 0.4366699586238272, "position_1_acc_epoch": 0.6425999769975381, "position_2_acc_epoch": 0.5320870531650069, "position_3_acc_epoch": 0.4550714077528418, "position_4_acc_epoch": 0.4030082730424792, "position_5_acc_epoch": 0.36737465932898755, "position_6_acc_epoch": 0.3395906355061875, "position_7_acc_epoch": 0.316182183562236}