prism-ml

/

Ternary-Bonsai-27B-gguf

like 1.26k

Follow

Prism ML 2.83k

You need to use "--spec-type draft-dspark" and with "--spec-draft-n-max 4", works with llama-server

Yeah, more info here:
https://github.com/PrismML-Eng/Bonsai-demo/blob/main/SPECULATIVE.md

Overall everything is setup at our demo repo, so usually should be able to find the exact setup needed (or try the AGENT.md file we have)
https://github.com/PrismML-Eng/Bonsai-demo

Hi, I reran everything following SPECULATIVE.md exactly and collected the logs.

Environment

• Windows
• RTX 3060 12 GB
• Ryzen 5 4500
• Ternary-Bonsai-27B-Q2_0
• Ternary-Bonsai-27B-dspark-Q4_1

Baseline (no speculative decoding)

Command:

.\llama-server.exe --host 0.0.0.0 --port 8080 -m "C:\Users\aceth\OneDrive\Desktop\llama.cpp\Models\Ternary-Bonsai-27B-Q2_0\Ternary-Bonsai-27B-Q2_0.gguf" -ngl 999 -ngld 999 -fa on -c 16384 -np 1

Relevant output:

common_speculative_init: no implementations specified for speculative decoding

Generation speed:

slot print_timing:
eval time = 15818.14 ms / 416 tokens
26.30 tokens/sec

So the baseline is around 26 tok/s.

Speculative decoding

Command:

.\llama-server.exe --host 0.0.0.0 --port 8080 -m "C:\Users\aceth\OneDrive\Desktop\llama.cpp\Models\Ternary-Bonsai-27B-Q2_0\Ternary-Bonsai-27B-Q2_0.gguf" -md "C:\Users\aceth\OneDrive\Desktop\llama.cpp\Models\Ternary-Bonsai-27B-Q2_0\Ternary-Bonsai-27B-dspark-Q4_1.gguf" --spec-type draft-dspark --spec-draft-n-max 4 -ngl 999 -ngld 999 -fa on -c 16384 -np 1

The drafter initializes correctly:

common_speculative_impl_draft_dspark:
- device markov resample (CUDA) ENABLED
adding speculative implementation 'draft-dspark'
block_size=4
markov_rank=256
has_markov=1

draft-dspark: target tap capture engaged on 5 layers
draft-dspark: speculative decoding context initialized

During generation I see:

draft acceptance = 0.30656 (271 accepted / 884 generated)

statistics draft-dspark:
#calls(b,g,a) = 3 227 227
#gen drafts = 227
#acc drafts = 135
#gen tokens = 908
#acc tokens = 277

However decode speed drops dramatically:

slot print_timing:
eval time = 280748.99 ms / 489 tokens
1.74 tokens/sec

The first request is also much slower:

slot print_timing:
eval time = 3714.56 ms / 10 tokens
2.69 tokens/sec

Comparison

Without speculative decoding:

~26.3 tok/s

With speculative decoding:

~1.7–2.7 tok/s

So speculative decoding is definitely being initialized and used (acceptance statistics are reported), but enabling it makes generation roughly 15× slower instead of faster.

I also previously tried running with my usual flags (--mmproj, --mlock, --no-mmap, --ctk, --ctv, --tools, etc.) and saw the same behavior: normal speed without speculation (~25–26 tok/s), but a severe slowdown when draft-dspark is enabled.

Is there anything specific you'd like me to test, or could this indicate an issue in the Windows/CUDA speculative path?

yeah sepculative decoding is still experimental with dspark as we added it last minute (the paper came out 2 weeks ago)

Depending on hardware might not give speed up out of the box, each hardware we need to tune the kernels for it, only had time to test it for few different GPUs so far.
We will do a more careful blogpost, technical docs on this in a few weeks.

I was able to compile your llama-server version and use the Dspark drafter but hit similar problems. Even thought I was actually able to get around 71tok/s vs no Dspark 41tok/s on 5060Ti 16GB ... but the strange thing was, that it somehow only works for the first 4096 tokens in the context, then drops to around 20tok/s ... but when i stay in a prompt under 4096 tokens the Dspark drafter works great... Would be amazing if you could pull it off... Your approach to model quantization is truly amazing and the model definitelly is at least smarter than 9B Qwythos while using the same or less vram space and maintaining nice speed!

Interesting thanks for the info, yeah need to check what goes wrong after 4096 tokens, it might be a fallback to slower kernels paths in llama.cpp.
So on 5060 you got 71 tok/s vs 41 tok? Thats great :D

Yes ! It's related but i also used the latest cuda 13.3 libraries compilation... this definitely works after few modifications... which i don't specifically remember now - I forced gpt 5.5 to rewrite source code to make it work at least with 4K context, but at the end even chatgpt wasn't able to dig dipper into the problem... but yeah Dspark is gonna be so OP when its fully implemented into llama-server

Can verify, same thing happens here. The DSpark drafter is limited to 4096 tokens, which makes it useless.

For anyone landing here from a search and wondering whether moving to upstream llama.cpp is a way around the 4096-token drafter ceiling that @LinkuStarto and @waseigo describe — it is not, yet, and it may save you the afternoon I spent to know why.

I run the target model on stock ggml-org llama.cpp 22dc605c4ead20e36f447cc67b55ef87e523bd55 (b10254) on a 4090. The target is fine there: Ternary-Bonsai-27B-Q2_g64.gguf runs at 89 t/s decode at 262,144 context with the BF16 mmproj and Q8_0 K/V. But neither published drafter will load on mainline, for two independent reasons:

1. The architecture string does not match what was merged. Mainline has the feature, but registered as dflash — LLM_ARCH_DFLASH → "dflash" (src/llama-arch.cpp:136), implemented in src/models/dflash.cpp, whose own comment reads "DeepSeek-V4 DSpark backbone". The published files declare general.architecture = "dspark", which mainline does not know, and their metadata keys are doubly namespaced (dspark.dspark.target_layers) where mainline reads dflash.target_layers. The BF16 drafter fails with unknown model architecture: 'dspark'.
2. The quantized tensors are packed one byte per block smaller than mainline's. Ternary-Bonsai-27B-dspark-Q4_1.gguf gets further and then dies with tensor 'dspark.fc.weight' has offset 337718592, expected 357584192. That is the group-128 vs group-64 split — 17 vs 18 bytes per 64 weights under the same ggml type id 42 — the same one documented for *-Q2_0.gguf vs *-Q2_0_g64.gguf.

Full write-up with the arithmetic that reproduces both numbers in that error message: https://github.com/PrismML-Eng/Bonsai-demo/issues/137

So the current state is: the fork loads the drafter but caps out around 4096 tokens as this thread reports, and mainline cannot load it at all. Making a mainline drafter would need the arch renamed to dflash with correctly-prefixed metadata keys, the tensors renamed to the merged names (markov_w1, markov_w2, conf_proj), and a g64 repack.

To be clear about what I did not test: I never got a drafter running, on either build, so I have no acceptance-rate or tokens/sec numbers of my own and cannot say anything about the 4096-token behaviour itself.

One thing that may help more than the drafter, depending on your workload — if you resend a long, mostly unchanged prompt each turn, prefix reuse is worth far more than speculative decoding and is disabled in the demo launcher's default flags. On a 178k-token prompt, adding --cache-ram 4096 --ctx-checkpoints 32 --cache-idle-slots took an identical repeat from 102.34 s of prefill down to 0.14 s. It does change greedy output though, so it is not a free switch: https://github.com/PrismML-Eng/Bonsai-demo/issues/147