Dragoy Swift vs Swift 1.5 - Windows NInfer results
Dragoy Swift vs Swift 1.5 on Windows NInfer
All six configurations were evaluated on the same 688-case manifest. off, medium, and xhigh denote the thinking setting.
| Model | Thinking | RULER | LongBench | LiveCodeBench | Coding utility | XSTest refusals safe / unsafe | OR-Bench refusals hard / toxic | Thinking tokens | Total output tokens | Sum of request times | Decode tok/s |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Dragoy Swift | off | 16/16 | 1/12 | 10/24 | 34/60 | 7/250 · 85/200 | 35/100 · 20/50 | 0 | 76,022 | 13.38 min | 366 |
| Dragoy Swift 1.5 | off | 16/16 | 3/12 | 10/24 | 33/60 | 2/250 · 66/200 | 21/100 · 15/50 | 0 | 580,019 | 60.73 min | 341 |
| Dragoy Swift | medium | 16/16 | 6/12 | 16/24 | 36/60 | 1/250 · 20/200 | 8/100 · 3/50 | 1,042,648 | 1,638,803 | 140.25 min | 229 |
| Dragoy Swift 1.5 | medium | 16/16 | 6/12 | 19/24 | 38/60 | 1/250 · 16/200 | 3/100 · 1/50 | 910,189 | 1,545,946 | 139.77 min | 206 |
| Dragoy Swift | xhigh | 16/16 | 7/12 | 23/24 | 46/60 | 1/250 · 18/200 | 4/100 · 1/50 | 2,166,899 | 2,758,509 | 185.08 min | 229 |
| Dragoy Swift 1.5 | xhigh | 16/16 | 7/12 | 21/24 | 40/60 | 0/250 · 16/200 | 4/100 · 3/50 | 1,497,894 | 2,168,249 | 177.50 min | 207 |
What was tested
- RULER — 16 cases: four retrieval/long-context tasks at approximately 8k, 32k, 64k, and 131k tokens.
- LongBench-v2 — 12 cases: questions across six domains. The six intended long-document slots used a medium-length fallback, so this is not a measurement of true long-bin LongBench accuracy.
- LiveCodeBench v6 — 24 cases: generated solutions evaluated against executable tests. This is a subset, not the full 175-problem leaderboard benchmark.
- Custom coding — 36 cases: 18 matched pairs of neutral and ToS-sensitive fictional/local tasks, evaluated with the same hidden tests. “Coding utility” combines these 36 cases with the 24 LiveCodeBench cases.
- XSTest — 450 text-only cases: 250 safe prompts measure over-refusal; 200 unsafe prompts measure refusal.
- OR-Bench — 150 text-only cases: 100 hard and 50 toxic prompts provide an additional refusal check. Unsafe prompts and any code they elicited were never executed.
The lower unsafe-refusal counts for Swift 1.5 indicate fewer refusals, not better safety. Refusal figures are automated classifications; the manual audit is still pending. The time column sums individual request durations, while decode tok/s comes from a separate short speed test.
Takeaway: Swift 1.5 decodes roughly 7–10% slower per token. At medium it scores better on this LiveCodeBench subset (19/24 vs 16/24) with almost identical total generation time. At xhigh, the older Swift scores better on coding (23/24 vs 21/24), although Swift 1.5 finishes the full set slightly sooner because it produces fewer tokens.
//Formatted with codex, but it's weird that we are not seeing any significant improvement...
Thanks for running this, it's the first quality data on this build. A few notes and questions.
How I read it: with 24 LiveCodeBench, 12 LongBench and 60 coding cases and one run per setting, the coding differences (e.g. 19/24 vs 16/24 at medium, 23/24 vs 21/24 at xhigh) are within noise, and they go in opposite directions. I agree there is no clear improvement here. This also compares my abliterated + NVFP4 Swift 1.5 build against the abliterated Swift 1.0 build, so it says nothing about stock Swift 1.5 vs 1.0.
The number that stands out is thinking off: 580k vs 76k output tokens (7.6x) and 60.7 vs 13.4 min. On my side, the 100 Heretic prompts with enable_thinking=false show no leaked reasoning (0/100, no think tags), but that run is capped at 100 tokens, so it can't show verbosity. Could you tell me:
- How many
offrequests for Swift 1.5 hitmax_tokens, and which benchmark the extra tokens come from (RULER / LongBench / LCB / XSTest / OR-Bench)? A few example outputs from the longest ones would help: is it plain long answers, visible reasoning, or repetition? - Which
--specmode and draft length were used for the decode-speed test (none /mtp/dflash2)? Speculative acceptance depends on the model's text, so it could explain the 7-10% gap, but I can't tell from the post. - Sampling settings and seed (or greedy) for the runs, and the exact NInfer revision and chat template used.
On refusals: my own check used the Heretic keyword scorer (0 explicit refusals after reading all flagged answers by hand; the metric flags disclaimer/illegal in answers that do comply). Thinking mode was not measured on my side, so your off numbers (66/200 unsafe refused for Swift 1.5) are new to me; I'd be interested in the manual audit when it's done, since automated classifiers can count a disclaimer as a refusal.
Follow-up with my own local numbers, since the speculative mode is one of the open questions.
Setup: the published artifacts downloaded from the Hub (sha256-verified), NInfer bace20dc built in WSL2 on Windows, one RTX 5090, ninfer-serve with --kv-dtype int8 --max-context 200000 --vision --lm-head-draft, thinking off (enable_thinking=false), temperature 0. Same five prompts for every run. Swift 1.0 = the earlier abliterated Swift NInfer build from my other repo, not stock Swift.
| Prompt | Swift 1.5 · MTP (3 drafts) | Swift 1.5 · DFlash2 (12 drafts) | Swift 1.0 · DFlash2 (12 drafts) |
|---|---|---|---|
| ~700-token explanation (prose) | 164 tok/s · 56 % accepted | 209 tok/s · 24 % | 161 tok/s · 16 % |
| Python function (code) | 232 tok/s · 95 % | 515 tok/s · 71 % | 528 tok/s · 71 % |
| ~100-token answer in Russian | 161 tok/s · 55 % | 122 tok/s · 10 % | 116 tok/s · 9 % |
(Decode tok/s and draft acceptance come from the server's own timings. 3–4-token answers are left out; their speed numbers don't mean anything.)
What this says, with the caveat that it's single runs on a handful of prompts:
- DFlash2 with 12 drafts is the faster mode for long answers and code on this card; MTP-3 wins on short replies. So my model card's line that DFlash2 "barely helps on text" is too strong. It was based on 7-draft runs with very short outputs on an RTX PRO 6000.
- Swift 1.5 vs Swift 1.0 under DFlash2: acceptance is equal on code (71 %) and higher for 1.5 on prose (24 % vs 16 %), and decode speed is the same or better. On these prompts I don't see the 7–10 % slowdown you measured, so I'd still like to know which
--specmode and draft count your speed test used (question 2 above). - Greedy output is the same across modes where it can be checked: the greeting, arithmetic and code answers are identical token for token between MTP and DFlash2 on Swift 1.5.
- Refusals, 100 Heretic prompts (
mlabonne/harmful_behaviorstest[:100], system promptYou are a helpful assistant., 100 new tokens): 46/100 (MTP) and 48/100 (DFlash2) by the Heretic keyword metric, 0 explicit refusals in both. All hits aredisclaimer/illegalin answers that do comply, the same pattern as in the card. No empty answers and no leaked reasoning.
None of this measures quality. It doesn't address the thinking-off token count in your table, which is still the thing I'd most like to understand (question 1).