hermitdave commited on
Commit
bfc9f1d
·
verified ·
1 Parent(s): 14e85ef

Add oMLX DFlash note: shipped MTP drafter fails (nested config), use z-lab/Qwen3.8-27B-DFlash2 (+45% tok/s)

Browse files
Files changed (1) hide show
  1. README.md +26 -0
README.md CHANGED
@@ -72,6 +72,32 @@ mtp.bind(base_model)
72
  The drafter is architecture-compatible with Qwen3.5's gated attention and was extracted from the original Agnes-3.0-Flash model. See the [MTP drafter repo](https://huggingface.co/hermitdave/Agnes-3.0-Flash-MTP-drafter) for details.
73
 
74
  **Note:** oMLX does not yet support MTP for text-only models. Use `mlx_vlm.server` or the Python API.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
  ## Attribution
76
 
77
  This conversion was produced by [Hermes Agent](https://hermes-agent.nousresearch.com) (Nous Research) — the autonomous research and conversion pipeline that identified the correct quantization parameters, fixed one-centered norm conversion, and validated output quality. Verified against the reference verison/Agnes-3.0-Flash-MLX-4bit model.
 
72
  The drafter is architecture-compatible with Qwen3.5's gated attention and was extracted from the original Agnes-3.0-Flash model. See the [MTP drafter repo](https://huggingface.co/hermitdave/Agnes-3.0-Flash-MTP-drafter) for details.
73
 
74
  **Note:** oMLX does not yet support MTP for text-only models. Use `mlx_vlm.server` or the Python API.
75
+
76
+ ### ⚠️ oMLX users: the shipped MTP drafter does not work as a DFlash drafter
77
+
78
+ The companion Agnes MTP drafter **cannot be paired with this model for oMLX DFlash speculative decoding**. Its `config.json` nests the model fields under `text_config` (qwen3_5_mtp convention), so oMLX's drafter construction fails and it **silently falls back to plain batched decoding** — no error in the UI, no speedup:
79
+
80
+ ```
81
+ DFlash start failed: DFlashDraftModelArgs.__init__() missing 11 required
82
+ positional arguments: 'hidden_size', 'num_hidden_layers', ...
83
+ ```
84
+
85
+ **Use [z-lab/Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) as the oMLX DFlash drafter instead** — same tokenizer (vocab 248,320), same hidden size (5,120), same qwen3_5-family architecture. Download it via the oMLX model browser (lands in `~/.omlx/models/z-lab/Qwen3.8-27B-DFlash2`), then enable DFlash in the model's settings, or via the admin API:
86
+
87
+ ```bash
88
+ curl -X PUT http://127.0.0.1:8000/admin/api/models/<model_id>/settings \
89
+ -H "Content-Type: application/json" \
90
+ -H "X-Api-Key: <your-api-key>" \
91
+ -d '{
92
+ "dflash_enabled": true,
93
+ "dflash_draft_model": "~/.omlx/models/z-lab/Qwen3.8-27B-DFlash2"
94
+ }'
95
+ ```
96
+
97
+ **Measured** (M3 Max 64 GB, oMLX, temp 0, 500-token code generations): plain batched 16.5–17.2 tok/s → DFlash with the Qwen3.8 drafter **24.3–24.6 tok/s (+45%)**.
98
+
99
+ Verify engagement in `~/.omlx/logs/server.log`: you want `DFlashEngine loaded`, not `DFlash start failed ... fallback from DFlash`.
100
+
101
  ## Attribution
102
 
103
  This conversion was produced by [Hermes Agent](https://hermes-agent.nousresearch.com) (Nous Research) — the autonomous research and conversion pipeline that identified the correct quantization parameters, fixed one-centered norm conversion, and validated output quality. Verified against the reference verison/Agnes-3.0-Flash-MLX-4bit model.