--- library_name: transformers license: apache-2.0 pipeline_tag: text-generation base_model: - alphakek/Qwen3.8-27B-heretic-ara-DFlash2 base_model_relation: quantized inference: false tags: - dflash2 - dflash - speculative-decoding - block-diffusion - draft-model - qwen3.8 - fp8 - block-fp8 - e4m3 - mixed-precision - quantized - safetensors - vllm - experimental --- # Qwen3.8-27B-heretic-ara-DFlash2 — selective FP8 experiment This is an **experimental, mixed-precision FP8 quantization** of [`alphakek/Qwen3.8-27B-heretic-ara-DFlash2`](https://huggingface.co/alphakek/Qwen3.8-27B-heretic-ara-DFlash2), prepared and validated by [`magiccodingman`](https://huggingface.co/magiccodingman). It is a target-specific speculative-decoding drafter, not a standalone language model. Use it only with [`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara) or a weight-compatible quantization of that exact target, such as the paired `magiccodingman/Qwen3.8-27B-heretic-ara-fp8` repository. ## Why selective FP8 Quantizing every drafter matrix was measurably less stable. The selected profile quantizes the large MLP projections and preserves DFlash-sensitive paths in BF16. | Property | Value | |---|---:| | Weight format | FP8 E4M3 with BF16 exclusions | | Scale granularity | 128 × 128 blocks | | Activation metadata | Dynamic FP8 | | FP8 matrices | 15: gate/up/down projections across five layers | | BF16 tensors | 66, byte-identical to the source | | Tensor payload | 2.51 GB (2.34 GiB) | | Source BF16 payload | 3.85 GB | | Payload reduction | **34.73%** | The following remain BF16 deliberately: - Q/K/V projections required by the current fused context-KV path - attention output projections - target-hidden-state adapter - dynamic convolution projections and kernels - candidate-selector projection and codebooks - norms and other small parameters ## Validation Eight prompts were run through the real BF16 27B target. Identical hidden states from target layers 5, 19, 33, 47, and 61 were supplied to the BF16 and selective- FP8 drafters, producing 56 compared draft positions through the target's actual 248,320-token LM head. | Metric | Result | |---|---:| | Mean `D_KL(P_BF16 || P_FP8)` | **0.0018930** | | Median proposal KL | 0.0015949 | | P95 proposal KL | 0.0030100 | | Maximum proposal KL | 0.0085216 | | Proposal top-1 agreement | **96.43%** | | Top-16 candidate overlap | **97.66%** | | Draft-hidden cosine similarity | **0.9995342** | | Selector path-token agreement | 85.71% | | Complete seven-token path agreement | 75.0% | Structural validation, block geometry, finite scales, preserved-tensor hashes, controlled MLP arithmetic, official `dflash` 0.1.0 loading, and repository checksums passed. See [`FP8_EXPERIMENT_VALIDATION.md`](./FP8_EXPERIMENT_VALIDATION.md) and the included JSON reports for full details. The selector-path metric is intentionally strict: one early token difference changes the predecessor used by later selector positions. Target verification protects the final output distribution, but proposal drift may still change acceptance length and throughput. ## Usage with the paired target The drafter repository intentionally has no tokenizer or multimodal processor. Those artifacts come from the target model repository and were not present in the upstream drafter repository. Example using the proposed Hub repository names: ```bash vllm serve magiccodingman/Qwen3.8-27B-heretic-ara-fp8 \ --tensor-parallel-size 4 \ --reasoning-parser qwen3 \ --kv-cache-dtype fp8 \ --speculative-config '{"model":"magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8","method":"dflash","num_speculative_tokens":6}' ``` At the time this checkpoint was validated, DFlash2 support for vLLM was still being developed in [`vllm-project/vllm#52816`](https://github.com/vllm-project/vllm/pull/52816). Use a compatible DFlash2-enabled build. Backend support for this selective block-FP8 layout must also be verified on the deployment system. > **Hardware note:** RTX 3090 does not execute native W8A8 FP8. The local tests > validated the serialized weights and proposal behavior by dequantizing FP8 > matrices to BF16. Native FP8 kernel throughput was not measured. ## Performance expectation The upstream BF16 drafter card reports **123.2 tok/s** and **4.24 average acceptance length** on its 4× RTX 3090 TP4 test, versus 116.9 tok/s and 4.01 for the stock `z-lab` drafter. Those are upstream BF16 results, **not results for this FP8 derivative**. Before relying on this checkpoint, compare the BF16 and FP8 drafters on the same hardware, prompts, concurrency, speculative-token count, and runtime build. Measure both tokens/second and acceptance length. ## Provenance and credits The derivative chain is: 1. [Qwen Team — `Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) 2. [`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara), the target model 3. [`z-lab/Qwen3.8-27B-DFlash2`](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2), the stock target-conditioned DFlash2 drafter 4. [`alphakek/Qwen3.8-27B-heretic-ara-DFlash2`](https://huggingface.co/alphakek/Qwen3.8-27B-heretic-ara-DFlash2), SpecForge-tuned for the Heretic ARA target and warm-started from the `z-lab` drafter 5. This selective-FP8 conversion by [`magiccodingman`](https://huggingface.co/magiccodingman) DFlash and its implementation are credited to the [`z-lab/dflash`](https://github.com/z-lab/dflash) project and its authors. The upstream target's ARA transformation credits [`p-e-w/heretic`](https://github.com/p-e-w/heretic), [`timrohrbaugh/heretic`](https://github.com/timrohrbaugh/heretic), and the [ARA contribution](https://github.com/p-e-w/heretic/pull/211). This quantization is an independent derivative and is not an official Qwen, Heretic, z-lab, Inco, alphakek, or vLLM release. ## Limitations and responsibility This checkpoint is experimental. It may load correctly yet provide no speedup if the runtime dequantizes its FP8 matrices or lacks compatible fused kernels. Selector-path drift can reduce speculative acceptance even when target verification preserves final-generation correctness. It also inherits the target model's limitations and reduced refusal behavior. Users are responsible for deployment safeguards, legal compliance, and generated content. ## License Apache License 2.0. See [`LICENSE`](./LICENSE). All upstream notices and attributions remain applicable.