README.md

---
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
base_model:
  - alphakek/Qwen3.8-27B-heretic-ara-DFlash2
base_model_relation: quantized
inference: false
tags:
  - dflash2
  - dflash
  - speculative-decoding
  - block-diffusion
  - draft-model
  - qwen3.8
  - fp8
  - block-fp8
  - e4m3
  - mixed-precision
  - quantized
  - safetensors
  - vllm
  - experimental
---

# Qwen3.8-27B-heretic-ara-DFlash2 — selective FP8 experiment

This is an **experimental, mixed-precision FP8 quantization** of
[`alphakek/Qwen3.8-27B-heretic-ara-DFlash2`](https://huggingface.co/alphakek/Qwen3.8-27B-heretic-ara-DFlash2),
prepared and validated by
[`magiccodingman`](https://huggingface.co/magiccodingman).

It is a target-specific speculative-decoding drafter, not a standalone language
model. Use it only with
[`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara)
or a weight-compatible quantization of that exact target, such as the paired
`magiccodingman/Qwen3.8-27B-heretic-ara-fp8` repository.

## Why selective FP8

Quantizing every drafter matrix was measurably less stable. The selected profile
quantizes the large MLP projections and preserves DFlash-sensitive paths in
BF16.

| Property | Value |
|---|---:|
| Weight format | FP8 E4M3 with BF16 exclusions |
| Scale granularity | 128 × 128 blocks |
| Activation metadata | Dynamic FP8 |
| FP8 matrices | 15: gate/up/down projections across five layers |
| BF16 tensors | 66, byte-identical to the source |
| Tensor payload | 2.51 GB (2.34 GiB) |
| Source BF16 payload | 3.85 GB |
| Payload reduction | **34.73%** |

The following remain BF16 deliberately:

- Q/K/V projections required by the current fused context-KV path
- attention output projections
- target-hidden-state adapter
- dynamic convolution projections and kernels
- candidate-selector projection and codebooks
- norms and other small parameters

## Validation

Eight prompts were run through the real BF16 27B target. Identical hidden states
from target layers 5, 19, 33, 47, and 61 were supplied to the BF16 and selective-
FP8 drafters, producing 56 compared draft positions through the target's actual
248,320-token LM head.

| Metric | Result |
|---|---:|
| Mean `D_KL(P_BF16 || P_FP8)` | **0.0018930** |
| Median proposal KL | 0.0015949 |
| P95 proposal KL | 0.0030100 |
| Maximum proposal KL | 0.0085216 |
| Proposal top-1 agreement | **96.43%** |
| Top-16 candidate overlap | **97.66%** |
| Draft-hidden cosine similarity | **0.9995342** |
| Selector path-token agreement | 85.71% |
| Complete seven-token path agreement | 75.0% |

Structural validation, block geometry, finite scales, preserved-tensor hashes,
controlled MLP arithmetic, official `dflash` 0.1.0 loading, and repository
checksums passed. See [`FP8_EXPERIMENT_VALIDATION.md`](./FP8_EXPERIMENT_VALIDATION.md)
and the included JSON reports for full details.

The selector-path metric is intentionally strict: one early token difference
changes the predecessor used by later selector positions. Target verification
protects the final output distribution, but proposal drift may still change
acceptance length and throughput.

## Usage with the paired target

The drafter repository intentionally has no tokenizer or multimodal processor.
Those artifacts come from the target model repository and were not present in
the upstream drafter repository.

Example using the proposed Hub repository names:

```bash
vllm serve magiccodingman/Qwen3.8-27B-heretic-ara-fp8 \
  --tensor-parallel-size 4 \
  --reasoning-parser qwen3 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"model":"magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8","method":"dflash","num_speculative_tokens":6}'
```

At the time this checkpoint was validated, DFlash2 support for vLLM was still
being developed in
[`vllm-project/vllm#52816`](https://github.com/vllm-project/vllm/pull/52816).
Use a compatible DFlash2-enabled build. Backend support for this selective
block-FP8 layout must also be verified on the deployment system.

> **Hardware note:** RTX 3090 does not execute native W8A8 FP8. The local tests
> validated the serialized weights and proposal behavior by dequantizing FP8
> matrices to BF16. Native FP8 kernel throughput was not measured.

## Performance expectation

The upstream BF16 drafter card reports **123.2 tok/s** and **4.24 average
acceptance length** on its 4× RTX 3090 TP4 test, versus 116.9 tok/s and 4.01 for
the stock `z-lab` drafter. Those are upstream BF16 results, **not results for
this FP8 derivative**.

Before relying on this checkpoint, compare the BF16 and FP8 drafters on the same
hardware, prompts, concurrency, speculative-token count, and runtime build.
Measure both tokens/second and acceptance length.

## Provenance and credits

The derivative chain is:

1. [Qwen Team — `Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B)
2. [`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara), the target model
3. [`z-lab/Qwen3.8-27B-DFlash2`](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2), the stock target-conditioned DFlash2 drafter
4. [`alphakek/Qwen3.8-27B-heretic-ara-DFlash2`](https://huggingface.co/alphakek/Qwen3.8-27B-heretic-ara-DFlash2), SpecForge-tuned for the Heretic ARA target and warm-started from the `z-lab` drafter
5. This selective-FP8 conversion by [`magiccodingman`](https://huggingface.co/magiccodingman)

DFlash and its implementation are credited to the
[`z-lab/dflash`](https://github.com/z-lab/dflash) project and its authors. The
upstream target's ARA transformation credits
[`p-e-w/heretic`](https://github.com/p-e-w/heretic),
[`timrohrbaugh/heretic`](https://github.com/timrohrbaugh/heretic), and the
[ARA contribution](https://github.com/p-e-w/heretic/pull/211).

This quantization is an independent derivative and is not an official Qwen,
Heretic, z-lab, Inco, alphakek, or vLLM release.

## Limitations and responsibility

This checkpoint is experimental. It may load correctly yet provide no speedup
if the runtime dequantizes its FP8 matrices or lacks compatible fused kernels.
Selector-path drift can reduce speculative acceptance even when target
verification preserves final-generation correctness. It also inherits the
target model's limitations and reduced refusal behavior. Users are responsible
for deployment safeguards, legal compliance, and generated content.

## License

Apache License 2.0. See [`LICENSE`](./LICENSE). All upstream notices and
attributions remain applicable.