Instructions to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8")

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8")
model = AutoModel.from_pretrained("magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8

SGLang

How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8 with Docker Model Runner:
```
docker model run hf.co/magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8
```

Qwen3.8-27B-heretic-ara-DFlash2-fp8 / FP8_EXPERIMENT_VALIDATION.md

magiccodingman

Upload folder using huggingface_hub

b6bbe8a verified about 1 month ago

preview code

Raw History Blame Contribute Delete

4.18 kB

Qwen3.8-27B Heretic ARA DFlash2 selective-FP8 experiment

Result

PASS — experimental checkpoint

Source: alphakek/Qwen3.8-27B-heretic-ara-DFlash2
FP8 derivative: magiccodingman/Qwen3.8-27B-heretic-ara-DFlash2-fp8
Profile: five decoder MLPs in native block-FP8; sensitive and currently incompatible paths remain BF16
FP8 format: E4M3 weights, 128 x 128 FP32 inverse scales, dynamic activation metadata
Quantized matrices: 15 (gate, up, and down projection in each of five layers)
Preserved tensors: 66, byte-identical to the BF16 source
Tensor payload: 2,512,200,960 bytes (2.51 GB decimal / 2.34 GiB)
BF16 tensor payload: 3,848,808,960 bytes
Tensor-payload reduction: 34.73%

Why this is mixed precision

The following remain BF16 deliberately:

Q/K/V projections, because the current DFlash fused context-KV path reads their raw weights and bypasses quantization dispatch
Attention output projections, because preserving only about 105 MB materially improved proposal KL and top-1 stability
The target-hidden-state fc adapter
Dynamic convolution projections and kernels
Candidate-selector projection and codebooks
Norms and other small parameters

The metadata contains both Transformers checkpoint names and vLLM runtime names for these exclusions.

Structural validation

Source tensor entries: 81
Candidate tensor entries: 96
FP8 scale tensors: 15
Preserved tensors verified byte-for-byte: 66
Aggregate relative weight RMSE: 0.0264555
FP8 dtype, FP32 scale dtype, positive finite scales, block geometry, tensor closure, and safetensors headers: passed

Controlled sublayer arithmetic

Fixed-seed RMSNorm-like BF16 activation vectors were evaluated through every MLP using the BF16 source and the serialized FP8 weights dequantized to BF16.

Aggregate MLP relative output RMSE: 0.0288080
Aggregate MLP output cosine similarity: 0.9995862
Preserved attention-output projection cosine similarity: 1.0
Result: PASS

Target-conditioned DFlash2 proposal validation

Eight diverse prompts were passed through the real BF16 27B target. Hidden states from target layers 5, 19, 33, 47, and 61 were supplied identically to the BF16 and selective-FP8 drafters. Their 56 draft positions were then projected through the target's actual 248,320-token LM head.

Mean exact proposal KL, D_KL(P_BF16 || P_FP8): 0.0018930
Median proposal KL: 0.0015949
P95 proposal KL: 0.0030100
Maximum proposal KL: 0.0085216
Proposal top-1 agreement: 96.43%
Top-16 candidate overlap: 97.66%
Draft-hidden relative RMSE: 0.0306350
Draft-hidden cosine similarity: 0.9995342
Selector path-token agreement: 85.71%
Complete seven-token selector-path agreement: 75.0%
Result: PASS

The selector metric is intentionally harsh: an early path difference changes the predecessor used by every later selector position. In this test, six of eight complete paths matched, one matched six of seven positions, and one diverged from its first position. Target verification still protects final model correctness; this difference can affect acceptance and speed.

Loader validation

The official dflash 0.1.0 DFlash2DraftModel loader successfully loaded the checkpoint with Transformers 5.15.0. On RTX 3090 it correctly dequantized the FP8 MLP weights to BF16 while retaining the selected BF16 modules.

Deployment limitations

RTX 3090 cannot execute native W8A8 FP8. These tests validate serialized FP8 weight quality, full DFlash2 proposal behavior with real target hidden states, and checkpoint loading, but not native R9700 kernel throughput.

DFlash2's vLLM support is still an experimental, unmerged branch as of this validation. A final R9700 trial must measure end-to-end acceptance length and tokens per second against the BF16 drafter. The BF16 model card baseline is acceptance length 4.24 and throughput 123.2 tok/s under its documented 4x3090 test. Promote this FP8 experiment only if native throughput improves without a material acceptance regression.

The BF16 source and the previously finalized 27B FP8 target were not modified.