MJPansa/Qwen3.8-Flash-Next-MXFP4-FP8-R12

Qwen3.8-Flash-Next MXFP4–FP8 R12

This is an experimental mixed-precision conversion of Qwen/Qwen3.8-Flash-Next-FP8. It reduces the routed-expert checkpoint storage while retaining the 12 expert layers found most sensitive in our calibration and whole-model evaluation.

Run on one DGX Spark

GitHub recipe and benchmarks · Deployment details

Run on the DGX Spark:

git clone --branch qwen38-single-spark-release https://github.com/marco-jeffrey/spark-vllm-docker.git
cd spark-vllm-docker
./run-recipe.sh qwen3.8-flash-next-mxfp4-fp8-r12 --setup --solo

Setting	Value
Hardware	1× DGX Spark, 128 GB unified memory
Context / max output	262,144 / 32,768 tokens per request
Shared KV cache	~700K token slots, FP8 E4M3
Concurrency	Up to 16 requests
Speculation / graphs	MTP-2; decode CUDA graphs
Images	Up to 32; 32,768 visual tokens total; automatic resizing
PLE n-gram table	Local NVMe offload
API	Port 8000, /v1/chat/completions

Setup builds the patched image on eugr's Spark base and reuses the standard Hugging Face cache. Allow approximately 146.55 GB for checkpoint files, plus Docker images and runtime caches. While this model is private, authenticate with hf auth login using an authorized account.

Quantization format

This is not a uniformly four-bit model:

• 36 of 48 main routed-expert layers use MXFP4 E2M1 weights with UE8M0 microscales;
• their activations are dynamically quantized to MXFP8 at inference time;
• 12 main routed-expert layers remain in the source block-FP8 format;
• the MTP routed experts and the 51.2B-value PLE n-gram table remain FP8; and
• attention, GDN, QSA, vision, embeddings, shared experts, routers, norms, hyperconnections, and the LM head retain their source checkpoint formats.

The repository name uses FP8 because FP8 is both the dynamic activation family and the retained secondary weight format. R12 means twelve main routed-expert layers are retained in FP8.

Compatibility

Use the linked recipe for this mixed checkpoint. It provides the ARM64/SM121 runtime, per-layer MXFP4/FP8 loading, NVMe PLE, packed draft head and image limits. An ordinary MXFP4 loader alone is insufficient. The pinned runtime, patches and known limitations are documented in the deployment guide.

The current runtime contract is:

Component	Storage	Runtime path
Converted routed experts, 36 layers	MXFP4 E2M1 + UE8M0 scales	FlashInfer/CUTLASS MXFP4 × dynamically quantized MXFP8
Retained routed experts, 12 layers	Source block FP8	Dynamic-FP8 routed-expert path
MTP routed experts	Source block FP8	Dynamic-FP8 routed-expert path
PLE n-gram table	Global-scale FP8 E4M3	NVMe-backed FP8 lookup followed by PLE dequantization
Separate draft-only output head	Packed MXFP4	FlashInfer MXFP4 × dynamic MXFP8; target BF16 LM head unchanged
Other model components	Source formats, primarily BF16	Existing Qwen vLLM kernels

Retained layers

The following zero-based decoder layers keep their main routed experts in block FP8:

19, 20, 21, 23, 28, 32, 33, 40, 41, 43, 44, 45

Every other main routed-expert layer uses the calibrated MXFP4 artifact. The exact machine-readable routing contract is stored in config.json under:

• quantization_config.fp8_routed_layers;
• quantization_config.fp8_mtp_routed_layers; and
• quantization_config.fp8_ple_embeddings.

Size

The indexed weight payload is 143,023,845,370 bytes: 143.024 GB decimal or approximately 133.201 GiB.

Component	Stored size	Share
Main routed experts	78.332 GB	54.77%
PLE n-gram table	51.200 GB	35.80%
GDN blocks	4.173 GB	2.92%
MTP module	2.698 GB	1.89%
Hyperconnections and norms	1.281 GB	0.90%
Token embedding	1.271 GB	0.89%
LM head	1.271 GB	0.89%
QSA attention and indexers	1.235 GB	0.86%
Vision tower	0.898 GB	0.63%
Shared experts	0.472 GB	0.33%
Routers and gates	0.126 GB	0.09%
PLE processing and metadata	0.066 GB	0.05%

The source FP8 checkpoint contains 185.502 GB of indexed tensor payload. This conversion saves 42.478 GB, or 22.90%, without further quantizing the PLE table or the 12 retained routed layers.

Dynamic MXFP8 activations are created at runtime and do not occupy checkpoint storage.

Quantization and layer selection

The conversion started from the released FP8 weights; a BF16 copy of the full model was not required.

The production calibration corpus used the NVIDIA ModelOpt-style source mix:

• 512 CNN/DailyMail samples; and
• 512 samples from nvidia/Nemotron-Post-Training-Dataset-v2.

Real routed activations were captured from the source FP8 model. Candidate MXFP4 expert artifacts were optimized and selected against held-out route-weighted expert outputs. All 48 routed layers were then measured on a separate sensitivity corpus. Whole-model retain-0/5/10/12/14 checkpoints were evaluated on a frozen next-token likelihood split before retain-12 was selected.

Retain-12 had the best point-estimate NLL among those predeclared candidates. Its difference from retain-10 and retain-14 was not statistically decisive; R12 is the frozen selector result, not a claim that twelve retained layers are universally optimal for every workload.

The full conversion provenance is retained in hybrid-conversion-manifest.json.

Accuracy compared with the FP8 source

Frozen selection split

The selection split contains 200 documents and 409,400 matched next-token positions. MTP and CUDA graphs were disabled for deterministic comparison.

Model	Mean NLL	Perplexity	Observed-token top-1	Greedy agreement with FP8
FP8 source	1.4866333373	4.4221824326	65.129702%	100.0000%
Retain-12	1.4876750205	4.4267913458	65.068881%	93.8283%

Retain-12's selection-split perplexity increase is 0.0046089133, or 0.104223%.

Untouched confirmation split

The final corpus was not used to select the retained-layer count. It contains 256 documents and 524,032 target positions.

Metric	FP8 source	Retain-12	Change
Mean token NLL	1.5165066356	1.5184679159	+0.0019612802 (+0.129329%)
Perplexity	4.5562806129	4.5652255250	+0.0089449120 (+0.196320%)
Observed target is greedy top-1	64.054485%	63.990558%	-0.063927 percentage points
Greedy next-token agreement	100%	93.740077%	-6.259923 percentage points

The paired relative-NLL 95% bootstrap interval was [+0.075775%, +0.181898%]. The accuracy difference is small but measurable and must not be described as lossless.

Full-vocabulary distribution comparison

Across 512 held-out contexts:

Metric	Result
Mean `KL(FP8
Median KL	0.00656678 nats
KL p95	0.08013794 nats
Jensen–Shannon divergence	0.00549240
Top-1 agreement	93.1641%

Behavioral diagnostic

The FP8 source passed 5/5 frozen DeepSeek Harness tasks and retain-12 passed 4/5. Retain-12 passed exact response, Python repair, structured JSON, and surgical editing. The one failure preserved the correct state and answer but added explanatory prose where the strict validator required an exact two-word reply. The harness adapter omitted explicit sampling controls, so this isolated sampled difference cannot be causally attributed to quantization. It remains disclosed rather than hidden.

Single-Spark runtime

The recipe keeps the approximately 51.2-GB PLE table on local NVMe and fetches the required rows instead of loading the complete table into shared memory. The remaining indexed weight payload is approximately 91.824 GB; runtime buffers, KV cache and draft-head storage are additional.

Decode results for coding, summarization and creative writing, plus 32K prefill at different context depths, are in the GitHub benchmark tables.

License and attribution

The converted weights remain subject to the included Qwen Community License. See LICENSE. This repository does not change or replace the source model's license terms.

Source model and architecture credit belongs to the Qwen team. This repository contains a third-party mixed-precision conversion and should not be interpreted as an official Qwen release.