leoncca/Qwen3.8-Flash-Next-NVFP4-QSA-FP8-E4M3-KV-Scales

Qwen3.8 Flash Next NVFP4 QSA FP8 E4M3 KV Scales

Scale payload size: 2.64 kB (2.58 KiB). This repository contains no model weights and is not a standalone checkpoint.

This is a calibrated 24-tensor K/V scale pack for the main QSA KV cache of:

• base checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4
• exact base revision: 7b719225242aacd3dbd3f9407468c2ee9a9d2594
• base model.safetensors.index.json SHA-256: da5ca9c3b65e48e151329e64e141c2fa700bf2f99aec53cc014e4b52a6ff7a84

Do not use these scales with another checkpoint revision. The included materializer checks the base index hash before creating an overlay.

Runtime status

The required QSA E4M3 runtime support is currently in draft 1Cat-vLLM PR #447. The validated performance tree also includes two optional, independently mergeable Triton follow-ups: the bitcast decoder in #452 and scale hoisting in #453. The exact pre-split performance tree is preserved by the submitter's fork tag evidence/qsa-e4m3-perf-combined-20260902. This scale pack is not loadable by upstream vLLM or an unpatched 1Cat-vLLM main at publication time.

The validated first-phase contract is:

• NVIDIA SM70 / Tesla V100;
• tensor parallel size 4;
• MTP disabled;
• FP16 activations;
• calibrated FP8 E4M3 main QSA K/V cache;
• FP16 raw and compressed QSA indexer caches;
• context lengths through 128K.

The runtime fails closed if all 24 finite, positive scale tensors are not explicitly loaded. Unit scale is not bundled as a fallback or as a negative control.

Create a loadable checkpoint view

Download the exact base revision and this scale pack:

hf download RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 \
  --local-dir /path/to/base-checkpoint

hf download leoncca/Qwen3.8-Flash-Next-NVFP4-QSA-FP8-E4M3-KV-Scales \
  --revision v1.1-base-7b719225 \
  --local-dir /path/to/scale-pack

Materialize a derived view. The base directory remains unchanged and must stay available because the derived view uses symlinks for its checkpoint files.

python /path/to/scale-pack/materialize_overlay.py \
  --pack-dir /path/to/scale-pack \
  --base-checkpoint /path/to/base-checkpoint \
  --output-dir /path/to/qsa-e4m3-checkpoint

The tool validates the base revision, the scale-pack checksum, the exact 24 tensor names, FP32 scalar types, positive finite values, and the non-negative- control marker. It then writes a merged standard model.safetensors.index.json plus kvscales-provenance.json. A separate scale index is intentionally not used because the current loader does not discover it automatically.

A representative validated server configuration was:

VLLM_SM70_NVFP4_TURBOMIND=1 \
VLLM_SM70_NVFP4_MOE_GROUPED_PREFILL=1 \
VLLM_SM70_QWEN4_EXP_ONLINE_QPN8=1 \
VLLM_1CAT_DISABLE_SM70_MTP_DEFAULTS=1 \
python -m vllm.entrypoints.openai.api_server \
  --model /path/to/qsa-e4m3-checkpoint \
  --trust-remote-code \
  --dtype half \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.89 \
  --max-model-len 131328 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --language-model-only \
  --no-async-scheduling \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'

Calibration

K/V activations were collected with FP16 KV through normal inference forwards after engine initialization. Dummy/profile/graph-capture forwards were excluded. The calibration distribution covered Chinese, English, code, multi-turn dialogue, tool use, and 1K/16K/64K/128K contexts. Calibration and quality inputs were disjoint. Raw private tool-use records are not distributed.

The converter contract is:

stored = e4m3fn(x / scale)
reconstructed = stored * scale
scale = observed_max_envelope / 448

The 24 FP32 scales range from 0.0171247218 to 0.0841238871. No saturation was observed for any K/V tensor. Per-layer max-abs, p99.9, p99.99, saturation, and shard-stability summaries are recorded in kvscales-manifest.json.

V100 validation

• Same memory budget: 507,093 FP16-KV tokens versus 931,100 E4M3-KV tokens (1.836x).
• 4K matched trace: identical first token and complete output token sequence.
• 128K retrieval at 0%, 50%, and 100% insertion depth, twice each: 6/6 for both FP16 and calibrated E4M3, with token-identical paired outputs.
• Held-out tool selection: 10/12 for both modes, zero correctness regressions, and identical first tokens in all 12 comparisons.
• Selected-block overlap: mean row Jaccard 0.990783, FP16 recall 0.995301, and micro Jaccard 0.987579.
• QSA core output: cosine 0.999311, relative L2 0.0324132, RMSE 0.0160753.
• Performance matrix: concurrency 1/4/8 by context 1K/4K/16K/32K/64K/128K; 17 E4M3 cells passed, with only C8 x 128K skipped by the capacity guard.
• At 16K and longer, E4M3 was about 2.94% lower in decode throughput and 3.38% higher in end-to-end elapsed time. This is a capacity result, not a global speedup claim.

FP16 XQA used P1024 while generic E4M3 G6 XQA used P256, so the full performance A/B includes reduction order as a second variable. The quality run did not enter XQA and was not affected by this difference.

Limitations

• TP8, MTP, prefix caching, standalone graph-versus-eager parity, and 256K context have not been accepted.
• The held-out tool set is a first-phase regression gate, not broad production acceptance.
• These scales are checkpoint-revision-bound and are not generic Qwen3.8 or QSA scales.
• This repository does not redistribute the base model. Follow the base and source model license terms.

Integrity

The authoritative scale file is model-kvscales.safetensors:

bbf767cd46fe3ac52793c87ec069a964bf6fb64c5b6183959c734b9772936cc9

See SHA256SUMS for the complete release manifest.