README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-9B
pipeline_tag: text-generation
tags:
- nvfp4
- compressed-tensors
- quantization
- qad
---

# Qwen3.5-9B-NVFP4-QAD-s4000

NVFP4 QAD (W4A16-trained (weight-only)) checkpoint of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) @ `c202236235762e1c871ad0ccb60c8ee5ba337b9a`, optimizer step 4000 of 4000.

**Stage-1 NVFP4 control arm at lr 1e-6** (matched-budget format comparison vs the MXFP4 arm). NVFP4's RTN init is already near-lossless on this model (serving A16 KL ~0.0486, GSM8K ~85 vs BF16 83.8), and at lr 1e-6 training moves almost nothing on top. **Superseded by the lr 1e-5 arm** ([Qwen3.5-9B-NVFP4-QAD-LR1e-5-s4000](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-LR1e-5-s4000)).

## How this checkpoint was produced

| item | value |
|---|---|
| training repo | [QATFactory](https://github.com/tonyzhang-together/QATFactory) branch `weili/mxfp4` @ `86b72a2` ([PR](https://github.com/tonyzhang-together/QATFactory/pull/5)) — `scripts/train_llm_qat.py` |
| method | QAD (quantization-aware distillation): student trains with fake-quantized forward; frozen BF16 teacher = the base model itself |
| objective | pure KL at temperature 1.0 (`distill_weight 1.0`, `hard_label_weight 0.0`), loss on assistant tokens only |
| base model / teacher | [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) @ `c202236235762e1c871ad0ccb60c8ee5ba337b9a` |
| dataset | `openperfectblend_100k_Qwen3.5-9B_think` — ~100K ChatML conversations (OpenPerfectBlend-derived prompts with Qwen3.5-9B think-mode responses; native `<think>` spans in the assistant turns), prepared in the QATFactory project. Train file from [togethercomputer/Qwen3.5-9B-reasonmix](https://huggingface.co/datasets/togethercomputer/Qwen3.5-9B-reasonmix) @ `b88c109` (940,793,581 bytes, md5 `0406bb3a7a482352360716a1bc5e9e04`; ~84.3k train conversations by the trainer's epoch accounting). Held-out eval = a disjoint 256-conversation split (md5 `af10c8c304c146a81ddc35e439d7ac4b`), 818,944 scored positions — the same corpus used for the serving-KL rows below |
| preprocessing | model chat template (ChatML), **assistant-only loss mask**, `max_seq_len` 8192, right-truncated |
| this checkpoint | **optimizer step 4000** of 4000 (32,000 conversations consumed ~= 0.38 epoch, no data repetition) |
| batch per weight update | **8 sequences** = 1/GPU x 8 GPUs x grad-accum 1 (<= 8 x 8192 = 65,536 tokens/update) |
| learning rate | peak **1.0e-6**, cosine decay to 0 over 4000 steps, linear warmup 1% (40 steps) — recipe default; later shown ~10x too small by the 500-step lr sweep (held-out eval -0.3% at 1e-6 vs -46% at 1e-5) |
| optimizer | AdamW (`adamw_torch`, beta1 0.9 / beta2 0.999), weight_decay 0.0, `max_grad_norm` 1.0 |
| precision / parallelism | bf16, FSDP2 `full_shard` on jbom 8xB200 (single node), gradient checkpointing (non-reentrant), `sdpa` attention |
| fake-quantized modules | all linear projections (`q/k/v/o_proj`, `gate/up/down_proj`, GatedDeltaNet in/out projections); embeddings, `lm_head`, norms and the vision tower stay BF16 |
| quantization config | `quant_format: nvfp4`, `fused_runtime_scales: true` (exportable native NVFP4) |
| seed / bookkeeping | seed 42; held-out eval every 100 steps; checkpoint every 1000 steps; step time ~2.0 s/step |
| in-loop held-out eval KL | 0.02903 -> 0.02878 (-0.9% over 4000 steps — flat) |
| W&B | [`10wx7xlv`](https://wandb.ai/weilixu2-university-of-illinois-urbana-champaign/qatfactory-qat/runs/10wx7xlv) (project [`qatfactory-qat`](https://wandb.ai/weilixu2-university-of-illinois-urbana-champaign/qatfactory-qat), public) |

**Training mode: W4A16 (weight-only fake-quant).** Weights are fake-quantized in the forward pass; activations stay BF16 during training. The exported artifact can still be served W4A4 — the tables below measure both.

## Serving

`compressed-tensors` NVFP4 artifact carrying the **full W4A4 schema** (FP4 E2M1 weights,
block-16 FP8-E4M3 scales + per-tensor global scales, **static input-activation scales**
learned during training). Default load in vLLM >= 0.25.1 on SM100+ serves **W4A4**
(measured 12.3-15.2k tok/s single-GPU greedy vs ~7.7-8.3k for W4A16 — ~1.6x). For clean
weight-only **W4A16** serving use the sibling `--weight-only` export:
[Qwen3.5-9B-NVFP4-QAD-s4000-weight-only](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-s4000-weight-only).

```bash
vllm serve weili-0234/Qwen3.5-9B-NVFP4-QAD-s4000 --max-model-len 24576
```

## Evaluation (step 4000)

| model | KL vs BF16 | GSM8K | GPQA-D | MMLU-Pro |
|---|---|---|---|---|
| BF16 teacher | — (reference) | 83.8 | 67.68 | 77.0 |
| **this ckpt, A16 (W4A16 serving)** | 0.0486 | 83.8 | 63.1 | 76.7 |
| **this ckpt, A4 (W4A4 serving)** | 0.0751 | 85.8 | 61.6 | 71.8 |

All rows come from one frozen harness: vLLM 0.25.1 on B200 (SM100), temperature 0; `KL` = top-20 + residual-bucket KL vs the BF16 teacher's next-token distribution over 818,944 held-out positions (nats, lower is better; pipeline validated by a BF16-self run at 0.0002); GSM8K test[:500] 0-shot CoT; GPQA-Diamond n=198 (single seed — treat deltas < ~6 points as noise); MMLU-Pro 1000-question subset.

## Related checkpoints

- Other steps of this arm: [s1000](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-s1000), [s2000](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-s2000), [s3000](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-s3000)
- QAD-trained BF16 master weights (pre-export): [Qwen3.5-9B-NVFP4-QAD-s4000-BF16](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-s4000-BF16)
- Higher-dose arm: [NVFP4-QAD-LR1e-5](https://huggingface.co/weili-0234/Qwen3.5-9B-NVFP4-QAD-LR1e-5-s4000)

---

Part of a monitored QAD experiment series with full bookkeeping (pre-registered predictions, exact SHAs/configs/seeds per run). Produced with AI assistance (Claude).