Update README.md
Browse files
README.md
CHANGED
|
@@ -45,8 +45,6 @@ Calibration datasets:
|
|
| 45 |
|
| 46 |
All 512 routed experts per layer were activated during calibration — no experts were missed entirely. Post-calibration, 256 out of ~184k routed expert quantizers (~0.14%) had their amaxes floored to median/10 of their peer group to stabilize rarely-hit experts. Gate/up projection weight scales were tied for fused w13 export compatibility.
|
| 47 |
|
| 48 |
-
Compared to [nvidia/Qwen3.5-397B-A17B-NVFP4](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4) (calibrated on cnn_dailymail + Nemotron), the input activation scales in this checkpoint show significantly higher entropy per layer (avg 3.92 vs 2.35 bits across all 60 layers), indicating more differentiated per-expert scales. This suggests the diverse calibration mix produced scales that better reflect each expert's actual activation range, rather than collapsing toward similar values from homogeneous calibration data.
|
| 49 |
-
|
| 50 |
### Quality
|
| 51 |
|
| 52 |
MMLU Pro results:
|
|
|
|
| 45 |
|
| 46 |
All 512 routed experts per layer were activated during calibration — no experts were missed entirely. Post-calibration, 256 out of ~184k routed expert quantizers (~0.14%) had their amaxes floored to median/10 of their peer group to stabilize rarely-hit experts. Gate/up projection weight scales were tied for fused w13 export compatibility.
|
| 47 |
|
|
|
|
|
|
|
| 48 |
### Quality
|
| 49 |
|
| 50 |
MMLU Pro results:
|