| --- |
| base_model: |
| - TheDrummer/Skyfall-31B-v4.1 |
| tags: |
| - quantization |
| - nvfp4 |
| - Magistral-Small-2509 |
| --- |
| # Model Card for ealexeev/TheDrummer-Skyfall-31B-v4.1-NVFP4 |
|
|
| This is an NVFP4 quantization of TheDrummer/Skyfall-31B-v4.1. |
|
|
| ## Quantization Details |
|
|
| Used https://github.com/ealexeev/llm-quantization script. |
|
|
| Calibration dataset size: 1024 |
| Calibration data: |
| - HuggingFaceH4/ultrachat_200k |
| - allenai/c4_en |
| - mrcedric98/fiction_books_v8 |
|
|
| These were shuffled and mixed at a ratio of 3:2:3 |
|
|
| ### Procedure |
|
|
| ```python ./quantize_nvfp4.py --model TheDrummer/Skyfall-31B-v4.1 --output ./TheDrummer/Skyfall-31B-v4.1 --size 1024 --seed 42 --ultra_chat 3 --c4_en 2 --fiction_v8 3``` |
|
|
| I had read in VLLM docs that NVFP4 quantization needs very few samples. I ran multiple quants of 128, 256, and 512 samples. This 1024 version hit the sweet spot in these particular evals. |
|
|
| ## Quantization Evals |
|
|
| | Metric | Base Model (BF16) | NVFP4 (Quantized) | Delta | |
| | :---------------------------------------- | :---------------- | :---------------- | :------ | |
| | **ARC Challenge** (Logic/Reasoning) | 0.6766 | 0.6561 | -3.03% | |
| | **IFEval** (Strict Instruction Following) | 0.478 | 0.4732 | -1% | |
| | **HellaSwag** (Flow/Common Sense) | 0.8396 | 0.8298 | -1.17% | |
| | **Winogrande** (Ambiguity Resolution) | 2.759 | 2.881 | +4.42% | |
| | **Lambada** (Perplexity) | 7.553 | 8.365 | +10.75% | |
|
|
| ## Bias, Risks, and Limitations |
|
|
| This is already a creative fine-tune. It was quantized with that usecase in mind. Probably not gonna pass any leet-coder challenges with this one. |
|
|
| ## How To Use |
|
|
| ``` |
| bash |
| vllm serve ealexeev/TheDrummer-Skyfall-31B-v4.1-NVFP4 \ |
| --tensor-parallel-size 1 \ # 1 GPU |
| --gpu-memory-utilization 0.8 \ # Else it will take it all for KV |
| ``` |
