Expand model card with quantization and benchmark details

README.md

@@ -7,45 +7,279 @@ base_model: ukisai/Swift-Qwen3.8-27b
 base_model_relation: quantized
 tags:
 - qwen3_8
+- qwen3_5
 - fp8
 - compressed-tensors
 - sglang
+- speculative-decoding
+- dflash
 - reasoning
+- efficient-thinking
+- conversational
 ---
 
 # Swift-Qwen3.8-27B-FP8
 
-FP8 quantization of `ukisai/Swift-Qwen3.8-27b`.
+FP8 quantization of
+[`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b).
 
-Independent community quantization; not an official UkisAI release.
+This is an independent community quantization and is **not an official
+UkisAI release**.
 
-## Quantization
+The goal of this checkpoint is to preserve the behavior of Swift-Qwen3.8-27B
+while reducing VRAM requirements and enabling high-throughput inference with
+SGLang, including speculative decoding with the model's native MTP head or an
+external DFlash2 draft model.
 
-- compressed-tensors
-- FP8_BLOCK
-- weight block: 128 x 128
-- dynamic FP8 activations
-- activation group size: 128
-- llmcompressor 0.13.0
-- checkpoint size: ~29 GB
-- context length: 262,144 tokens
+## Model summary
+
+- Upstream model: `ukisai/Swift-Qwen3.8-27b`
+- Architecture: `Qwen3_5ForConditionalGeneration`
+- Quantization format: `compressed-tensors`
+- Quantization scheme: `FP8_BLOCK`
+- Weight block size: `128 x 128`
+- Activations: dynamic FP8
+- Activation group size: `128`
+- Quantizer: `llmcompressor 0.13.0`
+- Declared context length: `262,144`
+- Checkpoint size: approximately `29 GB`
+- Native MTP components: retained
+
+The checkpoint was produced from the BF16 Swift-Qwen3.8-27B model rather than
+requantizing an already quantized derivative.
+
+## Quantization details
+
+The quantization process used the official Qwen3.8 FP8 configuration as a
+reference for the block-FP8 layout.
+
+Checkpoint audit:
+
+| Item | Count |
+|---|---:|
+| Total checkpoint tensors | 1,199 |
+| 2D weight tensors | 617 |
+| FP8 quantization candidates | 407 |
+| Effectively excluded / preserved modules | 626 |
+| Incompatible FP8 candidates after validation | 0 |
+
+Matrices that are not compatible with the required `128 x 128` block
+structure were preserved instead of being forcibly quantized.
+
+The following classes of tensors were intentionally preserved where
+appropriate:
+
+- embeddings
+- `lm_head`
+- normalization parameters
+- non-2D weights
+- matrices whose dimensions are incompatible with the FP8 block layout
+
+The native MTP layers are retained. Compatible MTP projection matrices are
+quantized to FP8, while incompatible components remain unquantized.
+
+## Validation
+
+Validated locally on:
+
+- NVIDIA RTX PRO 6000 Blackwell 96 GB
+- SGLang `0.5.19.dev135+ga4ffb996d`
+- `compressed-tensors 0.18.0`
+- CUDA-capable Linux deployment
+- single-GPU tensor parallelism (`TP=1`)
+
+SGLang successfully loads the checkpoint as:
 
-Validated with SGLang on NVIDIA RTX PRO 6000 Blackwell 96 GB.
+```text
+type=Qwen3_5ForConditionalGeneration
+quant=compressed-tensors
+```
+
+Observed target-model weight memory during loading:
+
+```text
+28.47 GB
+```
+
+OpenAI-compatible `/v1/chat/completions` inference was validated successfully.
+
+Multimodal inference has not yet been separately benchmarked for this
+quantized checkpoint.
 
 ## Performance
 
-| Configuration | Throughput |
+All measurements below are local measurements from a single
+NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.
+
+They are intended to document this deployment, not to serve as standardized
+cross-model benchmarks.
+
+### Fixed 4,096-token generation
+
+Same prompt and generation setup for all configurations:
+
+| Configuration | Median throughput |
 |---|---:|
-| Swift FP8 | 50.45 tok/s |
-| Swift FP8 + NEXTN | 103.38 tok/s |
-| Swift FP8 + DFlash2 | **132.62 tok/s** |
+| Swift FP8, target model only | 50.45 tok/s |
+| Swift FP8 + native NEXTN/MTP | 103.38 tok/s |
+| Swift FP8 + DFlash2, 8 draft tokens | **132.62 tok/s** |
+
+Measured DFlash2 runs:
+
+```text
+131.24 tok/s
+132.65 tok/s
+132.62 tok/s
+median: 132.62 tok/s
+```
+
+Compared with target-only generation, DFlash2 produced approximately
+**2.63x** higher output throughput in this test.
+
+Compared with native NEXTN/MTP, DFlash2 was approximately **28% faster**.
+
+The DFlash2 draft model is external and is **not included in this repository**.
+
+## Reasoning-heavy agent test
+
+A separate local A/B test used the same diagnostic prompt, sampling
+parameters, seed, and reasoning setting for both systems.
 
-DFlash2 is external and is not included in this repository.
+The prompt asked the model to diagnose an intermittently slow
+OpenAI-compatible inference deployment with high GPU utilization,
+large KV cache, speculative decoding, variable context sizes and
+concurrency-sensitive latency.
+
+| Metric | Previous Qwen FP8 production | Swift FP8 + DFlash2 |
+|---|---:|---:|
+| Wall time | 193.48 s | **139.63 s** |
+| Prompt tokens | 229 | 229 |
+| Reasoning tokens | 14,838 | **10,716** |
+| Completion tokens | 22,848 | **16,170** |
+| Finish reason | stop | stop |
+| Effective completion throughput | 118.09 tok/s | 115.80 tok/s |
+
+Observed change:
+
+- wall-clock time: approximately **-27.8%**
+- reasoning tokens: approximately **-27.8%**
+- completion tokens: approximately **-29.2%**
+
+The main benefit in this test was not higher raw per-token throughput.
+Swift reached a similarly useful diagnostic answer with substantially fewer
+reasoning and completion tokens.
+
+This is a local workload test and should not be interpreted as a standardized
+quality benchmark.
+
+## SGLang usage
+
+### Basic serving
+
+```bash
+python -m sglang.launch_server \
+  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
+  --served-model-name Swift-Qwen3.8-27B-FP8 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --attention-backend flashinfer \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen3_coder
+```
+
+### Native NEXTN / MTP speculative decoding
+
+The retained native MTP head can be used with SGLang:
+
+```bash
+python -m sglang.launch_server \
+  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
+  --served-model-name Swift-Qwen3.8-27B-FP8 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --attention-backend flashinfer \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen3_coder \
+  --speculative-algorithm NEXTN \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4
+```
+
+In the validated SGLang build, NEXTN is internally represented through the
+EAGLE speculative-decoding path.
+
+### DFlash2 speculative decoding
+
+Best local throughput was obtained with a compatible external DFlash2 draft
+checkpoint:
+
+```bash
+python -m sglang.launch_server \
+  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
+  --served-model-name Swift-Qwen3.8-27B-FP8 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --attention-backend flashinfer \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen3_coder \
+  --speculative-algorithm DFLASH \
+  --speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
+  --speculative-num-draft-tokens 8
+```
+
+The DFlash2 weights are not redistributed in this repository.
+
+Because speculative decoding verifies proposed tokens against the target
+model, the external draft model affects acceptance rate and speed rather than
+replacing the target model's token distribution.
+
+## Notes
+
+This repository contains the quantized target checkpoint only.
+
+It does not include:
+
+- a DFlash2 draft checkpoint
+- the original BF16 Swift checkpoint
+- SGLang runtime binaries or containers
+
+Performance depends heavily on GPU architecture, SGLang version, attention
+backend, context length, concurrency, KV-cache configuration and speculative
+decoding parameters.
 
 ## License
 
-Derived from `ukisai/Swift-Qwen3.8-27b`.
+This checkpoint is derived from:
+
+[`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)
+
+and follows the **Swift Open License v1.0** applicable to the upstream model.
+
+Please refer to the upstream repository and its license text for the
+authoritative licensing terms.
+
+No additional rights to the upstream model are granted by this repository.
+
+## Attribution
+
+Original model:
+
+- UkisAI
+- `ukisai/Swift-Qwen3.8-27b`
+
+FP8 conversion, validation and local performance measurements for this
+repository were performed independently by the repository maintainer.
+
+## Citation
 
-This checkpoint follows the Swift Open License v1.0.
+For the underlying Swift model, please cite or reference the upstream project:
 
-https://huggingface.co/ukisai/Swift-Qwen3.8-27b
+```bibtex
+@misc{swift-qwen3.8-27b,
+  title  = {Swift-Qwen3.8-27B},
+  author = {UkisAI},
+  year   = {2026},
+  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
+}
+```