File size: 4,152 Bytes
37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d 37396c2 876351d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
base_model:
- Qwen/Qwen3.5-2B
license: apache-2.0
---
# Vishva007/Qwen3.5-2B-W4A16-AutoRound
This is a **W4A16 (4-bit weight, 16-bit activation)** quantized version of [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B), produced using [AutoRound](https://github.com/intel/auto-round) — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention.
**MTP** Enabled model quantization
## Quantization Details
| Parameter | Value |
|---|---|
| Method | AutoRound (W4A16) |
| Group Size | 32 |
| Symmetric | Yes |
| Iterations | 1000 |
| Calibration Samples | 512 |
| Sequence Length | 4096 |
| Torch Compile | Enabled |
## Key Notes
- **High accuracy configuration** — 1000 iterations with 512 calibration samples targets production-grade quality with minimal degradation from the base model.
- **W4A16** — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
- **~50% memory reduction** compared to the FP16 base model, enabling deployment on consumer and mid-range GPUs.
- **MTP (Multi-Token Prediction) Enabled** — Supports speculative decoding for faster inference.
## MTP / Speculative Decoding
This model supports **Multi-Token Prediction (MTP)** for improved inference throughput using speculative decoding.
When serving with compatible backends (e.g., vLLM), enable MTP using:
```bash
--speculative_config '{"method":"mtp","num_speculative_tokens":3}'
````
### Notes
* `num_speculative_tokens=1` is a stable default for balancing speed and accuracy.
* You can experiment with higher values for better throughput, depending on your hardware and latency requirements.
## Usage
This model is compatible with `transformers` and backends that support AutoRound GPTQ-format weights (e.g., vLLM, SGLang, AutoGPTQ). For full model details, architecture, and capabilities, refer to the [base model page](https://huggingface.co/Qwen/Qwen3.5-2B).
### 🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
> 🎁 **Need GPU compute?** Sign up via [RunPod](https://runpod.io?ref=iabrlp7z) and get **$5–$500 in free credits** when you add your first $10.
#### PyTorch 2.13
| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.13 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.13-runpod` | `gmlupxnxfk` | [](https://runpod.io/console/deploy?template=gmlupxnxfk&ref=iabrlp7z) |
| **PyTorch 2.13 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.13-runpod` | `y3j8xvk4f4` | [](https://runpod.io/console/deploy?template=y3j8xvk4f4&ref=iabrlp7z) |
| **PyTorch 2.13 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.13-runpod` | `vigpissn5w` | [](https://runpod.io/console/deploy?template=vigpissn5w&ref=iabrlp7z) |
#### PyTorch 2.12
| Template | CUDA Version | Docker Image | Template ID | Deploy |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch 2.12 (CUDA 12.6)** | 12.6 | `vishva123/cuda-12.6-pytorch-2.12-runpod` | `ctmz86zmf0` | [](https://runpod.io/console/deploy?template=ctmz86zmf0&ref=iabrlp7z) |
| **PyTorch 2.12 (CUDA 13.0)** | 13.0 | `vishva123/cuda-13.0-pytorch-2.12-runpod` | `qjko5yiwzi` | [](https://runpod.io/console/deploy?template=qjko5yiwzi&ref=iabrlp7z) |
| **PyTorch 2.12 (CUDA 13.2)** | 13.2 | `vishva123/cuda-13.2-pytorch-2.12-runpod` | `ifg6xmye0f` | [](https://runpod.io/console/deploy?template=ifg6xmye0f&ref=iabrlp7z) |
--- |