---
base_model: Jackrong/Qwopus3.5-9B-Coder
base_model_relation: quantized
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
  - awq_w4a16
  - compressed-tensors
  - quantized
  - vllm
---
# Qwopus3.5-9B-Coder - AWQ-W4A16

[`Jackrong/Qwopus3.5-9B-Coder`](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder) quantized to **AWQ-W4A16** (4-bit weights).

## What this is

Same footprint as W4A16 but calibrates faster and usually holds up better on instruction-tuned and multilingual models. Start here.

> **Caveat.** Asymmetric; a few older vLLM kernels prefer symmetric W4A16.

## Details

| | |
|---|---|
| Source | `Jackrong/Qwopus3.5-9B-Coder` |
| Scheme | `AWQ-W4A16` (4-bit) |
| Format | compressed-tensors |
| Parameters | 9.7B |
| Size on disk | 8.6 GB |
| Compression | 2.24x smaller than the 19.3 GB source |
| Calibration | `HuggingFaceH4/ultrachat_200k`, 256 samples |
| Left unquantized | `lm_head`, `re:.*visual.*`, `re:.*vision_tower.*`, `re:.*vision_model.*`, `re:.*vision.*`, `re:.*multi_modal_projector.*`, `re:.*merger.*` |
| Quantized on | H100 NVL |
| Quantized by | [Sohailhosseini](https://huggingface.co/Sohailhosseini) |

## Usage

```bash
vllm serve Sohailhosseini/Qwopus3.5-9B-Coder-AWQ-W4A16 \
  --max-model-len 32768
```

```python
from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Sohailhosseini/Qwopus3.5-9B-Coder-AWQ-W4A16", max_model_len=32768)
    out = llm.chat(
        [{"role": "user", "content": "What is quantization? Answer in one sentence."}],
        SamplingParams(temperature=0.6, max_tokens=512),
    )
    print(out[0].outputs[0].text)
```

## Provenance

Produced with HF-quantized. `recipe.yaml` in this repo is the exact modifier stack that was applied, and the scheme, ignored layers, calibration set and hardware are in the table above.

Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.