---
base_model: google/gemma-4-31b-it
tags:
- awq
- 4-bit
- rdna4
- gfx1201
- rocm
- sglang
- quantized
- thinking
license: apache-2.0
---

# Gemma 4 31B Dense AWQ 4-bit

In-house AWQ 4-bit calibration of [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it), end-to-end from the upstream BF16 base. Thinking + vision aware calibration via `balanced_thinking_vision` corpus (40% AM-Thinking-v1-Distilled / 30% LLaVA-Instruct / 15% NuminaMath / 15% UltraChat).

Replaces the older [`mattbucci/gemma-4-31B-it-AutoRound-AWQ`](https://huggingface.co/mattbucci/gemma-4-31B-it-AutoRound-AWQ) which was a repack of Intel's AutoRound GPTQ output (50.4% negative scales). This ship is fully in-house: standard AWQ scales, thinking traces preserved, vision tower kept BF16.

## Model Details

| | |
|---|---|
| **Base model** | [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it) |
| **Architecture** | Dense with sliding window attention (50 SWA + 10 full attention layers) |
| **Parameters** | 31B |
| **Layers** | 60 |
| **Quantization** | AWQ 4-bit, group_size=128 |
| **Calibration** | 512 samples × 1024 tokens, `balanced_thinking_vision` recipe (text-only — vision tower BF16) |
| **Scale audit** | 0 / 410 quantized tensors flagged (clean) |

## Capability Validation (R9700 / SGLang v0.5.11)

| Probe | Result | Notes |
|---|:---:|---|
| basic ("What is the capital of France?") | ✅ | clean 'paris', finish=stop |
| thinking | ✅ | 460 tok reasoning, terminated cleanly |
| vision (red circle on white) | ⚠ crashes | see Known Limitations |

## Known Limitations

- **Vision: BROKEN on RDNA4.** The model generates a coherent vision response but the server crashes mid-decode with `HSA_STATUS_ERROR_EXCEPTION 0x1016` in `torch_native_backend.py:332 forward_decode`. This is the same upstream "Gemma 4 31B Dense — 400-token attention degradation" issue that affects this dense variant on ROCm regardless of recipe. Cross-team validation on Ampere/3090 stack pending — if Ampere passes, this is purely an RDNA4-side ROCm SDPA limitation. **For vision workloads, use [`mattbucci/gemma-4-26B-AWQ`](https://huggingface.co/mattbucci/gemma-4-26B-AWQ)** (the multimodal MoE flagship, fully working) **or [`mattbucci/Qwen3.6-27B-AWQ`](https://huggingface.co/mattbucci/Qwen3.6-27B-AWQ)** (DeltaNet hybrid VL, smaller but vision works end-to-end on RDNA4).
- **Triton attention degrades at 400+ tokens** on Gemma4's 60-layer SWA. Use `--attention-backend torch_native` (the `gemma4-31b` launch preset already defaults to this).
- **Decode speed**: 15 tok/s single-user on 2x R9700 (BF16 activations + Triton GEMV).

## Usage with SGLang

```bash
git clone https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference
cd 2x-R9700-RDNA4-GFX1201-sglang-inference
./scripts/setup.sh
scripts/launch.sh gemma4-31b
```

The `gemma4-31b` preset uses `torch_native` attention + Triton GEMV with FP32 dequant for stability on RDNA4.

## Hardware

Tested on 2x AMD Radeon AI PRO R9700 (gfx1201, RDNA4, 32+34 GB VRAM) with ROCm 7.2 and SGLang v0.5.11 + RDNA4 patches.

## License

Apache 2.0, inherited from the upstream Gemma 4 base.