README.md

---
base_model: ReadyArt/gemma-4-31B-it-scotoma-2
tags:
  - gptq
  - quantized
  - w8a16
  - int8
  - compressed-tensors
  - vllm
  - gemma4
library_name: transformers
pipeline_tag: text-generation
---

# gemma-4-31B-it-scotoma-2-GPTQ-W8A16

This is an 8-bit weight-only (W8A16) GPTQ quantization of [ReadyArt/gemma-4-31B-it-scotoma-2](https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2), produced with [`llmcompressor`](https://github.com/vllm-project/llm-compressor) and intended for fast, memory-efficient inference with vLLM.

## About the base model

scotoma-2 is a 31B-parameter derivative of Google's `gemma-4-31B-it`. Per its model card, it applies a bounded refusal-direction edit (an abliteration LoRA projected through Gemma's J-Space) combined with several rounds of DPO preference training aimed at reducing repetitive stylistic tics (reflexive negation, em-dash asides, stacked adjectives) rather than removing the base model's safety behavior — the authors describe it explicitly as **"not uncensored."** See the base model's own card for full details on the method and its limitations; this repo only covers the quantization and adds nothing to the underlying behavior.

## Quantization details

| | |
|---|---|
| Method | GPTQ (`GPTQModifier`, one-shot) via `llmcompressor` |
| Scheme | W8A16 — 8-bit integer weights, 16-bit activations |
| Format | `compressed-tensors` (native vLLM support) |
| Calibration data | 128 samples from `HuggingFaceH4/ultrachat_200k` (train_sft split) |
| Calibration sequence length | 2048 tokens |
| Layers excluded from quantization | `lm_head`, embedding layers, vision tower layers |
| Quantized on | NVIDIA A100 80GB |

Weight-only quantization keeps activations at full precision, which preserves accuracy well while roughly halving VRAM footprint relative to bf16. It primarily helps memory usage and single/low-batch latency; for very high-throughput serving, full activation quantization (W8A8) can offer more gains, but that path isn't well supported on Ampere-class GPUs, making W8A16 the practical choice here.

## Usage

### vLLM (recommended)

```bash
vllm serve Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16
```

vLLM will auto-detect the `compressed-tensors` quantization config from the checkpoint. On Ampere (A100/A10) GPUs this runs through the Marlin weight-only kernel automatically.

```python
from vllm import LLM, SamplingParams

llm = LLM(model="Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
sampling_params = SamplingParams(temperature=1.0, max_tokens=512)

output = llm.generate(["Your prompt here"], sampling_params)
print(output[0].outputs[0].text)
```

### Transformers

The checkpoint also loads directly via `transformers` + `compressed-tensors` for testing outside vLLM, though vLLM is recommended for production serving speed.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
```

## Hardware requirements

Quantized weights occupy roughly half the VRAM of the bf16 original (~31B params → ~31 GB in int8 vs. ~62 GB in bf16, before KV cache and activation overhead). Tested on a single A100 80GB; should also fit on a single A100 40GB or comparable Ampere GPU depending on context length and batch size.

## Notes and limitations

- This is a **weight-only** quantization; no dataset filtering, safety alignment, or behavioral changes were made during this process. All behavior inherited from the base model (including the caveats noted in its own card) applies unchanged here.
- Quantization can introduce small quality regressions versus the bf16 original, particularly on long-context or reasoning-heavy tasks. If you notice degradation, consider comparing outputs against the original model before relying on this checkpoint for sensitive use cases.
- Licensing follows the base model's terms (Gemma license, as inherited from `google/gemma-4-31B-it` and passed through `ReadyArt/gemma-4-31B-it-scotoma-2`). Review the base model's license and usage restrictions before deploying.

## Credits

- Base model & fine-tune: [ReadyArt](https://huggingface.co/ReadyArt)
- Original architecture: Google, Gemma 4
- Quantization: this repo, via `llmcompressor` GPTQModifier (W8A16)