gemma-4-31B-it-scotoma-2-GPTQ-W8A16

This is an 8-bit weight-only (W8A16) GPTQ quantization of ReadyArt/gemma-4-31B-it-scotoma-2, produced with llmcompressor and intended for fast, memory-efficient inference with vLLM.

About the base model

scotoma-2 is a 31B-parameter derivative of Google's gemma-4-31B-it. Per its model card, it applies a bounded refusal-direction edit (an abliteration LoRA projected through Gemma's J-Space) combined with several rounds of DPO preference training aimed at reducing repetitive stylistic tics (reflexive negation, em-dash asides, stacked adjectives) rather than removing the base model's safety behavior — the authors describe it explicitly as "not uncensored." See the base model's own card for full details on the method and its limitations; this repo only covers the quantization and adds nothing to the underlying behavior.

Quantization details

Method	GPTQ (GPTQModifier, one-shot) via llmcompressor
Scheme	W8A16 — 8-bit integer weights, 16-bit activations
Format	compressed-tensors (native vLLM support)
Calibration data	128 samples from HuggingFaceH4/ultrachat_200k (train_sft split)
Calibration sequence length	2048 tokens
Layers excluded from quantization	lm_head, embedding layers, vision tower layers
Quantized on	NVIDIA A100 80GB

Weight-only quantization keeps activations at full precision, which preserves accuracy well while roughly halving VRAM footprint relative to bf16. It primarily helps memory usage and single/low-batch latency; for very high-throughput serving, full activation quantization (W8A8) can offer more gains, but that path isn't well supported on Ampere-class GPUs, making W8A16 the practical choice here.

Usage

vLLM (recommended)

vllm serve Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16

vLLM will auto-detect the compressed-tensors quantization config from the checkpoint. On Ampere (A100/A10) GPUs this runs through the Marlin weight-only kernel automatically.

from vllm import LLM, SamplingParams

llm = LLM(model="Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
sampling_params = SamplingParams(temperature=1.0, max_tokens=512)

output = llm.generate(["Your prompt here"], sampling_params)
print(output[0].outputs[0].text)

Transformers

The checkpoint also loads directly via transformers + compressed-tensors for testing outside vLLM, though vLLM is recommended for production serving speed.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")

Hardware requirements

Quantized weights occupy roughly half the VRAM of the bf16 original (~31B params → ~31 GB in int8 vs. ~62 GB in bf16, before KV cache and activation overhead). Tested on a single A100 80GB; should also fit on a single A100 40GB or comparable Ampere GPU depending on context length and batch size.

Notes and limitations

• This is a weight-only quantization; no dataset filtering, safety alignment, or behavioral changes were made during this process. All behavior inherited from the base model (including the caveats noted in its own card) applies unchanged here.
• Quantization can introduce small quality regressions versus the bf16 original, particularly on long-context or reasoning-heavy tasks. If you notice degradation, consider comparing outputs against the original model before relying on this checkpoint for sensitive use cases.
• Licensing follows the base model's terms (Gemma license, as inherited from google/gemma-4-31B-it and passed through ReadyArt/gemma-4-31B-it-scotoma-2). Review the base model's license and usage restrictions before deploying.

Credits

• Base model & fine-tune: ReadyArt
• Original architecture: Google, Gemma 4
• Quantization: this repo, via llmcompressor GPTQModifier (W8A16)