Create README.md

README.md

@@ -0,0 +1,85 @@
+---
+base_model: ReadyArt/gemma-4-31B-it-scotoma-2
+tags:
+  - gptq
+  - quantized
+  - w8a16
+  - int8
+  - compressed-tensors
+  - vllm
+  - gemma4
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# gemma-4-31B-it-scotoma-2-GPTQ-W8A16
+
+This is an 8-bit weight-only (W8A16) GPTQ quantization of [ReadyArt/gemma-4-31B-it-scotoma-2](https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2), produced with [`llmcompressor`](https://github.com/vllm-project/llm-compressor) and intended for fast, memory-efficient inference with vLLM.
+
+## About the base model
+
+scotoma-2 is a 31B-parameter derivative of Google's `gemma-4-31B-it`. Per its model card, it applies a bounded refusal-direction edit (an abliteration LoRA projected through Gemma's J-Space) combined with several rounds of DPO preference training aimed at reducing repetitive stylistic tics (reflexive negation, em-dash asides, stacked adjectives) rather than removing the base model's safety behavior — the authors describe it explicitly as **"not uncensored."** See the base model's own card for full details on the method and its limitations; this repo only covers the quantization and adds nothing to the underlying behavior.
+
+## Quantization details
+
+| | |
+|---|---|
+| Method | GPTQ (`GPTQModifier`, one-shot) via `llmcompressor` |
+| Scheme | W8A16 — 8-bit integer weights, 16-bit activations |
+| Format | `compressed-tensors` (native vLLM support) |
+| Calibration data | 128 samples from `HuggingFaceH4/ultrachat_200k` (train_sft split) |
+| Calibration sequence length | 2048 tokens |
+| Layers excluded from quantization | `lm_head`, embedding layers, vision tower layers |
+| Quantized on | NVIDIA A100 80GB |
+
+Weight-only quantization keeps activations at full precision, which preserves accuracy well while roughly halving VRAM footprint relative to bf16. It primarily helps memory usage and single/low-batch latency; for very high-throughput serving, full activation quantization (W8A8) can offer more gains, but that path isn't well supported on Ampere-class GPUs, making W8A16 the practical choice here.
+
+## Usage
+
+### vLLM (recommended)
+
+```bash
+vllm serve Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16
+```
+
+vLLM will auto-detect the `compressed-tensors` quantization config from the checkpoint. On Ampere (A100/A10) GPUs this runs through the Marlin weight-only kernel automatically.
+
+```python
+from vllm import LLM, SamplingParams
+
+llm = LLM(model="Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
+sampling_params = SamplingParams(temperature=1.0, max_tokens=512)
+
+output = llm.generate(["Your prompt here"], sampling_params)
+print(output[0].outputs[0].text)
+```
+
+### Transformers
+
+The checkpoint also loads directly via `transformers` + `compressed-tensors` for testing outside vLLM, though vLLM is recommended for production serving speed.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16",
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained("Captain1Q/gemma-4-31B-it-scotoma-2-GPTQ-W8A16")
+```
+
+## Hardware requirements
+
+Quantized weights occupy roughly half the VRAM of the bf16 original (~31B params → ~31 GB in int8 vs. ~62 GB in bf16, before KV cache and activation overhead). Tested on a single A100 80GB; should also fit on a single A100 40GB or comparable Ampere GPU depending on context length and batch size.
+
+## Notes and limitations
+
+- This is a **weight-only** quantization; no dataset filtering, safety alignment, or behavioral changes were made during this process. All behavior inherited from the base model (including the caveats noted in its own card) applies unchanged here.
+- Quantization can introduce small quality regressions versus the bf16 original, particularly on long-context or reasoning-heavy tasks. If you notice degradation, consider comparing outputs against the original model before relying on this checkpoint for sensitive use cases.
+- Licensing follows the base model's terms (Gemma license, as inherited from `google/gemma-4-31B-it` and passed through `ReadyArt/gemma-4-31B-it-scotoma-2`). Review the base model's license and usage restrictions before deploying.
+
+## Credits
+
+- Base model & fine-tune: [ReadyArt](https://huggingface.co/ReadyArt)
+- Original architecture: Google, Gemma 4
+- Quantization: this repo, via `llmcompressor` GPTQModifier (W8A16)