Bruce001/gemma4-12b-ashq1-4850

Gemma-4-12B-Obliterated (ASHQ1-4850 Mix-Quant GGUF)

This repository contains an experimental, highly optimized ASHQ1 (Adaptive Layer-Budget Hybrid Quantization) GGUF quantization of Gemma-4-12B-Obliterated.

By employing importance matrix (imatrix) guided layer-by-layer classification, this model achieves a balance: preserving language/reasoning capabilities at Q4-level quality while keeping the total size within 4.85 GB (3.2 BPW) （not tested yet）.

---

🌟 Key Experiment Highlights

Unlike traditional uniform quantization (such as standard IQ3_M or Q3_K_M), which applies the same bit-width indiscriminately across all layers, this ASHQ1-4850 build uses dynamic layer budget allocation:

• Attention Layers Protection (Q4_K): Attention mechanisms (attn_q, attn_k, attn_v, attn_output) are prioritized and bumped up to Q4_K. This preserves in-context reasoning, instruction following, and needle-in-a-haystack capabilities.
• Normalization & Scales Saved (F16): All norm and scale tensors are strictly locked at F16 (~1.5 MiB total) to prevent numerical instability and activation overflow.
• FFN Down-Projection Compression (IQ2_XXS): The redundant FFN down-projection layers (ffn_down) are aggressively quantized to IQ2_XXS to squeeze the overall file size down to meet strict RAM/VRAM budgets.
• FFN Up/Gate Projections (IQ4_XS): Gate and Up projections are kept at higher precision to retain dense feature activation before down-projection.

---

📊 Quantization Allocation & Strategy

Layer / Tensor Type	Assigned Precision	Rationale / Strategic Value
All Norms & Scales	F16	Zero accuracy loss on critical activation boundaries with negligible size penalty (~1.5 MiB).
Attention Key/Query/Value	Q4_K	Protects self-attention matrix and long-context reasoning (~1.83 GB budget).
FFN Gate & Up Projections	IQ4_XS	Retains feature expansion richness.
FFN Down Projections	IQ2_XXS	Trades non-critical factual memory density for extreme file compression.
Token Embeddings	Q4_K	Prevents vocabulary collapse.
Output Head	Q5_K	Ensures precise token probability distributions.

Target Budget: 4850 MiB (~4.74 GB)
Actual Size: ~4.75 GB
Effective Bit-Per-Weight (BPW): ~3.2 bpw

---

💡 Performance Expectation

• Logic & Instruction Following: ~85%–90% of native Q4_K_M. Handles multi-step reasoning, JSON formatting, and code structure seamlessly.
• Factual Knowledge: Slightly reduced compared to uncompressed models due to 2-bit FFN down-projections (rare facts/niche details may show hallucination).
• Best Suited For: Hardware constrained environments (e.g., 6GB VRAM GPUs, 8GB Apple Silicon Macs, or edge deployment) where standard Q4_K (~7.2GB) cannot fit.

---

🚀 How to Run

1. Using llama.cpp (Command Line)

Make sure you have an up-to-date build of llama.cpp supporting Gemma 4 architecture:

./llama-cli -m gemma4-12b-obliterated-ashq1-4850.gguf \
    -p "You are a helpful AI assistant." \
    -n 512 \
    --ngl 99 \
    --temp 0.7

2. Using Ollama

Create a file named Modelfile:

Dockerfile

FROM ./gemma4-12b-obliterated-ashq1-4850.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Run the model:

Bash

ollama create gemma4-ashq1 -f Modelfile
ollama run gemma4-ashq1

3. LM Studio / Chatbox

1. Move gemma4-12b-obliterated-ashq1-4850.gguf to your local LM Studio models folder (~/.cache/lm-studio/models/ or custom directory).
2. Select the model from the top bar and start chatting.

🛠️ Reproduction & Quantization Logs

Quantized using the ASHQ1 framework with llama.cpp CUDA build.

Bash

python3 main.py \
    --model /path/to/gemma4-f16.gguf \
    --imatrix /path/to/Gemma-4-12B-OBLITERATED.imatrix.gguf \
    --size 4850 \
    --allow-q3-or-lower \
    --output ./gemma4-12b-obliterated-ashq1-4850.gguf \
    --run

Acknowledgements

• Base Model: Gemma-4-12B-Obliterated
• Quantization Algorithm: ASHQ1 Engine by @wepiqx
• Inference Backend: llama.cpp