README.md

---
license: apache-2.0
tags:
- gemma
- gemma-4
- multimodal
- vision
- audio
- mixed-quantization
- 8bit
- mlx
- inference
pipeline_tag: image-text-to-text
library_name: mlx
language:
- en
base_model:
- google/gemma-4-12b-it
---

# gemma-4-12B-it-Mixed-8bit

**Mixed-precision 8-bit quantization of [Gemma 4 12B IT](https://huggingface.co/google/gemma-4-12b-it)** — Google's unified multimodal model (text + vision + audio + video), optimized for efficient local inference with minimal quality loss.

## Quick Stats

| Metric | Value |
|--------|-------|
| Base model | [google/gemma-4-12b-it](https://huggingface.co/google/gemma-4-12b-it) |
| Architecture | Gemma4Unified (48-layer dense, hybrid attention) |
| Quantization | MLP-only 8-bit, group-size=64 |
| BF16 size | ~24 GB |
| **Mixed 8-bit size** | **~15 GB (≈38% reduction)** |
| Modalities | Text, Vision, Audio, Video |
| License | Gemma License (inherits from base model) |

## Quantization Strategy

This model uses **surgical mixed-precision quantization**: only the SwiGLU MLP projection matrices (`gate_proj`, `up_proj`, `down_proj`) in the language model are quantized to 8-bit. All other components remain in BF16:

- ✅ **Quantized (8-bit)**: LLM MLP layers (144 tensors across 48 layers, ~60% of model mass)
- ✅ **Protected (BF16)**: All attention (sliding + full), vision tower, audio embedder, embeddings, LM head, MultimodalEmbedder, all normalization layers, layer_scalars

**Rationale**: MLP layers are "knowledge lookup" / feature transformation layers that are extremely quantization-tolerant. Attention mechanisms carry reasoning and context-window recall. Vision, audio, and multimodal projection layers are kept in BF16 to preserve modality fidelity.

## Model Architecture

### Language Model (Text)

- **48 dense layers** in pattern `[sliding_attention × 5, full_attention × 1] × 8`
- **Hybrid attention**: Sliding window attention (40 layers, head_dim=256) + full global attention (8 layers, head_dim=512)
- **Heads**: 16 query / 8 KV / 1 global KV
- **Hidden size**: 3840, **Intermediate**: 15360
- **Vocab**: 262,144, **Context**: 262,144 tokens
- **RoPE**: Default (θ=10,000) for sliding; Proportional (θ=1,000,000, partial_rotary=0.25) for full attention
- **No MoE** (dense architecture)

### Vision Tower

- **ViT** with patch_size=16, pooling_kernel=3
- **280 soft tokens** per image, projected to 3840-dim
- Position embedding size: 1120

### Audio Tower

- **Audio embedder**: 640-dim embedding, 640 samples per token
- Simple linear projection (no convolutional audio tower in this checkpoint)

### Multimodal

- **Unified processor**: `Gemma4UnifiedProcessor` handling text, images, audio, and video
- **Special tokens**: `<|image|>`, `<|audio|>`, `<|video|>` with BOI/EOI, BOA/EOA delimiters
- **MultimodalEmbedder** projects vision and audio into the shared 3840-dim text space

## MTP Speculative Decoding

This model supports **Multi-Token Prediction (MTP)** speculative decoding via the companion assistant/drafter model. This model achieves 20-30% faster token generation with gemma-4-12B-it-assistant in BF16 format.

## Usage

### oMLX (Recommended)

Loads natively with MTP and vision support. 

### MLX / mlx-vlm

```python
from mlx_vlm.utils import load

model, processor = load("your-username/gemma-4-12B-it-Mixed-8bit", lazy=True)
# Model loads in ~15 GB vs ~24 GB for BF16
```

## Files

| File | Description |
|------|-------------|
| `model-00001.safetensors` – `model-00003.safetensors` | Sharded model weights (~5 GB each) |
| `config.json` | Model configuration with quantization metadata |
| `tokenizer.json` / `tokenizer_config.json` | Tokenizer (262K vocab) |
| `chat_template.jinja` | Chat template with thinking/tool support |
| `generation_config.json` | Generation parameters |
| `processor_config.json` | Unified processor config |

## Quantization Details

| Parameter | Value |
|-----------|-------|
| Quantization mode | Affine (8-bit) |
| Group size | 64 |
| Layers quantized | 48 (all MLP: gate_proj, up_proj, down_proj) |
| Tensors quantized | ~144 |
| Layers protected | Attention (Q/K/V/O), Vision, Audio, Embeddings, Norms, Scalars |
| Script | `quantization_scripts/mixed_quantize_gemma4_12b.py` |

## Reproducing

```bash
# Quantize from the BF16 base model
mlx-env/bin/python quantization_scripts/mixed_quantize_gemma4_12b.py \
    --bits 8 --group-size 64
```

The script:
1. Lazily loads the BF16 checkpoint via `mlx_vlm.utils.load()`
2. Applies 8-bit quantization to MLP projections only (via `class_predicate` filter)
3. Saves config with quantization metadata
4. Flattens, evaluates, and shards to 5 GB safetensors

## Limitations

- Mixed quantization may show minor quality degradation on edge cases compared to BF16
- The vision tower, audio embedder, and multimodal projector are kept in BF16 — they are not quantized in this version
- Audio support uses a simple linear projection (no convolutional audio tower); complex audio understanding may be limited
- MTP speculative decoding requires the companion assistant model to be loaded alongside
- The base model uses `transformers_version: 5.10.0.dev0` — some fields may not be recognized by stable transformers releases yet

## License

This model inherits the Apache 2.0 license of the base [google/gemma-4-12b-it](https://huggingface.co/google/gemma-4-12b-it).