---
license: apache-2.0
base_model: yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1
base_model_relation: quantized
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- quantized
- mixed-precision
- 4bit
- 8bit
- optiq
- apple-silicon
- image-text-to-text
- vision-language
- gemma4
- coding
- code
- reasoning
- thinking
---

# mlx-community/gemma-4-12B-coder-fable5-composer2.5-v1-OptiQ-4bit

> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)

A 4-bit mixed-precision MLX quant of [yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1), a coding fine-tune of Gemma-4-12B. Sensitive layers are kept at 8-bit and robust ones at 4-bit.

22 GB of bf16 weights become **8.4 GB**, which fits comfortably on a 16 GB Mac.

**Image input works.** Gemma-4-12B is the encoder-free `gemma4_unified` variant, and its multimodal embedder is kept at bf16 in a sidecar.

## Quantization details

| Property | Value |
|---|---|
| Predominant precision | 4-bit |
| Layers at 8-bit (sensitive) | 156 |
| Layers at 4-bit (robust) | 172 |
| Total quantized layers | 328 |
| Achieved bits per weight | 5.216 |
| Group size | 64 |
| Multimodal embedder | bf16, in `optiq/optiq_vision.safetensors` |
| Size on disk | 8.4 GB, from a 22 GB bf16 base |

We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is the predominant precision, not the weighted average.

### How the bit-widths were chosen

The per-layer allocation is transferred from [mlx-community/gemma-4-12B-it-OptiQ-4bit](https://huggingface.co/mlx-community/gemma-4-12B-it-OptiQ-4bit), where it was derived by a KL-divergence sensitivity sweep against the bf16 reference on a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix).

This model is a fine-tune of `google/gemma-4-12B-it` with an unchanged architecture (every shape field of the text config matches), so all 328 quantizable layers map across exactly and the allocation lands at the same 5.216 bits per weight when recomputed against this model's own tensors.

These are **measured** bit-widths, not a static rule-of-thumb recipe. But they were measured on the base model, not on this fine-tune. Fine-tuning shifts weights, so this model's own per-layer sensitivities could differ somewhat from the base's. Which layers are fragile is mostly a property of the architecture, so the transfer is sound, but it is a transfer and you should know that.

Only the language tower is quantized. The multimodal embedder stays at bf16, which is how every OptiQ VLM ships.

## Usage

`gemma4_unified` needs OptiQ's model registration, so load it through `mlx-optiq`:

```bash
pip install mlx-optiq
```

```python
import optiq  # registers the gemma4_unified architecture with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/gemma-4-12B-coder-fable5-composer2.5-v1-OptiQ-4bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a Python function to merge two sorted lists."}],
    add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))
```

Or serve it over an OpenAI-compatible endpoint:

```bash
optiq serve --model mlx-community/gemma-4-12B-coder-fable5-composer2.5-v1-OptiQ-4bit
```

This model reasons before answering (it emits a `<|channel>thought` block), so give it enough `max_tokens` to finish.

## Verification

Generation was exercised on the finished artifact before release: factual recall, arithmetic, and code generation (it correctly writes `is_palindrome`, reasoning about edge cases such as the empty string).

The quantization was also checked numerically: dequantizing individual layers out of the artifact and comparing them against the bf16 checkpoint gives 0.6-0.7% mean relative error on the 8-bit layers and 11-13% on the 4-bit layers, which is what each bit-width should cost.

No task benchmarks were run on this quant; for measured quality numbers on the base architecture, see the [gemma-4-12B-it OptiQ card](https://huggingface.co/mlx-community/gemma-4-12B-it-OptiQ-4bit).

Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as [the original](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1).