README.md

---
pipeline_tag: image-text-to-text
library_name: mlx-vlm
license: other
base_model: kai-os/Grug-12B
base_model_relation: quantized
tags:
- mlx
- mlx-vlm
- gemma4_unified
- gemma-4
- vision-language
- image-text-to-text
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
datasets:
- hotdogs/uka-glm-5.2
- Scale-or-Reason/general-reasoning-ift-pairs
- samcheng0/lumia-reasoning-sft-v1
- HSH-Intelligence/verified-math-reasoning-3k
- kd13/CodeDebug-Instruct-v2-Reasoning
- Madarabr/cortex-adaptive-thinking
- CL-From-Nothing/code_rose_initial_1_7B_SFT_10K_rollouts_Qwen3-4B-Thinking-2507_k12_t0.7_maxtok12288
---

# Grug-12B VLM MLX

Apple Silicon MLX VLM quantizations of
[`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B), packaged as a
single Hugging Face repo with one folder per quantization level.

`Grug-12B` is a compact-reasoning fine-tune of
[`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it). The
source model was released as merged Transformers/safetensors weights after
QLoRA training. This repo only provides MLX quantized derivatives for Apple
Silicon inference and keeps the original vision-language model structure.

## Highlights

- Vision-language support is preserved through the Gemma 4 unified VLM config.
- Three MLX affine quantizations are available in one repo: 8-bit, 6-bit, and 4-bit.
- Benchmarked with oMLX on the MLX LM engine; screenshots are included below.
- The original BF16 Transformers weights remain in the source repo.

## Available variants

| Variant | Folder | Quantization | Size | Best fit |
| --- | --- | --- | ---: | --- |
| MLX 8-bit | `mlx-8bit/` | affine, group size 64 | 12 GB | Highest-quality local MLX run. |
| MLX 6-bit | `mlx-6bit/` | affine, group size 64 | 9.1 GB | Balanced quality, memory, and speed. |
| MLX 4-bit | `mlx-4bit/` | affine, group size 64 | 6.3 GB | Smallest footprint and best peak memory. |

These are not GGUF files and are not llama.cpp quants. They are MLX safetensors
folders intended for `mlx-vlm`.

## Benchmarks

Benchmarks were run with [oMLX](https://github.com/jundot/omlx), using the
`Force mlx-lm` engine. Each run used prompt prefill sizes of 1024, 4096, and
8192 tokens with 128 generated tokens. Values below are copied from the
captured benchmark output.

Hardware: Apple Mac Studio with M4 Max and 64 GB unified memory.

| Variant | pp1024 tg TPS | pp4096 tg TPS | pp8192 tg TPS | pp8192 E2E | Peak mem |
| --- | ---: | ---: | ---: | ---: | ---: |
| `mlx-8bit` | 30.3 tok/s | 20.4 tok/s | 31.6 tok/s | 20.189 s | 13.80 GB |
| `mlx-6bit` | 38.9 tok/s | 38.7 tok/s | 37.8 tok/s | 19.795 s | 11.03 GB |
| `mlx-4bit` | 21.7 tok/s | 15.7 tok/s | 50.9 tok/s | 18.540 s | 8.26 GB |

Continuous batching at `pp1024 / tg128`:

| Variant | Batch 1 tg TPS | Batch 2 tg TPS | Batch 2 speedup |
| --- | ---: | ---: | ---: |
| `mlx-8bit` | 30.3 tok/s | 34.2 tok/s | 1.13x |
| `mlx-6bit` | 38.9 tok/s | 40.5 tok/s | 1.04x |
| `mlx-4bit` | 21.7 tok/s | 56.1 tok/s | 2.59x |

<details>
<summary>Benchmark screenshots</summary>

### MLX 8-bit

![oMLX benchmark for Grug-12B VLM 8-bit](assets/benchmarks/omlx-8bit.png)

### MLX 6-bit

![oMLX benchmark for Grug-12B VLM 6-bit](assets/benchmarks/omlx-6bit.png)

### MLX 4-bit

![oMLX benchmark for Grug-12B VLM 4-bit](assets/benchmarks/omlx-4bit.png)

</details>

## Usage

Download only the variant you want:

```python
from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-12B-VLM-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```

Run with `mlx-vlm`:

```bash
python -m mlx_vlm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Describe this image." \
  --image /path/to/image.jpg \
  --max-tokens 256
```

For text-only prompts, omit the `--image` argument.

## Provenance and attribution

- Source model: [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B)
- Base model: [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `ad3feab42542e3361dcaf0ebe795d55009765918`
- Conversion target: Gemma 4 unified VLM with `vision_config` preserved

The source model card describes the original training recipe, datasets, local
evaluation, limitations, and acknowledgements. Please refer to that card for
the full model provenance and license context.

## Limitations

Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original BF16 Transformers weights.