File size: 4,414 Bytes
5d3a8a0 6531282 5d3a8a0 80cc9af 5d3a8a0 80cc9af 5d3a8a0 80cc9af 5d3a8a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | ---
language:
- en
- zh
- ja
- ko
- de
- fr
- es
- pt
- it
- ar
- hi
license: gemma
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
library_name: mlx
pipeline_tag: image-text-to-text
base_model: 0xSero/gemma-4-19b-a4b-it-REAP
tags:
- mlx
- gemma4
- reap
- ple-safe
- quantized
- apple-silicon
- vision
- moe
---
# gemma-4-19b-a4b-it-REAP-MLX-4bit
**PLE-safe** MLX 4-bit weights for [0xSero/gemma-4-19b-a4b-it-REAP](https://huggingface.co/0xSero/gemma-4-19b-a4b-it-REAP) on Apple Silicon.
[REAP](https://arxiv.org/abs/2510.13999) (Router-weighted Expert Activation Pruning) removes 30% of MoE experts while keeping the same active parameters per token (8 of 90 experts selected). Combined with PLE-safe 4-bit quantization, this model runs in **12.6 GB** — fits on 24GB+ Macs.
| | Original 26B | REAP (19B MoE (30% pruned)) | **This model** |
|---|---|---|---|
| Experts/layer | 128 | 90 | 90 |
| Precision | BF16 | BF16 | **4-bit** |
| Disk size | ~52 GB | ~36 GB | **12.6 GB** |
## Also available
- [gemma-4-21b-a4b-it-REAP-MLX-4bit](https://huggingface.co/ukint-vs/gemma-4-21b-a4b-it-REAP-MLX-4bit) — 21B MoE (20% pruned), 13.9 GB
## Accuracy Benchmarks
0-shot generative, thinking enabled, 50 samples per task, identical eval harness. Apple M4 Max 36GB.
| Task | 26B-A4B 4-bit (16.4 GB) | **This model (12.6 GB)** |
|---|---|---|
| Elementary Mathematics | 84% | **44%** |
| Philosophy | 66% | **54%** |
| World Religions | 66% | **34%** |
| College Computer Science | 58% | **34%** |
| High School Mathematics | 26% | **22%** |
| Abstract Algebra | 44% | **36%** |
| College Mathematics | 36% | **16%** |
| Gsm8K | 64% | **62%** |
The 30% expert pruning compounds with 4-bit quantization. Note: high extraction failure rates (up to 60%) on some tasks — the model generates verbose explanations instead of single-letter answers, so true accuracy may be higher than reported. Consider the [21B variant](https://huggingface.co/ukint-vs/gemma-4-21b-a4b-it-REAP-MLX-4bit) for better accuracy.
Extraction failures (unparseable responses) are counted as incorrect. REAP-19B: 113/400 (28%). True accuracy may be higher. Full methodology: [GitHub](https://github.com/ukint-vs/mlx-gemma4-reap).
## Quantization Details
- **Bits:** 4
- **Group size:** 64
- **Strategy:** PLE-safe — only large `nn.Linear` and `SwitchLinear` (MoE expert) layers are quantized. All PLE/ScaledLinear/vision layers stay in bf16.
| Quantized (4-bit) | Kept in bf16 |
|---|---|
| Attention projections (q/k/v/o_proj) | ScaledEmbedding (embed_tokens) |
| MLP layers (gate/up/down_proj) | ScaledLinear (PLE pathway) |
| MoE expert layers (SwitchLinear) | Per-layer embeddings (per_layer_*) |
| | Vision encoder |
| | All norms and scalars |
## Usage
```bash
pip install -U mlx-vlm
```
### Vision
```python
from mlx_vlm import load, generate
model, processor = load("ukint-vs/gemma-4-19b-a4b-it-REAP-MLX-4bit")
tokenizer = processor.tokenizer
messages = [{"role": "user", "content": [
{"type": "image", "url": "photo.jpg"},
{"type": "text", "text": "Describe this image in detail."},
]}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
out = generate(model, processor, prompt, ["photo.jpg"],
max_tokens=200, repetition_penalty=1.2, temperature=0.7)
print(out.text)
```
### Text
```python
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
out = generate(model, processor, prompt, max_tokens=100, temperature=0.0)
print(out.text)
```
## Validation
Trimodal validation: 10/10 vision, 3/3 chat (EN/ZH/JA). Full results: [GitHub](https://github.com/ukint-vs/mlx-gemma4-reap).
## Bugs Fixed in mlx-vlm
| # | Bug | Fix |
|---|---|---|
| 1 | `ScaledLinear` inherits `nn.Module` not `nn.Linear` | Change to `ScaledLinear(nn.Linear)` |
| 2 | Standard quantization quantizes PLE layers | PLE-safe `class_predicate` |
| 3 | `processor.save_pretrained()` strips audio config | Copy `processor_config.json` from source |
| 4 | `SwitchLinear` (MoE) not quantized | Check `hasattr(module, 'to_quantized')` |
| 5 | `embed_scale` double-scaling (mlx-vlm 0.4.4+) | Set `Gemma4TextModel.embed_scale = 1.0` |
## License
Model weights: [Google Gemma License](https://ai.google.dev/gemma/terms). Quantization scripts: MIT.
|