Gemma-4-26B-A4B-Heretic — MLX 4-bit

MLX 4-bit quantisation of coder3101/gemma-4-26B-A4B-it-heretic.

Changes: weights quantised to 4-bit from the BF16 source with mlx_vlm.convert. No fine-tuning, no merging, no re-alignment.

Measured

Converted and measured on one machine — Apple M3 Ultra, 96 GB unified memory, macOS 27 — as part of a full ladder. Every rung in this family came from the same BF16 source with the same group size, so bit width is the only variable between them.


Size on disk	15.37 GB
Perplexity	168.023
Relative to best rung in family	1.67×
Throughput (1 req / 8 concurrent)	43.8 / 143.8 tok/s

Perplexity measured on allenai/tulu-3-sft-mixture, 192 samples of 512 tokens, seed 123 — identical for every rung.

Perplexity is only comparable within this family. Tokenizers differ between model families, so a number here should never be compared against a different base model's. The × column above is the meaningful one.

Usage

pip install mlx-vlm
mlx_vlm.generate --model shoemoney/Gemma-4-26B-A4B-Heretic-MLX-q4 --prompt "Hello" --max-tokens 256

Load with mlx-vlm, not mlx-lm — this architecture is registered in mlx-vlm.

Provenance

mlx_vlm.convert --hf-path coder3101/gemma-4-26B-A4B-it-heretic \
                --mlx-path Gemma-4-26B-A4B-Heretic-q4 -q --q-bits 4 --q-group-size 64

License

apache-2.0, inherited from the base model. Attribution above.

Downloads last month: 54

Safetensors

Model size

26B params

Tensor type

U32

BF16

MLX

Hardware compatibility

4-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for shoemoney/Gemma-4-26B-A4B-Heretic-MLX-q4

Base model

google/gemma-4-26B-A4B

Finetuned

google/gemma-4-26B-A4B-it

Finetuned

coder3101/gemma-4-26B-A4B-it-heretic

Quantized

(28)

this model