granite-4.1-8b-8bit-mlx

Quantized version of ibm-granite/granite-4.1-8b for Apple Silicon using MLX.

Quantization: Affine integer quantization
Precision: 8-bit (~8.5 bits/weight avg)
Group size: 64
Disk size: 8499 MB
Quantized by: sahilchachra

About this variant

Affine quantization at 8-bit with group size 64. Closest to FP16 quality. Recommended when memory allows and quality is the priority.

Benchmark results

Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).

Performance

	This model	FP16 baseline
Prefill (tok/s)	438.5	334.25
Decode (tok/s)	32.93	17.88
Peak memory (GB)	9.783	17.549
Disk size (MB)	8499	16778

Quality

Benchmark	This model	FP16 baseline	Task
GSM8K	93.3%	90.0%	Math reasoning (25 samples)
MMLU	56.0%	60.0%	World knowledge (50 samples)
HumanEval	33.3%	30.0%	Code pass@1 (20 samples)

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	31.8
~256 tokens	31.7
~512 tokens	31.7
~1024 tokens	31.5

Usage

Install

pip install mlx-lm

Generate

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/granite-4.1-8b-8bit-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)

Stream

from mlx_lm import load, stream_generate

model, tokenizer = load("sahilchachra/granite-4.1-8b-8bit-mlx")
for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
    print(chunk.text, end="", flush=True)

All variants in this collection

Model	Method	Bits/weight
sahilchachra/granite-4.1-8b-4bit-mlx	Affine int4 (group 64)
sahilchachra/granite-4.1-8b-5bit-mlx	Affine int5 (group 64)
sahilchachra/granite-4.1-8b-6bit-mlx	Affine int6 (group 64)
sahilchachra/granite-4.1-8b-8bit-mlx	Affine int8 (group 64)	← this model
sahilchachra/granite-4.1-8b-mixed4_6-mlx	Mixed 4+6 bit
sahilchachra/granite-4.1-8b-mxfp4-mlx	Block float MX FP4
sahilchachra/granite-4.1-8b-mxfp8-mlx	Block float MX FP8

Notes

• Requires Apple Silicon (M1 or later) with MLX
• Benchmarks run on Apple M5 Pro, 24 GB unified memory
• Sample sizes are small (25–50 per benchmark) — treat accuracy figures as indicative, not definitive
• Base model license: Apache 2.0

Original model

See ibm-granite/granite-4.1-8b for full model details, training information, and intended use.