File size: 3,632 Bytes
710aad6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
language: en
license: apache-2.0
base_model: ibm-granite/granite-4.1-8b
tags:
- mlx
- quantized
- granite
- apple-silicon
---
# granite-4.1-8b-mxfp4-mlx
Quantized version of [ibm-granite/granite-4.1-8b](https://huggingface.co/ibm-granite/granite-4.1-8b) for Apple Silicon using [MLX](https://github.com/ml-explore/mlx).
**Quantization**: Block floating-point MX FP4 (microscaling)
**Precision**: ~4 bits/weight
**Group size**: 32
**Disk size**: 4253 MB
**Quantized by**: [sahilchachra](https://huggingface.co/sahilchachra)
## About this variant
Microscaling (MX) block floating-point quantization at FP4 precision. Uses a shared floating-point exponent per block of 32 weights instead of integer affine scaling. Different numerical properties vs affine int4 — may suit different workloads.
## Benchmark results
Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).
### Performance
| | This model | FP16 baseline |
|---|---:|---:|
| Prefill (tok/s) | 549.08 | 334.25 |
| Decode (tok/s) | 61.9 | 17.88 |
| Peak memory (GB)| 5.37 | 17.549 |
| Disk size (MB) | 4253 | 16778 |
### Quality
| Benchmark | This model | FP16 baseline | Task |
|---|---:|---:|---|
| GSM8K | 76.7% | 90.0% | Math reasoning (25 samples) |
| MMLU | 56.0% | 60.0% | World knowledge (50 samples) |
| HumanEval | 30.0% | 30.0% | Code pass@1 (20 samples) |
### Context scaling (decode tok/s)
| Context length | Decode tok/s |
|---:|---:|
| ~128 tokens | 59.7 |
| ~256 tokens | 59.4 |
| ~512 tokens | 59.4 |
| ~1024 tokens | 58.8 |
## Usage
### Install
```bash
pip install mlx-lm
```
### Generate
```python
from mlx_lm import load, generate
model, tokenizer = load("sahilchachra/granite-4.1-8b-mxfp4-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)
```
### Stream
```python
from mlx_lm import load, stream_generate
model, tokenizer = load("sahilchachra/granite-4.1-8b-mxfp4-mlx")
for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
print(chunk.text, end="", flush=True)
```
## All variants in this collection
| Model | Method | Bits/weight |
|---|---|---|
| [sahilchachra/granite-4.1-8b-4bit-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-4bit-mlx) | Affine int4 (group 64) | |
| [sahilchachra/granite-4.1-8b-5bit-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-5bit-mlx) | Affine int5 (group 64) | |
| [sahilchachra/granite-4.1-8b-6bit-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-6bit-mlx) | Affine int6 (group 64) | |
| [sahilchachra/granite-4.1-8b-8bit-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-8bit-mlx) | Affine int8 (group 64) | |
| [sahilchachra/granite-4.1-8b-mixed4_6-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-mixed4_6-mlx) | Mixed 4+6 bit | |
| [sahilchachra/granite-4.1-8b-mxfp4-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-mxfp4-mlx) | Block float MX FP4 | ← this model |
| [sahilchachra/granite-4.1-8b-mxfp8-mlx](https://huggingface.co/sahilchachra/granite-4.1-8b-mxfp8-mlx) | Block float MX FP8 | |
## Notes
- Requires Apple Silicon (M1 or later) with MLX
- Benchmarks run on Apple M5 Pro, 24 GB unified memory
- Sample sizes are small (25–50 per benchmark) — treat accuracy figures as indicative, not definitive
- Base model license: [Apache 2.0](https://huggingface.co/ibm-granite/granite-4.1-8b/blob/main/LICENSE)
## Original model
See [ibm-granite/granite-4.1-8b](https://huggingface.co/ibm-granite/granite-4.1-8b) for full model details, training information, and intended use. |