README.md

---
license: apache-2.0
base_model:
  - ProCreations/grug-27b
library_name: mlx
---

# grug-27b-oQ5E-fp16

**Grug 27B** quantized with **oMLX oQ5E** (exponent-aware 5-bit) at **float16** precision. A token-efficient, reasoning-optimized variant of [ProCreations/grug-27b](https://huggingface.co/ProCreations/grug-27b), built on the Qwen3.6-27B architecture.

## Model Description

Grug 27B is a LoRA fine-tuned reasoning model that adopts the "grug" thinking style — producing concise, token-efficient chain-of-thought reasoning. The model was fine-tuned with `r=32` on all text-stack linear layers and the LoRA weights were merged back into the base bf16 weights. This quantized variant converts the model to **float16** for ~20% faster prefill on Apple Silicon (M1/M2/M3/M4), then applies the **oQ5E** exponent-aware 5-bit quantization scheme from oMLX v0.5.7.

The "grug" thinking style achieves **8–40× fewer tokens** than the base model while maintaining strong benchmark performance, with zero repetition loops observed in stress testing.

### Architecture (Qwen3 Decoder-Only Transformer)

| Parameter | Value |
|---|---|
| Hidden size | 5,120 |
| Num hidden layers | 40 |
| Num attention heads | 40 |
| Num key-value heads (GQA) | 8 |
| Intermediate size | 29,568 |
| Max position embeddings | 131,072 |
| Layer norm eps | 1e-06 |
| RMS norm eps | 1e-05 |
| RoPE theta | 1,000,000.0 |
| Vocabulary size | 152,064 |

## Quantization Details

| Property | Value |
|---|---|
| Quantization format | oQ5E (oMLX 5-bit exponent-aware) |
| oMLX version | 0.5.7 |
| Bits per weight | ~5.8 BPW |
| Base dtype | float16 (fp16) |
| Original dtype | bfloat16 |
| Exponent aware | Yes |
| Quality loss vs bf16 | 1–3% perplexity increase |
| Memory savings | ~62–65% reduction from full precision |
| Estimated quantized size | ~19–21 GB |

The oQ5E format uses exponent-aware quantization to preserve the dynamic range of larger-magnitude weights while compressing the bulk of the weight distribution. The fp16 base dtype was chosen over bf16 for ~20% faster prefill throughput on Apple Silicon GPUs.

## Training Details

| Property | Value |
|---|---|
| Base model | Qwen/Qwen3.6-27B |
| Fine-tune method | LoRA (r=32, all text-stack linears) |
| LoRA merge | Merged into bf16 weights |
| Training data | grug-think-v3-10k agent trajectories + fresh GPT-5.5 set |
| Loss function | Think-only loss on trajectory data; full loss on fresh high-quality data |

## Performance Benchmarks (v2.1)

| Benchmark | Metric | Score |
|---|---|---|
| HumanEval | pass@1 | 87.2 |
| MBPP | pass@1 | 85.0 |
| GSM8K | exact match | 95.5 |
| MATH-500 | (unseen surface) | 68.7 |
| SWE-bench Tool | valid / match / args | 97.1 / 92.6 / 100.0 |

Token efficiency: **8–40× fewer tokens** than the base Qwen3.6-27B model across reasoning tasks.

## Hardware Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| RAM / Unified Memory | 16 GB (M1/M2/M3/M4 base) | 24 GB+ (M1/M2/M3/M4 Pro/Max) |
| Disk space | ~22 GB | ~22 GB |

## Usage

### Python (mlx-lm)

```python
from mlx_lm import load, generate

model, tokenizer = load("your-username/grug-27b-oQ5E-fp16")
response = generate(
    model, tokenizer,
    prompt="Your prompt here",
    max_tokens=1024,
    temp=0.6,
    top_p=0.95
)
print(response)
```

### CLI

```bash
mlx_lm.run   --model your-username/grug-27b-oQ5E-fp16   --max-tokens 4096   --temp 0.6   --top-p 0.95
```

### Recommended Inference Parameters

| Parameter | Value | Notes |
|---|---|---|
| `temp` | 0.6 | Balanced creativity / determinism |
| `top_p` | 0.95 | Standard nucleus sampling |
| `max_tokens` | 4096–16384 | Adjust based on task length |

## License

This quantized model card and accompanying files are released under the **Apache License 2.0**, inherited from the original [ProCreations/grug-27b](https://huggingface.co/ProCreations/grug-27b) model. The base architecture is derived from [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B).

See the original model card for full license terms: https://huggingface.co/ProCreations/grug-27b

## Citation

```bibtex
@misc{grug27b,
  title={Grug 27B — Token-Efficient Reasoning with the "Grug" Thinking Style},
  author={ProCreations},
  year={2025},
  url={https://huggingface.co/ProCreations/grug-27b}
}

@misc{qwen3,
  title={Qwen3.6-27B Technical Report},
  author={Qwen Team},
  year={2025},
  url={https://huggingface.co/Qwen/Qwen3.6-27B}
}
```

## Acknowledgements

- **Qwen Team** for the Qwen3.6-27B base architecture
- **ProCreations** for the original grug-27b model and training methodology
- **oMLX** team for the exponent-aware oQ5E quantization scheme (v0.5.7)
- **mlx-lm** for the inference runtime

## Links

- Original model: https://huggingface.co/ProCreations/grug-27b
- Base model: https://huggingface.co/Qwen/Qwen3.6-27B
- oMLX quantization: https://github.com/ml-explore/mlx-examples