grug-27b-oQ5E-fp16

Grug 27B quantized with oMLX oQ5E (exponent-aware 5-bit) at float16 precision. A token-efficient, reasoning-optimized variant of ProCreations/grug-27b, built on the Qwen3.6-27B architecture.

Model Description

Grug 27B is a LoRA fine-tuned reasoning model that adopts the "grug" thinking style — producing concise, token-efficient chain-of-thought reasoning. The model was fine-tuned with r=32 on all text-stack linear layers and the LoRA weights were merged back into the base bf16 weights. This quantized variant converts the model to float16 for ~20% faster prefill on Apple Silicon (M1/M2/M3/M4), then applies the oQ5E exponent-aware 5-bit quantization scheme from oMLX v0.5.7.

The "grug" thinking style achieves 8–40× fewer tokens than the base model while maintaining strong benchmark performance, with zero repetition loops observed in stress testing.

Architecture (Qwen3 Decoder-Only Transformer)

Parameter	Value
Hidden size	5,120
Num hidden layers	40
Num attention heads	40
Num key-value heads (GQA)	8
Intermediate size	29,568
Max position embeddings	131,072
Layer norm eps	1e-06
RMS norm eps	1e-05
RoPE theta	1,000,000.0
Vocabulary size	152,064

Quantization Details

Property	Value
Quantization format	oQ5E (oMLX 5-bit exponent-aware)
oMLX version	0.5.7
Bits per weight	~5.8 BPW
Base dtype	float16 (fp16)
Original dtype	bfloat16
Exponent aware	Yes
Quality loss vs bf16	1–3% perplexity increase
Memory savings	~62–65% reduction from full precision
Estimated quantized size	~19–21 GB

The oQ5E format uses exponent-aware quantization to preserve the dynamic range of larger-magnitude weights while compressing the bulk of the weight distribution. The fp16 base dtype was chosen over bf16 for ~20% faster prefill throughput on Apple Silicon GPUs.

Training Details

Property	Value
Base model	Qwen/Qwen3.6-27B
Fine-tune method	LoRA (r=32, all text-stack linears)
LoRA merge	Merged into bf16 weights
Training data	grug-think-v3-10k agent trajectories + fresh GPT-5.5 set
Loss function	Think-only loss on trajectory data; full loss on fresh high-quality data

Performance Benchmarks (v2.1)

Benchmark	Metric	Score
HumanEval	pass@1	87.2
MBPP	pass@1	85.0
GSM8K	exact match	95.5
MATH-500	(unseen surface)	68.7
SWE-bench Tool	valid / match / args	97.1 / 92.6 / 100.0

Token efficiency: 8–40× fewer tokens than the base Qwen3.6-27B model across reasoning tasks.

Hardware Requirements

Requirement	Minimum	Recommended
RAM / Unified Memory	16 GB (M1/M2/M3/M4 base)	24 GB+ (M1/M2/M3/M4 Pro/Max)
Disk space	~22 GB	~22 GB

Usage

Python (mlx-lm)

from mlx_lm import load, generate

model, tokenizer = load("your-username/grug-27b-oQ5E-fp16")
response = generate(
    model, tokenizer,
    prompt="Your prompt here",
    max_tokens=1024,
    temp=0.6,
    top_p=0.95
)
print(response)

CLI

mlx_lm.run   --model your-username/grug-27b-oQ5E-fp16   --max-tokens 4096   --temp 0.6   --top-p 0.95

Recommended Inference Parameters

Parameter	Value	Notes
temp	0.6	Balanced creativity / determinism
top_p	0.95	Standard nucleus sampling
max_tokens	4096–16384	Adjust based on task length

License

This quantized model card and accompanying files are released under the Apache License 2.0, inherited from the original ProCreations/grug-27b model. The base architecture is derived from Qwen/Qwen3.6-27B.

See the original model card for full license terms: https://huggingface.co/ProCreations/grug-27b

Citation

@misc{grug27b,
  title={Grug 27B — Token-Efficient Reasoning with the "Grug" Thinking Style},
  author={ProCreations},
  year={2025},
  url={https://huggingface.co/ProCreations/grug-27b}
}

@misc{qwen3,
  title={Qwen3.6-27B Technical Report},
  author={Qwen Team},
  year={2025},
  url={https://huggingface.co/Qwen/Qwen3.6-27B}
}

Acknowledgements

• Qwen Team for the Qwen3.6-27B base architecture
• ProCreations for the original grug-27b model and training methodology
• oMLX team for the exponent-aware oQ5E quantization scheme (v0.5.7)
• mlx-lm for the inference runtime

Links

• Original model: https://huggingface.co/ProCreations/grug-27b
• Base model: https://huggingface.co/Qwen/Qwen3.6-27B
• oMLX quantization: https://github.com/ml-explore/mlx-examples