grug-27b-oQ6E-fp16

grug-27b-oQ6E-fp16 is a highly token-efficient reasoning model built on top of ProCreations/grug-27b, quantized with oMLX oQ6E (6-bit exponent-aware) using oMLX version 0.5.7, with a float16 (fp16) base dtype optimized for Apple Silicon performance.

Model Description

grug-27b-oQ6E-fp16 is a quantized variant of the grug-27b model, which itself is a LoRA fine-tune of Qwen/Qwen3.6-27B. The model employs the distinctive "grug" thinking style — a token-efficient reasoning approach that dramatically reduces token consumption while maintaining strong benchmark performance.

Key Features

• Token-efficient reasoning: Up to 198x fewer tokens than the base model on reasoning tasks
• oQ6E quantization: 6-bit exponent-aware quantization via oMLX for minimal quality loss
• fp16 base dtype: Float16 base dtype provides ~20% faster prefill on M1/M2 Apple Silicon
• LoRA fine-tuned: r=32 fine-tune on all text-stack linear layers, merged to bf16
• Zero repetition loops: Stress-tested with no repetition loop failures

Architecture

Parameter	Value
Architecture	Qwen3 decoder-only transformer
Model Type	qwen3
Hidden Size	5,120
Num Hidden Layers	40
Num Attention Heads	40
Num Key-Value Heads (GQA)	8
Intermediate Size	29,568
Max Position Embeddings	131,072
Layer Norm Eps	1e-6
RMS Norm Eps	1e-5
RoPE Theta	1,000,000.0
Vocabulary Size	152,064

Quantization Details

This model is quantized using the oMLX library (version 0.5.7) with the oQ6E format — a 6-bit exponent-aware quantization scheme.

Property	Value
Quantization Format	oQ6E (oMLX 6-bit exponent-aware)
oMLX Version	0.5.7
Bits Per Weight	~6.8 BPW
Exponent Aware	Yes
Quality Loss vs bf16	1–1.5% perplexity increase
Memory Savings	~55–60% reduction from full precision
Estimated Quantized Size	~22–24 GB
Base Dtype	float16 (fp16)
Original Dtype	bfloat16 (bf16)

Note: The fp16 base dtype is selected for ~20% faster prefill on M1/M2/M3/M4 Apple Silicon. The output model name appends -fp16 to distinguish it from the bf16 variant.

Training Details

Fine-tuning

• Base Model: Qwen/Qwen3.6-27B
• LoRA Rank (r): 32
• Target Layers: All text-stack linear layers
• Merge Strategy: LoRA weights merged into bf16 base model

Training Data

• grug-think-v3-10k — agent trajectories (think-only loss)
• Fresh gpt-5.5 dataset — high-quality general data (full loss)

Loss Strategy

• Trajectory data: Think-only loss on reasoning trajectories
• Fresh data: Full token-level loss on high-quality general data

Performance Benchmarks

Benchmarks reported under v2.1 evaluation:

Benchmark	Metric	Score
HumanEval	pass@1	87.2
MBPP	pass@1	85.0
GSM8K	exact match	95.5
MATH-500	(unseen surface)	68.7
SWE-bench	valid / match / args	97.1 / 92.6 / 100.0

Token Efficiency

Metric	Value
Token savings vs base model	8–40x fewer tokens
Max compression on reasoning	Up to 198x fewer tokens

Hardware Requirements

Requirement	Minimum	Recommended
Unified Memory (Apple Silicon)	16 GB	32 GB+
Chip Generation	M1/M2/M3/M4	M1/M2/M3/M4 Pro/Max
Disk Space	~25 GB	~25 GB

Note: This model is designed for Apple Silicon (MLX) deployment. The fp16 dtype provides optimal performance on M1/M2 series chips; Pro/Max variants benefit from larger memory for longer contexts.

Usage

Python (MLX-LM)

from mlx_lm import load, generate

model, tokenizer = load("your-username/grug-27b-oQ6E-fp16")
response = generate(
    model, tokenizer,
    prompt="Your prompt here",
    max_tokens=1024,
    temp=0.6,
    top_p=0.95
)
print(response)

CLI

mlx_lm.run   --model your-username/grug-27b-oQ6E-fp16   --max-tokens 4096   --temp 0.6   --top-p 0.95

Recommended Inference Parameters

Parameter	Value	Notes
temp	0.6	Balanced creativity vs determinism
top_p	0.95	Standard nucleus sampling
max_tokens	1024–4096	Adjust based on use case

License

This model is derived from ProCreations/grug-27b, which is licensed under the Apache License 2.0.

The quantization and fp16 conversion are performed using oMLX (version 0.5.7).

See the LICENSE file for full terms.

Citation

@misc{grug27b,
  title={grug-27b},
  author={ProCreations},
  year={2025},
  url={https://huggingface.co/ProCreations/grug-27b}
}

@misc{qwen3,
  title={Qwen3.6-27B},
  author={Qwen Team},
  year={2025},
  url={https://huggingface.co/Qwen/Qwen3.6-27B}
}

Acknowledgements

• Qwen Team for the Qwen3.6-27B base model
• ProCreations for the grug-27b fine-tune
• oMLX for the oQ6E quantization format
• MLX for the Apple Silicon inference framework