leonsarmiento/Huihui-gemma-4-26B-A4B-it-6bit-XL-mlx

This model was converted to MLX format from huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated using BaseQuant_XL 6/8-bit mixed quantization optimized for Apple Silicon. The vision encoder is preserved (full multimodal).

BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying aggressive quantization to the bulk parameters. Note: tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).

This is the abliterated (uncensored) version of Gemma-4-26B-A4B, a 25.2B-parameter MoE (Mixture of Experts) model with 128 experts per layer (avg. 3.8B active parameters per token) and a vision encoder. The unquantized BF16 base was used as the source for maximum fidelity.

Note: This is a re-quantization from the correct non-QAT source (huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated). The previous version of this model (Huihui-gemma-4-26B-A4B-it-qat-6bit-XL-mlx) was mistakenly quantized from the QAT variant and has been deprecated.

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

Use with mlx

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/Huihui-gemma-4-26B-A4B-it-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"

BaseQuant_XL Quantization Strategy

Bit Depth	Layers	Rationale
bf16 (unquantized)	router.proj (gate), mlp (shared/dense expert), vision tower	Routing decisions and the dense expert that processes every token — errors here cascade through all tokens
8-bit	embed_tokens, self_attn	Every-token layers with moderate sensitivity — 8-bit is near-lossless
6-bit	experts.switch_glu (routed experts)	Bulk of parameters — only a subset of experts active per token

Quantization Details

Layer	Bits	Group Size
router.proj (gate)	bf16	—
mlp (shared/dense expert)	bf16	—
Vision tower (full)	bf16	—
embed_tokens	8	64
self_attn	8	64
experts.switch_glu (routed experts)	6	64
Default fallback	8	64

• Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
• Bits per weight: ~7.1
• Total size: ~21 GB (5 shards)
• Group size: 64
• Method: Custom quant_predicate via mlx_vlm
• Note: tie_word_embeddings=True — no separate lm_head

Recommended Inference Parameters

Parameter	Value
temperature	1.0
top_p	1.0
top_k	64
min_p	0.01
repeat_penalty	1.05

Reasoning Parsing in LM Studio and Others

• Start string: <|channel|>thought
• End string: <|channel|>

Chat Template

The chat template is synced to the canonical Google upstream version from google/gemma-4-26B-A4B-it/chat_template.jinja. Do not rely on the HF cache copy — it may be stale.

Key markers in the latest template: preserve_thinking, continues_into_next, prev_non_tool_role, null handling in format_argument.

The template is injected into both chat_template.jinja (standalone file) and tokenizer_config.json (chat_template JSON field).