hermitdave/Agnes-3.0-Flash-MLX-8bit-MTP

Agnes-3.0-Flash — MLX 8-bit + MTP

8-bit MLX quantization of Agnes-AI/Agnes-3.0-Flash (Apache-2.0), grafted with Qwen3.8's vision tower and Agnes's native MTP head for enhanced inference.

• Affine 8-bit, group size 64 — 8.50 bits/weight, ~37.8GB (8 shards)
• 262,144-token context, thinking on/off via the original chat template
• MTP head grafted: Agnes native MTP head with FFN padded 17408→19456 (zero-pad, mathematically identical), quantized to 4-bit
• Vision tower grafted from Qwen3.8 (333 bf16 tensors, byte-identical to Agnes original)

Architecture

Agnes-3.0-Flash is a 33B hybrid attention model:

• 54 recurrent (Gated DeltaNet) layers + 18 global attention layers
• GQA 6:1 (24q/4kv heads, dim 256), 3-axis RoPE, causal conv kernel 4
• Hidden size 5120, intermediate size 17408 (Agnes native)
• MTP head intermediate size 19456 (Qwen3.8 compatible, padded from 17408)

What changed

Converted from the original Agnes format to standard Qwen3.5 architecture:

• Folded parallel FFN into main MLP via concatenation (intermediate_size: 19456)
• Renamed delta_attn → linear_attn, global_attn → self_attn
• Converted one-centered RMSNorm to standard format
• Cast bf16 → fp16 for serialization compatibility

MTP graft:

• Padded Agnes native MTP head FFN layers (gate_proj, up_proj: +2048 rows; down_proj: +2048 cols) to match Qwen3.8's expected 19456
• Quantized MTP weights to 4-bit affine (group_size=64)
• Grafted 333 vision tensors from Qwen3.8 (byte-identical to Agnes original)

Benchmark

Measured on M3 Max 64GB, oMLX, temperature 0:

Metric	Baseline (no MTP)	With MTP	Speedup
Throughput	15.0 tok/s	21.7 tok/s	1.44×
Acceptance rate	—	85.7%	—
Tokens per cycle	—	2.46	—

Usage

pip install mlx-lm
mlx_lm.generate --model hermitdave/Agnes-3.0-Flash-MLX-8bit-MTP --prompt "Hello" --max-tokens 200

Or with oMLX: place under ~/.omlx/models/hermitdave/ and enable mtp_enabled + vlm_mtp_enabled in model settings.

Attribution

This conversion was produced by Hermes Agent (Nous Research). MTP graft methodology: pad Agnes native MTP head FFN to Qwen3.8 size (19456), quantize to 4-bit, graft Qwen3.8 vision tower.