mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX

Ornith-1.5-35B-A3B — REAP-192 · mxfp4 · MLX (tuned for 24 GB Apple Silicon)

A compressed derivative of ornith-ai/Ornith-1.5-35B-A3B built to run fast, swap-free, single-user inference on a base M4 Mac mini with 24 GB of unified memory — a machine on which the plain 4-bit conversion (19.5 GB) does not run at all (hard Metal out-of-memory against the ~18 GB working-set cap).

This repo is the recommended quality-preserving build: 49.8 tok/s decode with task quality measurably above an uncompressed-expert 3-bit reference. A max-speed variant reaching 51.2 tok/s (at lower quality) is documented below and reproducible with the included scripts.

	Decode	Prefill (8k ctx)	Warm TTFT	Task quality*	NLL	Weights
This build (recommended)	49.8 tok/s	451 tok/s	0.31 s	0.9814	0.8950	14.2 GB
Max-speed variant (recipe below)	51.2 tok/s	448 tok/s	0.31 s	0.9314	0.9039	14.1 GB
Reference anchor (unpruned, 3-bit experts)	36.7 tok/s	410 tok/s	0.43 s	0.9543	0.8977	16.1 GB

*Composite pass-rate over a 42-task executable suite (unit-tested coding, exact-answer math, JSON tool calls, schema-validated structured output, format compliance, stability checks) plus rotated held-out task generations; NLL measured on a fixed mixed code/prose corpus. All numbers measured on a base M4 Mac mini (24 GB, macOS 26.2) serving over an OpenAI-compatible HTTP endpoint, median of repeated runs, greedy decoding. An independent fresh-context audit reproduced the recommended line bit-exactly on quality/NLL and within 0.2 % on decode.

What exactly was done (lineage)

Source: ornith-ai/Ornith-1.5-35B-A3B-MLX @ revision 1713baf2e38984a3c557195caf167759144d1d7e (bf16, 69.3 GB). The Ornith team's card notes Ornith-1.5 was developed on top of Qwen3.5 and Gemma4 lineages; see the original model card for training details and benchmark results of the uncompressed model.

Transformations, in order (each reproducible with the scripts in this repo):

1. REAP expert pruning, 256 → 192 experts per layer (25 %), following REAP: Router-weighted Expert Activation Pruning (Lasby et al., Cerebras, 2025): saliency = mean over routed tokens of (router gate probability × ‖expert output‖₂); the 64 lowest-saliency experts per layer are removed and the router renormalizes over survivors. Saliency was collected on-device over ~295k mixed-domain calibration tokens (code, prose, math, chat, tool-call JSON). The ranking is robust: recollection on an independently seeded calibration mix changed the survivor set by <1 % (top-192 Jaccard 0.992). Shared experts are never pruned.
2. Quantization (mxfp4 mode): routed experts, attention, and embeddings in mxfp4 (group 32); shared expert in 8-bit affine (it runs on every token — this is what lifts quality above cheaper builds); router gates in 8-bit affine. Effective ~4.33 bits/weight.
3. Top-k reduced 8 → 7 (num_experts_per_tok: 7 in config.json): −12.5 % active bytes per token for +0.55 % NLL. k=6 was measured and rejected (three variants, all fail quality gates).

No fine-tuning, no weight editing beyond the above; the tokenizer and chat template are unchanged from the source.

Use it

Requires an Apple-silicon Mac (≈16 GB free memory; built for 24 GB machines) and mlx-lm ≥ 0.31:

pip install -U mlx-lm

# generate
mlx_lm.generate --model mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX \
  --prompt "Write a Python function that merges overlapping intervals." --max-tokens 400

# OpenAI-compatible server (stock)
mlx_lm.server --model mchen04/Ornith-1.5-35B-A3B-REAP192-mxfp4-MLX --port 8080

Optional +8–10 % serving throughput: the stock server spends ~2.4 ms/token in its batching/parsing layer. scripts/minimal_server.py is a drop-in single-user replacement (same OpenAI-compatible API, adaptive prompt-prefix caching, reasoning/content split) that serves this model at the bare-API decode rate:

python scripts/build_minimal_server.py   # builds a patched runtime copy next to your mlx-lm
PYTHONPATH=./patched-minsrv python -m mlx_lm server --model <this-model> --port 8080

Max-speed variant (51.2 tok/s)

Quantize everything (including the shared expert) in mxfp4 instead of step 2 above — recipe mxfp4 in scripts/quantize_variant.py, then the same pruning and k=7 config. Costs ~5 task-quality points (0.9814 → 0.9314) for +1.4 tok/s; measured details in the table. We recommend this build instead unless raw decode is all that matters. (A middle build, 8-bit shared + 3-bit head, lands at 50.8 tok/s / 0.9564 — recipe mxfp4-sh8h3.)

Measured behavior on the target machine

• Peak memory ≈ 14.3 GB; zero sustained swap; macOS memory pressure stays normal; no thermal derating over sustained generation (decode-rate decay 0.99 over 1024 tokens).
• Prompt-prefix reuse (e.g. stable system prompts) cuts TTFT ~25× under the included server; cold start (process spawn → first token) ≈ 8 s.
• Context: KV cache is tiny for this hybrid architecture (10 of 40 layers are full attention; ~20.5 kB/token) — long contexts are memory-cheap; 8k-token prompts prefill at ~450 tok/s.

Limitations

• Compression is not free. The recommended build's task-suite composite (0.9814) and NLL (0.8950) sit at-or-above the measured uncompressed-expert reference on this suite, but the suite is finite (coding/math/tool/structured/format/stability, English); expect some degradation vs the bf16 original on knowledge-heavy or multiple-choice tasks — REAP's authors measured exactly that pattern at 25 % pruning. The bf16 original could not be run on any available hardware for direct comparison.
• k=7 routing and expert pruning shift the output distribution; borderline greedy tokens can differ from the source model's.
• Tuned for single-user serving; the included minimal server is deliberately not a multi-user production server.
• Measured on macOS 26.2 / mlx 0.32.2 / mlx-lm 0.31.3; other versions may differ.

Safety

This derivative changes efficiency characteristics only: no fine-tuning was performed and no safety-relevant components were intentionally altered. It inherits the behavior, alignment properties, and content policies (and their gaps) of Ornith-1.5-35B-A3B — refer to the original card. Compression can subtly shift refusal boundaries and factual reliability; the quality gate used here measured task competence, not safety behavior. Apply your own safeguards for user-facing deployments, and do not treat outputs as authoritative without verification.

Reproduce / verify

scripts/ contains the exact tooling: quantize_variant.py (named recipes, pinned source revision), reap_collect.py + reap_prune.py (saliency + pruning; prune-of-quantized is exact because per-expert quantization groups are independent), build_minimal_server.py / minimal_server.py (serving runtime), and DERIVATION.json (machine-readable provenance for these exact weights). Deriving from scratch needs the 69.3 GB source checkpoint and any Apple-silicon Mac with ~90 GB free disk (conversion is shard-streamed; 16 GB RAM suffices).

Attribution & citations

• Base model: Ornith-1.5-35B-A3B by the Ornith Team (MIT) — please cite their release: @misc{ornith_1_5, title={{Ornith-1.5}: From Self-Scaffolding to Self-Improvement}, url={https://ornith.ai/ornith_1_5.html}, author={{Ornith Team}}, year={2026}}
• Pruning method: REAP the Experts (Lasby et al., 2025); independent reimplementation in scripts/.
• Runtime: MLX and mlx-lm (Apple, MIT).
• Compression, evaluation, and serving work by mchen04, 2026.

License: MIT, inherited from the base model's declared license; see LICENSE.