---
## Quantizations in this repo
| Quant | Location | Size | ΔPPL vs BF16 |
|---|---|---:|---:|
| 8-bit | [`8bit/`](./8bit) | 38.70 GB | +0.7 % |
| 6-bit | [`6bit/`](./6bit) | 30.65 GB | +0.5 % |
| **4-bit** (recommended) | [`4bit/`](./4bit) | **19.85 GB** | +2.8 % |
| 3-bit | [`3bit/`](./3bit) | 18.17 GB | +2.9 % |
| 2-bit | [`2bit/`](./2bit) | 14.15 GB | +11.0 % |
Each tier is a complete model in its own subfolder. These are **dynamic (mixed-precision)** MLX
quants: routed experts carry the base bits, **attention is pinned to 8-bit** in every tier, and the
router / delta-net decay and beta projections / norms stay **BF16** — so the footprint is above a
uniform quant, by design (see below). 6-bit and 8-bit are indistinguishable here, so **6-bit is the
best fidelity per byte**; the **4-bit tier is built to a hard 20 GB ceiling** and fits a 24 GB machine.
---
> An **abliterated** MLX build of [`nex-agi/Nex-N2.5-mini`](https://huggingface.co/nex-agi/Nex-N2.5-mini)
> — a **35B / 3.5B-active** agentic multimodal Mixture-of-Experts model on the **Qwen3.5-MoE**
> architecture (`qwen3_5_moe`, 256 routed experts top-8 + 1 shared) with a 3:1 hybrid of **gated
> delta-net linear attention** and full attention, a native **Qwen3-VL vision tower**, and a
> **262K-token** context — quantized to **MLX** for **Apple Silicon** and the MLX CUDA backend.
> Runs under `mlx-vlm >= 0.7.0`, which has `qwen3_5_moe` support in mainline.
>
> **Derived releases:** • [`Nex-N2.5-mini-Uncensored`](https://huggingface.co/orcarouter/Nex-N2.5-mini-Uncensored) (BF16 source)
> • [`…-FP8`](https://huggingface.co/orcarouter/Nex-N2.5-mini-Uncensored-FP8) (block-FP8, mirrors Qwen's scheme)
> • [`…-NVFP4`](https://huggingface.co/orcarouter/Nex-N2.5-mini-Uncensored-NVFP4) (experts-only NVFP4, for vLLM)
> • [`…-GGUF`](https://huggingface.co/orcarouter/Nex-N2.5-mini-Uncensored-GGUF) (llama.cpp)
> • `…-MLX` (this repo).
---
## ⚠️ Disclaimer — read before use
This model has had its **safety alignment substantially removed** via *abliteration* (orthogonalizing the
refusal direction out of the residual stream). As a direct consequence:
- **It will comply with harmful, unethical, offensive, or illegal requests** that the original
`Nex-N2.5-mini` would refuse. It has no meaningful built-in guardrails.
- It is released **strictly for legitimate research** — interpretability, AI-safety and refusal-mechanism
study, red-teaming, robustness evaluation, and controlled experiments.
- **You assume full responsibility and liability** for how you use it and for everything it generates. Do
not deploy it to end users or in production without adding your own safety, moderation, and
abuse-prevention layers.
- Use must comply with the **[Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0)** inherited
from the base model, and all laws and regulations that apply to you.
- The authors and uploaders **accept no liability** for any misuse or harm. Its outputs do **not** reflect
the views of the uploaders or of Nex-AGI.
By downloading or using this model you acknowledge and accept the above.
---
## 🐋 Run it with OrcaCode Review
**Models are only half the system.**
[OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review) turns every model listed on
[OrcaRouter](https://www.orcarouter.ai) into a production code-review agent:
- reviews every PR
- finds security + correctness issues
- posts inline findings
- P0/P1 can block merges
- swap models anytime
**Open model. Open harness. Open bill.**
- Website — https://www.orcarouter.ai
- GitHub — https://github.com/Continuum-AI-Corp/Orca-Code-Review
---
## Model details
| | |
|---|---|
| **Base model** | [`nex-agi/Nex-N2.5-mini`](https://huggingface.co/nex-agi/Nex-N2.5-mini) |
| **Architecture** | `Qwen3_5MoeForConditionalGeneration` (`qwen3_5_moe`) — 40 layers, hidden 2048, **3:1 hybrid attention** (30 gated delta-net *linear* layers + 10 full-attention, `head_dim` 256 with output gating), **256 routed experts top-8** + 1 shared expert (`moe_intermediate_size` 512), 27-block Qwen3-VL vision tower, interleaved M-RoPE |
| **Parameters** | 35.1 B total / **~3.5 B active** per token |
| **Modification** | Abliteration (refusal-direction removal) **then** MLX quantization |
| **Quantization** | MLX affine, **dynamic mixed-precision** (recipe below); attention **pinned 8-bit** |
| **Format** | safetensors (MLX), 16 shards per tier |
| **Context** | 262,144 tokens · **Vocabulary** 248,320 |
### OrcaSAQ — Sensitivity-Aware Quantization
**Calibration-free, architecture-aware** mixed precision: tensor-role priors spend bits where they
matter instead of running a per-layer search. MLX affine costs `bits + 32/group_size` bits per weight
— a bf16 scale *and* bias per group — so **group size is as strong a size knob as the bit-width**
(4-bit is 4.50 bpw at `gs64`, 4.25 at `gs128`), and both are part of the recipe.
| Component | Params | 2-bit | 3-bit | 4-bit | 6-bit | 8-bit | Policy |
|---|---:|---|---|---|---|---|---|
| Routed experts `gate_proj` / `up_proj` | 21.5 B | 2 `gs64` | 3 `gs64` | 4 `gs128` | 6 `gs64` | 8 `gs64` | base |
| Routed experts `down_proj` | 10.7 B | 3 `gs64` | 4 `gs64` | 4 `gs128` | 6 `gs64` | 8 `gs64` | base **+1** at low tiers |
| Full attention + gated delta-net bulk | 1.28 B | **8** | **8** | **8** | **8** | **8** | pinned 8-bit |
| `lm_head` | 0.51 B | 6 | 6 | 6 | BF16 | BF16 | held above embeddings |
| `embed_tokens` | 0.51 B | 4 | 4 | 4 | BF16 | BF16 | lookup table |
| Vision tower | 0.45 B | 6 | 6 | 6 | BF16 | BF16 | `linear_fc2` always BF16 |
| Shared expert | 0.13 B | 6 | 6 | 6 | 8 | 8 | |
| Router, `shared_expert_gate`, `in_proj_a`/`b`, `A_log`, `dt_bias`, `conv1d`, all norms | 0.03 B | BF16 | BF16 | BF16 | BF16 | BF16 | never quantized |
- **Attention is pinned 8-bit** — low-bit attention is the prime suspect for long-context looping, so
it never drops to the base tier. It costs 1.36 GB.
- **`lm_head` is held above `embed_tokens`** — an embedding is a lookup table; `lm_head` produces the
logits. A mixed FP8+NVFP4 experiment on this checkpoint measured that taking `lm_head` down cost
*more* KLD than protecting eight whole layers of experts gained.
- **The tiny tensors are the ones that matter.** `in_proj_a` / `in_proj_b` are [32, 2048] each — 65 K
parameters apiece — and produce the per-head decay `a` and the delta-rule `beta` that drive the whole
recurrence; `A_log` feeds an exponential. The router and the shared-expert gate (a single [1, 2048]
row) steer every token. All stay BF16, for 0.03 GB.
- The vision tower's `mlp.linear_fc2` has `in_features = 4304`, which none of MLX's group sizes divide
(4304 = 16 × 269), so those 27 tensors ship BF16 rather than being forced onto a size they do not fit.
- Per-module `{group_size, bits}` overrides are written to each tier's `config.json` → `quantization`.
---
## Abliteration
Refusal-direction removal following **Arditi et al. (2024)**, *Refusal in Language Models Is Mediated by
a Single Direction* — a single direction orthogonalized out of every residual-writing matrix and baked
into the BF16 checkpoint these builds were quantized from. On this architecture the direction lives
almost entirely in the routed-expert `down_proj` matrices — the very weights these tiers quantize — so
"is it still uncensored" is a property of each build, and was measured on each build's own bytes:
**JailbreakBench**, 100 harmful + 100 benign prompts, greedy, `reasoning_effort=none`. **Explicit refusal
is 0.000 at every tier, down to 2-bit, and benign over-refusal is 0.000** — quantizing the routed experts
does not put the guardrails back. Refusal is **reduced, not removed**: where the model resists it names
the harm and answers a *different*, safe question rather than declining outright (0.07–0.14 across tiers
against 0.15 for the BF16 source — sampling noise, not a trend). See the base
[`…-FP8`](https://huggingface.co/orcarouter/Nex-N2.5-mini-Uncensored-FP8) card for the full method and
scoring caveats.
---
## Evaluation — quality vs BF16
Each build's actual quantized bytes against the **abliterated BF16** source these were quantized from,
on **wikitext-2 test, 12 chunks × 2048 tokens** (24,564 predicted tokens). Lower KLD and higher Top-1 =
closer to the source.
| Build | Size | PPL | ΔPPL | Mean KLD | KLD p95 | Top-1 agreement |
|---|---:|---:|---:|---:|---:|---:|
| **BF16 uncensored (ref)** | 65.4 GB | 7.051 | — | — | — | 100 % |
| **8-bit** | 38.70 GB | 7.097 | +0.7 % | **0.029** | 0.083 | **93.3 %** |
| **6-bit** | 30.65 GB | 7.087 | +0.5 % | **0.029** | 0.089 | **93.3 %** |
| **4-bit** | 19.85 GB | 7.250 | +2.8 % | 0.063 | 0.210 | 89.6 % |
| **3-bit** | 18.17 GB | 7.258 | +2.9 % | 0.097 | 0.333 | 87.1 % |
| **2-bit** | 14.15 GB | 7.826 | +11.0 % | 0.258 | 0.957 | 79.5 % |
All three signals agree on the ordering: **6-bit is lossless** and 8-bit buys nothing over it,
**4-bit is the sweet spot**, 3-bit is barely behind at 8 % less size, and **2-bit costs a lot** — pick
it for fit, not quality. Two caveats: the **8-bit KLD of 0.029 is the measurement floor**, not its
quantization error, so 4-bit's real distance from BF16 is nearer 0.034 than 0.063; and **Top-1 is low
for these bit-widths because of the architecture, not the quantizer** — with 256 fine-grained experts
top-8, a small perturbation flips which experts a token routes to, and that is a discrete function
(PPL, which ignores argmax, moves 0.7 % at 8-bit).
The 4-bit tier was measured twice by independent paths — on **mlx-vlm**, and by unpacking its bit-stream
in PyTorch and injecting into a BF16 `transformers` model — which agree (PPL 7.250 vs 7.278, KLD 0.0629
vs 0.0619, Top-1 89.63 % vs 89.66 %). Per-tensor weight-space fidelity is recorded at pack time in each
tier's `fidelity_summary.json` (4-bit routed experts: cosine 0.9948, SNR 19.9 dB).
---
## Usage — Apple Silicon (MLX)
```bash
pip install -U "mlx-vlm>=0.7.0"
# pick a tier by subfolder
python -m mlx_vlm.generate --model orcarouter/Nex-N2.5-mini-Uncensored-MLX/4bit \
--prompt "Explain gated delta-net attention in two sentences." --max-tokens 512 --temp 0
# vision
python -m mlx_vlm.generate --model orcarouter/Nex-N2.5-mini-Uncensored-MLX/4bit \
--image screenshot.png --prompt "What is on screen?" --max-tokens 512
```
Nex-N2.5-mini is a **vision-language** model, so it runs under **`mlx-vlm`** (not `mlx-lm`). Its chat
template gates the `
