README.md

---
license: apache-2.0
library_name: mlx
tags:
- moe
- edge-inference
- prerouter
- lora
- ssd-offload
base_model:
- Qwen/Qwen3.5-MoE-35B-A3B
pipeline_tag: text-generation
---

<div align="center">

<img src="20260908-223115.jpg" alt="edge0" width="100%">

# Edge0-35b-a3b Preview

**A 35B-class sparse MoE that runs on a phone — 2.9 GiB of active memory, experts streamed from SSD.**

[![GitHub](https://img.shields.io/badge/GitHub-Edge0--AI%2Fedge0-black?style=for-the-badge&logo=github)](https://github.com/Edge0-AI/edge0)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Edge0--35b--a3b--preview-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Edge0-35b-a3b-preview)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Edge0--8b--a1b--preview-yellow?style=for-the-badge)](https://huggingface.co/Edge0/Edge0-8b-a1b-preview)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge)](https://github.com/Edge0-AI/edge0/blob/main/LICENSE)

</div>


**Edge0-35b-a3b** — an 35B MoE LLM that runs at viable speed on portable devices in under **2.9 GiB of active memory** (1/8 of its 23 GB weight footprint), 
via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework.

The key is streaming: experts are memory-mapped and fetched from SSD only as routed, 
so RAM holds just the active weights. What makes that viable — instead of stalling like plain parameter offloading — 
is a trained prerouter head that **predicts the next token's expert routing one step ahead**, hiding storage latency behind compute.

> **Preview status:** this is an early preview release of the edge0
> pipeline. The checkpoint ships as int4 quantization plus LoRA and
> prerouter adapters trained for this framework.

## Model summary

| | |
|---|---|
| Base model | Qwen3.5-MoE 35B-A3B |
| Quantization | 4-bit |
| Layers | 40 |
| Experts / active per token | 256 / 4 (K=4) |
| Framework | [edge0](https://github.com/Edge0-AI/edge0) (MLX backend) |
| Contents | base checkpoint + `lora_edge0_35b.safetensors` + `prerouter_edge0_35b.safetensors` |

The LoRA and prerouter adapters are co-located with the base checkpoint
and load automatically — this repository is a complete, ready-to-run
model directory for `edge0`.

## Quality (self-evaluation)

Internal self-evaluation of this checkpoint (int4 + adapters) relative to
the fp16 base model — the loss of the edge0 pipeline is small: **3.9
points on average** (max 100, all self-run):

| Benchmark | edge0-35b (int4) | Qwen3.5-MoE 35B-A3B (fp16) |
|---|---:|---:|
| AIME 2026 | 86.6 | 92.7 |
| HumanEval | 90.9 | 95.1 |
| GPQA-Diamond | 79.8 | 81.8 |
| MMLU-Pro | 81.0 | 84.6 |
| IFBench | 57.9 | 61.7 |
| **Average** | **79.2** | **83.2** |

## Performance

Measured with `examples/bench.py` on a Mac mini M4 Pro, 24 GB:

| Decode speed | Prefill throughput (cold / warm) | Peak active memory* |
|---|---|---|
| 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB |

*Short contexts; long contexts add KV cache. Expert weights stream from
SSD via mmap and are not resident.

## Quick start

```bash
pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'

# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview

# Run it
export EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview
edge0 chat --name edge0-35b --prompt "Introduce yourself"

# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-35b --port 8085
```

For full usage (Python API, streaming options, prerouter details), see the
[edge0 documentation](https://github.com/Edge0-AI/edge0#documentation).

## License

Apache 2.0. See [LICENSE](https://github.com/Edge0-AI/edge0/blob/main/LICENSE).