Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: mlx
|
| 4 |
+
tags:
|
| 5 |
+
- moe
|
| 6 |
+
- edge-inference
|
| 7 |
+
- prerouter
|
| 8 |
+
- lora
|
| 9 |
+
- ssd-offload
|
| 10 |
+
base_model:
|
| 11 |
+
- inclusionAI/Ling-3.0-tiny-base
|
| 12 |
+
pipeline_tag: text-generation
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
<div align="center">
|
| 16 |
+
|
| 17 |
+
<img src="20260908-223115.jpg" alt="edge0" width="100%">
|
| 18 |
+
|
| 19 |
+
# Edge0-8b-a1b Preview
|
| 20 |
+
|
| 21 |
+
**An 8B-class sparse MoE that runs on a phone — 1.0 GiB of active memory, experts streamed from SSD.**
|
| 22 |
+
|
| 23 |
+
[](https://github.com/Edge0-AI/edge0)
|
| 24 |
+
[](https://huggingface.co/Edge0/Edge0-35b-a3b-preview)
|
| 25 |
+
[](https://huggingface.co/Edge0/Edge0-8b-a1b-preview)
|
| 26 |
+
[](https://github.com/Edge0-AI/edge0/blob/main/LICENSE)
|
| 27 |
+
|
| 28 |
+
</div>
|
| 29 |
+
|
| 30 |
+
This repository hosts the **edge0-8b** checkpoint of the
|
| 31 |
+
[edge0](https://github.com/Edge0-AI/edge0) streaming MoE inference
|
| 32 |
+
framework. The full 4-bit checkpoint (≈4.2 GB) stays on storage; edge0
|
| 33 |
+
mmaps it and streams MoE experts from SSD on demand, with a trained
|
| 34 |
+
**prerouter** head that predicts the next token's expert routing one
|
| 35 |
+
step ahead so expert loads hide completely behind the forward pass. The
|
| 36 |
+
result: **an 8B-class MoE with ≈1.0 GiB of active memory** — a
|
| 37 |
+
phone-class memory budget, with no upfront weight download into RAM and
|
| 38 |
+
no model sharding.
|
| 39 |
+
|
| 40 |
+
> **Preview status:** this is an early preview release of the edge0
|
| 41 |
+
> pipeline. The checkpoint ships as int4 quantization plus LoRA and
|
| 42 |
+
> prerouter adapters trained for this framework.
|
| 43 |
+
|
| 44 |
+
## Model summary
|
| 45 |
+
|
| 46 |
+
| | |
|
| 47 |
+
|---|---|
|
| 48 |
+
| Base model | inclusionAI Ling 3.0 tiny (bailing hybrid, MLA + MoE, ≈7.9B total / ≈1.2B active) |
|
| 49 |
+
| Quantization | 4-bit |
|
| 50 |
+
| Layers | 24 |
|
| 51 |
+
| Experts / active per token | 128 / 8 (K=8) |
|
| 52 |
+
| Framework | [edge0](https://github.com/Edge0-AI/edge0) (MLX backend) |
|
| 53 |
+
| Contents | base checkpoint + `lora_edge0_8b.safetensors` + `prerouter_edge0_8b.safetensors` |
|
| 54 |
+
|
| 55 |
+
The LoRA and prerouter adapters are co-located with the base checkpoint
|
| 56 |
+
and load automatically — this repository is a complete, ready-to-run
|
| 57 |
+
model directory for `edge0`.
|
| 58 |
+
|
| 59 |
+
## Quality (self-evaluation)
|
| 60 |
+
|
| 61 |
+
Internal self-evaluation of this checkpoint (int4 + adapters) relative to
|
| 62 |
+
the fp16 base model — the loss of the edge0 pipeline is small: **2.8
|
| 63 |
+
points on average**, with MMLU-Pro above the base (max 100, all
|
| 64 |
+
self-run):
|
| 65 |
+
|
| 66 |
+
| Benchmark | edge0-8b (int4) | Base fp16 |
|
| 67 |
+
|---|---:|---:|
|
| 68 |
+
| AIME 2026 | 63.3 | 73.3 |
|
| 69 |
+
| HumanEval | 91.5 | 92.7 |
|
| 70 |
+
| GPQA-Diamond | 70.7 | 71.2 |
|
| 71 |
+
| MMLU-Pro | 70.1 | 65.8 |
|
| 72 |
+
| IFBench | 53.9 | 60.6 |
|
| 73 |
+
| **Average** | **69.9** | **72.7** |
|
| 74 |
+
|
| 75 |
+
## Performance
|
| 76 |
+
|
| 77 |
+
Measured with `examples/bench.py` on a Mac mini M4 Pro, 24 GB:
|
| 78 |
+
|
| 79 |
+
| Decode speed | Prefill throughput (cold / warm) | Peak active memory* |
|
| 80 |
+
|---|---|---|
|
| 81 |
+
| 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB |
|
| 82 |
+
|
| 83 |
+
*Short contexts; long contexts add KV cache (≈3.3 GiB at 3.3k tokens).
|
| 84 |
+
Expert weights stream from SSD via mmap and are not resident.
|
| 85 |
+
|
| 86 |
+
## Quick start
|
| 87 |
+
|
| 88 |
+
```bash
|
| 89 |
+
pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'
|
| 90 |
+
|
| 91 |
+
# Download this repository into a local directory
|
| 92 |
+
huggingface-cli download Edge0/Edge0-8b-a1b-preview --local-dir ./Edge0-8b-a1b-preview
|
| 93 |
+
|
| 94 |
+
# Run it
|
| 95 |
+
export EDGE0_8B_MODEL=$PWD/Edge0-8b-a1b-preview
|
| 96 |
+
edge0 chat --name edge0-8b --prompt "Introduce yourself"
|
| 97 |
+
|
| 98 |
+
# Or serve an OpenAI-compatible HTTP API
|
| 99 |
+
edge0 serve --name edge0-8b --port 8083
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
For full usage (Python API, streaming options, prerouter details), see the
|
| 103 |
+
[edge0 documentation](https://github.com/Edge0-AI/edge0#documentation).
|
| 104 |
+
|
| 105 |
+
## License
|
| 106 |
+
|
| 107 |
+
Apache 2.0. See [LICENSE](https://github.com/Edge0-AI/edge0/blob/main/LICENSE).
|