wanglamao commited on
Commit
a2bf13d
·
verified ·
1 Parent(s): 8c5b623

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -9
README.md CHANGED
@@ -27,15 +27,13 @@ pipeline_tag: text-generation
27
 
28
  </div>
29
 
30
- This repository hosts the **edge0-35b** checkpoint of the
31
- [edge0](https://github.com/Edge0-AI/edge0) streaming MoE inference
32
- framework. The full 4-bit checkpoint (≈23 GB) stays on storage; edge0
33
- mmaps it and streams MoE experts from SSD on demand, with a trained
34
- **prerouter** head that predicts the next token's expert routing one
35
- step ahead so expert loads hide completely behind the forward pass. The
36
- result: **a 35B-A3B MoE with ≈2.9 GiB of active memory** small enough
37
- for phone-class memory budgets, with no upfront weight download into
38
- RAM and no model sharding.
39
 
40
  > **Preview status:** this is an early preview release of the edge0
41
  > pipeline. The checkpoint ships as int4 quantization plus LoRA and
 
27
 
28
  </div>
29
 
30
+
31
+ **Edge0-35b-a3b** an 35B MoE LLM that runs at viable speed on portable devices in under **2.9 GiB of active memory** (1/8 of its 23 GB weight footprint),
32
+ via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework.
33
+
34
+ The key is streaming: experts are memory-mapped and fetched from SSD only as routed,
35
+ so RAM holds just the active weights. What makes that viable — instead of stalling like plain parameter offloading —
36
+ is a trained prerouter head that **predicts the next token's expert routing one step ahead**, hiding storage latency behind compute.
 
 
37
 
38
  > **Preview status:** this is an early preview release of the edge0
39
  > pipeline. The checkpoint ships as int4 quantization plus LoRA and