bupalinyu commited on
Commit
b42e70c
·
verified ·
1 Parent(s): a1a7d92

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -30,7 +30,7 @@ pipeline_tag: text-generation
30
  </div>
31
 
32
 
33
- **Edge0-35b-a3b** — a 35B MoE LLM that runs at viable speed on portable devices in under **2.9 GiB of active memory** (1/8 of its 23 GB weight footprint),
34
  via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework.
35
 
36
  > **Preview status:** this is an early preview release of the edge0
@@ -41,10 +41,10 @@ via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework
41
 
42
  - **Runs in phone-class memory**: the full 4-bit checkpoint stays on
43
  storage and experts are streamed on demand, so only the active
44
- weights are in RAM — under **2.9 GiB**, with no sharding and no
45
  upfront download of the weights into memory.
46
- - **Fast enough for interactive use**: 14.9–17.7 tok/s decode on a
47
- Mac mini M4 Pro; long prompts fill in at 113–140 tok/s.
48
  - **Quality kept after quantization**: Recover-LoRA distillation keeps
49
  the int4 model within **3.9 points** of its fp16 base.
50
  - **Works out of the box**: base, LoRA and prerouter adapters ship
@@ -126,7 +126,7 @@ SSD on demand and are not resident.
126
  - The MLX backend currently targets Apple Silicon; other backends are
127
  on the edge0 roadmap.
128
  - Long contexts grow the KV cache; use shorter contexts to keep peak
129
- memory at 2.9 GiB.
130
 
131
  ## Quick start
132
 
 
30
  </div>
31
 
32
 
33
+ **Edge0-35b-a3b** — a 35B MoE LLM that runs at viable speed on a phone in under **3 GiB of active memory** (1/8 of its 23 GB weight footprint),
34
  via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework.
35
 
36
  > **Preview status:** this is an early preview release of the edge0
 
41
 
42
  - **Runs in phone-class memory**: the full 4-bit checkpoint stays on
43
  storage and experts are streamed on demand, so only the active
44
+ weights are in RAM — under **3 GiB**, with no sharding and no
45
  upfront download of the weights into memory.
46
+ - **Fast enough for interactive use**: 15 tok/s decode on a phone;
47
+ long prompts fill in at 140 tok/s.
48
  - **Quality kept after quantization**: Recover-LoRA distillation keeps
49
  the int4 model within **3.9 points** of its fp16 base.
50
  - **Works out of the box**: base, LoRA and prerouter adapters ship
 
126
  - The MLX backend currently targets Apple Silicon; other backends are
127
  on the edge0 roadmap.
128
  - Long contexts grow the KV cache; use shorter contexts to keep peak
129
+ memory at 3 GiB.
130
 
131
  ## Quick start
132