Instructions to use Edge0/Edge0-8B-A1B-preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Edge0/Edge0-8B-A1B-preview with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("Edge0/Edge0-8B-A1B-preview")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use Edge0/Edge0-8B-A1B-preview with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "Edge0/Edge0-8B-A1B-preview"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Edge0/Edge0-8B-A1B-preview"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use Edge0/Edge0-8B-A1B-preview with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "Edge0/Edge0-8B-A1B-preview"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "Edge0/Edge0-8B-A1B-preview"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "Edge0/Edge0-8B-A1B-preview",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use Edge0/Edge0-8B-A1B-preview with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "Edge0/Edge0-8B-A1B-preview"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Edge0/Edge0-8B-A1B-preview

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Edge0/Edge0-8B-A1B-preview with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "Edge0/Edge0-8B-A1B-preview"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Edge0/Edge0-8B-A1B-preview" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Edge0-8B-A1B-preview / README.md

bupalinyu

Upload README.md with huggingface_hub

fbee063 verified 22 days ago

preview code

Raw History Blame

3.89 kB

metadata

license: apache-2.0
library_name: mlx
tags:
  - moe
  - edge-inference
  - prerouter
  - lora
  - ssd-offload
base_model:
  - inclusionAI/Ling-3.0-tiny-base
pipeline_tag: text-generation

Edge0-8b-a1b Preview

An 8B-class sparse MoE that runs on a phone — 1.0 GiB of active memory, experts streamed from SSD.

This repository hosts the edge0-8b checkpoint of the edge0 streaming MoE inference framework. The full 4-bit checkpoint (≈4.2 GB) stays on storage; edge0 mmaps it and streams MoE experts from SSD on demand, with a trained prerouter head that predicts the next token's expert routing one step ahead so expert loads hide completely behind the forward pass. The result: an 8B-class MoE with ≈1.0 GiB of active memory — a phone-class memory budget, with no upfront weight download into RAM and no model sharding.

Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.

Model summary


Base model	inclusionAI Ling 3.0 tiny (bailing hybrid, MLA + MoE, ≈7.9B total / ≈1.2B active)
Quantization	4-bit
Layers	24
Experts / active per token	128 / 8 (K=8)
Framework	edge0 (MLX backend)
Contents	base checkpoint + `lora_edge0_8b.safetensors` + `prerouter_edge0_8b.safetensors`

The LoRA and prerouter adapters are co-located with the base checkpoint and load automatically — this repository is a complete, ready-to-run model directory for edge0.

Quality (self-evaluation)

Internal self-evaluation of this checkpoint (int4 + adapters) relative to the fp16 base model — the loss of the edge0 pipeline is small: 2.8 points on average, with MMLU-Pro above the base (max 100, all self-run):

Benchmark	edge0-8b (int4)	Base fp16
AIME 2026	63.3	73.3
HumanEval	91.5	92.7
GPQA-Diamond	70.7	71.2
MMLU-Pro	70.1	65.8
IFBench	53.9	60.6
Average	69.9	72.7

Performance

Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:

Decode speed	Prefill throughput (cold / warm)	Peak active memory*
23.9–25.3 tok/s	500 / 1428 tok/s	1.0 GiB

*Short contexts; long contexts add KV cache (≈3.3 GiB at 3.3k tokens). Expert weights stream from SSD via mmap and are not resident.

Quick start

pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'

# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-8b-a1b-preview --local-dir ./Edge0-8b-a1b-preview

# Run it
export EDGE0_8B_MODEL=$PWD/Edge0-8b-a1b-preview
edge0 chat --name edge0-8b --prompt "Introduce yourself"

# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-8b --port 8083

For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.

License

Apache 2.0. See LICENSE.