Instructions to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL
# Run inference directly in the terminal:
llama cli -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL
# Run inference directly in the terminal:
llama cli -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL
# Run inference directly in the terminal:
./llama-cli -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Use Docker

docker model run hf.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

LM Studio
Jan

vLLM

How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Ollama
How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with Ollama:
```
ollama run hf.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL
```
Unsloth Desktop

How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with Docker Model Runner:
```
docker model run hf.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL
```

Lemonade

How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Run and chat with the model

lemonade run user.Qwen3.8-Flash-Next-REAP-320-GGUF-UD-Q2_K_XL

List all available models

lemonade list

Hermes Agent

How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF:UD-Q2_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

AnonimousA commited on Sep 6

Commit

5fb294f

verified ·

1 Parent(s): 0d79102

Card: prefill section -- the 26.8 GiB PLE table and --lazy-mode on-direct (PR 28136), measured cold on Windows

Browse files

Files changed (1) hide show

README.md +34 -1

README.md CHANGED Viewed

@@ -125,11 +125,44 @@ but leaves under 1 GiB free.
 > make sure your timer sees the reasoning stream** — and require two independent clocks to
 > agree before you believe a number.
 ## Traps (measured, not theoretical)
 1. `--parallel > 1` requires `--kv-unified`.
 2. Empty `content` with tight `max_tokens`: reasoning burns the budget. Detect and retry with a higher cap.
-3. First requests after a cold load pay disk; throughput climbs over ~3-4 requests to its warm ceiling.
 4. **`--spec-type draft-mtp` exists in llama.cpp mainline but does not work here.** Mainline has the flag and not the MTP graph for `qwen4exp`, nor the tensor borrowing the *shared* heads need; it fails with `model doesn't contain MTP layers`. Use Unsloth's build.
 5. `"reasoning_effort": "none"` returns **HTTP 500** — the chat template accepts only `xhigh` / `medium` / `low`.

 > make sure your timer sees the reasoning stream** — and require two independent clocks to
 > agree before you believe a number.
+## New (2026-09-06): prefill — the PLE table is why your prompt speed swings 3×, and `--lazy-mode on-direct` fixes it
+Flash-Next carries a 26.8 GiB per-layer-embedding table (`per_layer_token_embd.weight`, 320M rows × 90 B) that
+is untouched by expert pruning and stays on the host. llama.cpp reads it through `mmap`: one page fault per
+~90-byte row, on every token of the prompt. Whether those pages are in the OS cache decides your prefill
+speed — and on a machine that also loads other 20–60 GB models, they usually are not. Our production log
+showed 245–2652 tok/s on 11–25k-token prompts for exactly this reason, and repetitive benchmark prompts
+(few distinct rows) hide it completely.
+[PR #28136](https://github.com/ggml-org/llama.cpp/pull/28136) adds `--lazy-mode on-direct`: positioned reads
+from a worker pool instead of demand faults. Measured on the `Q2/` build (RTX 5090, Windows 11, 95 GB RAM,
+NVMe) with a genuinely cold cache (`RAMMap -Et` + fresh process per arm), a real 25k-token mixed prose+code
+prompt, `cache_prompt: false`, two cycles:
+| prefill tok/s | cold | warm (same prompt) | different rows, warm process |
+|---|---:|---:|---:|
+| `--lazy-mode on` (mmap, the default) | **167** | 681 | 442 |
+| **`--lazy-mode on-direct`** | **621** | 656 | 619 |
+| `--lazy-mode off` (table resident, +26.8 GiB RAM) | 590 | 673 | 573 |
+`on-direct` is the one that gives the same number every time; `off` is fast while the resident table
+survives but loses it whenever Windows trims the working set. With Unsloth's MTP head loaded (same
+protocol): 259 → 506 cold, 404 → 604 on new rows, decode unchanged (72 tok/s). A 28k-token prompt now
+prefills cold in 44 s instead of 74–152 s.
+To use it today: the PR is still open, it is **POSIX-only as written** (Windows prints
+`--lazy-mode on-direct is not supported on this platform` and silently falls back to mmap — you need
+[this Windows port](https://gist.github.com/mjungnickel18/083da5cde978ca3b102952029d4a991a) from the PR
+thread), and the prebuilt Unsloth binaries reject the value. We build Unsloth's b10798 mix source + the PR +
+the Windows port, which keeps the MTP head working. Full numbers, including the Windows-native table, are
+[in the PR thread](https://github.com/ggml-org/llama.cpp/pull/28136). If you cannot build:
+`--lazy-mode off` (mainline and Unsloth builds accept it) gets most of the cold gain for 26.8 GiB of RAM.
 ## Traps (measured, not theoretical)
 1. `--parallel > 1` requires `--kv-unified`.
 2. Empty `content` with tight `max_tokens`: reasoning burns the budget. Detect and retry with a higher cap.
+3. First requests after a cold load pay disk; throughput climbs over ~3-4 requests to its warm ceiling. The disk cost is the PLE table (see the prefill section above); `--lazy-mode on-direct` removes it.
 4. **`--spec-type draft-mtp` exists in llama.cpp mainline but does not work here.** Mainline has the flag and not the MTP graph for `qwen4exp`, nor the tensor borrowing the *shared* heads need; it fails with `model doesn't contain MTP layers`. Use Unsloth's build.
 5. `"reasoning_effort": "none"` returns **HTTP 500** — the chat template accepts only `xhigh` / `medium` / `low`.