Instructions to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0
# Run inference directly in the terminal:
llama cli -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0
# Run inference directly in the terminal:
llama cli -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Use Docker

docker model run hf.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

LM Studio
Jan

vLLM

How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Ollama
How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with Ollama:
```
ollama run hf.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0
```
Unsloth Desktop

How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with Docker Model Runner:
```
docker model run hf.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0
```

Lemonade

How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Run and chat with the model

lemonade run user.Qwen3.8-Flash-CIRU-STRIX-IU4-Q8_0

List all available models

lemonade list

Hermes Agent

How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

jcbtc commited on 15 days ago

Commit

88fd261

verified ·

1 Parent(s): 2e78ed4

Clarify v2.0 Git source, complete MTP profile and draft-depth scope

Browse files

Files changed (4) hide show

README.md +39 -3
benchmarks/v2.0/METHODOLOGY.md +7 -0
benchmarks/v2.0/git-source.json +18 -0
v2.0-checksums.sha256 +3 -2

README.md CHANGED Viewed

@@ -39,7 +39,7 @@ tags:
 Version 2.0 brings together CIRU's retained ROCm 10 correctness and execution improvements. The measured strengths are **42.3 tok/s on the controlled non-thinking coding probe**, **382 tok/s cold prefill at 2K**, and **0.03045 mean forward KL to BF16** on the shared diagnostic panel. The target, MTP and PLE weights are unchanged.
 > [!IMPORTANT]
-> **Use the [v2.0 runtime source](runtime/v2.0/ciru-runtime-v2.0-source.tar.gz).** This text-only package requires the custom CIRU runtime and all three `ple/` files. The `mtp/` draft enables speculation. Stock `llama.cpp` and Hugging Face hosted inference do not run this package correctly.
 >
 > **The target GGUF, MTP draft and PLE weights are unchanged.** Existing users only need the new runtime and profile. v2.0 identifies the runtime/package release.
@@ -161,13 +161,23 @@ cd ciru-runtime-v2.0
 ./scripts/ciru/setup-linux-amd.sh --install-host-deps
 ```
-The source archive includes `docs/BUILD_LINUX.md`, `docs/RUNNING.md`, provenance, licenses and third-party notices. Do not use the old GitHub v1.1.1 tag as the v2.0 source; this release's exact source is the archive linked above.
 Benchmarked on NixOS with ROCm10. The runtime also passed an Ubuntu 24.04 container build and GPU smoke test.
 ## Run the production profile
-From the extracted runtime directory:
 ```bash
 BUILD_DIR="$PWD/build-gfx1151-sdk" \
@@ -177,6 +187,30 @@ BUILD_DIR="$PWD/build-gfx1151-sdk" \
 The profile uses one slot, a 262,144-token context, batch 2048/microbatch 512, 8 target threads, F16 target KV, Q8_0 draft KV, flash attention on, maximum MTP depth 6, p-min 0 and p-split 0.10. It enables a separate 4 GiB PLE cache with 16 workers, an 8 GiB prompt cache, idle-slot caching and context checkpoints. The server binds to `127.0.0.1:8080`.
 ```bash
 curl http://127.0.0.1:8080/v1/chat/completions \
   -H 'Content-Type: application/json' \
@@ -195,6 +229,8 @@ For target-only serving, set `ENABLE_MTP=0` before launching. This omits the dra
 Thinking defaults are temperature 1.0, top-p 0.95, top-k 20, min-p 0. For non-thinking mode, send `"chat_template_kwargs":{"enable_thinking":false}` with temperature 0.7, top-p 0.8 and presence penalty 1.5. Normal chat can retain prefix caching; the cold benchmark settings are not the production preset.
 ## Historical task-quality results on the unchanged weights
 These earlier H96/depth-1 results were **not rerun as full suites on v2.0**. The current release evidence comprises the bounded BF16 panel, controlled coding probe, context sweeps, and scoped runtime correctness/build checks described above.

 Version 2.0 brings together CIRU's retained ROCm 10 correctness and execution improvements. The measured strengths are **42.3 tok/s on the controlled non-thinking coding probe**, **382 tok/s cold prefill at 2K**, and **0.03045 mean forward KL to BF16** on the shared diagnostic panel. The target, MTP and PLE weights are unchanged.
 > [!IMPORTANT]
+> **Use the [v2.0 runtime archive](runtime/v2.0/ciru-runtime-v2.0-source.tar.gz) or matching [GitHub `v2.0` tag](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/tree/v2.0).** This text-only package requires the custom CIRU runtime and all three `ple/` files. The `mtp/` draft enables speculation. Stock `llama.cpp` and Hugging Face hosted inference do not run this package correctly.
 >
 > **The target GGUF, MTP draft and PLE weights are unchanged.** Existing users only need the new runtime and profile. v2.0 identifies the runtime/package release.
 ./scripts/ciru/setup-linux-amd.sh --install-host-deps
 ```
+The source archive includes build/run instructions, provenance, licenses and third-party notices. The matching [GitHub `v2.0` tag](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/tree/v2.0) now provides the same source in Git: all 3,537 file contents and Git modes match the published archive. [Release downloads](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/releases/tag/v2.0) · [Diff from v1.1.1](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/compare/v1.1.1...v2.0) · [Recorded source identity](benchmarks/v2.0/git-source.json).
+As an alternative to extracting the archive, with Git installed:
+```bash
+git clone --branch v2.0 --single-branch https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4.git ciru-runtime-v2.0
+cd ciru-runtime-v2.0
+./scripts/ciru/setup-linux-amd.sh --install-host-deps
+```
+The original archive and Hugging Face `v2.0` revision remain unchanged. Later documentation clarifications appear on `main`; use the [current running instructions](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/blob/main/docs/RUNNING.md) for profile verification and draft-depth guidance.
 Benchmarked on NixOS with ROCm10. The runtime also passed an Ubuntu 24.04 container build and GPU smoke test.
 ## Run the production profile
+From the cloned or extracted runtime directory:
 ```bash
 BUILD_DIR="$PWD/build-gfx1151-sdk" \
 The profile uses one slot, a 262,144-token context, batch 2048/microbatch 512, 8 target threads, F16 target KV, Q8_0 draft KV, flash attention on, maximum MTP depth 6, p-min 0 and p-split 0.10. It enables a separate 4 GiB PLE cache with 16 workers, an 8 GiB prompt cache, idle-slot caching and context checkpoints. The server binds to `127.0.0.1:8080`.
+### Confirm the MTP profile
+The launcher exports `profiles/strix-halo-production.env`, including `CIRU_MTP_SHORTLIST=32768` and `CIRU_MTP_TOPK10=1`. Compiling v2.0 and setting `--spec-draft-n-max 6` alone does not enable those optimizations. With MTP enabled, confirm this startup line:
+```text
+CIRU MTP shortlist enabled: 32768 / 248320 vocabulary rows; full target verification retained
+```
+For direct `llama-server` launches, use the [complete exported environment and flags](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/blob/main/docs/RUNNING.md#2-launch-the-audited-public-profile).
+### Choose a draft depth
+**Depth 6 is the released default; the fastest depth depends on the workload.** When later draft tokens are frequently rejected, depth 3 may be faster, including for some short interactive completions. Restart with this command to change only the maximum depth:
+```bash
+MTP_DEPTH=3 BUILD_DIR="$PWD/build-gfx1151-sdk" \
+  MODEL_DIR=/absolute/path/to/Qwen3.8-Flash-CIRU-STRIX-IU4 \
+  ./scripts/ciru/run-server.sh
+```
+The controlled coding result above does not establish that six is optimal for short chat, long contexts or batched serving. The context sweeps have MTP off and do not compare draft depths. No adaptive depth selection is enabled. Single sampled completions are useful directional feedback, but need matched settings and repeated measurements to support a general speed claim. See [comparison details](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/blob/main/docs/RUNNING.md#confirm-the-mtp-profile-and-choose-a-draft-depth).
+### Send a chat request
 ```bash
 curl http://127.0.0.1:8080/v1/chat/completions \
   -H 'Content-Type: application/json' \
 Thinking defaults are temperature 1.0, top-p 0.95, top-k 20, min-p 0. For non-thinking mode, send `"chat_template_kwargs":{"enable_thinking":false}` with temperature 0.7, top-p 0.8 and presence penalty 1.5. Normal chat can retain prefix caching; the cold benchmark settings are not the production preset.
+`min_p=0` explicitly follows [Qwen's model-specific recommendation](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/f5d08274bafd880402bd16f5e3e6c514136ec06c/README.md#best-practices); v1.1 could inherit llama.cpp's `0.05`. This disables an additional probability filter and can change sampled output. We have not established that either value gives better answer quality in a controlled A/B test. Set `MIN_P=0.05` with the launcher to select the previous inherited value; request-level settings can override the server default.
 ## Historical task-quality results on the unchanged weights
 These earlier H96/depth-1 results were **not rerun as full suites on v2.0**. The current release evidence comprises the bounded BF16 panel, controlled coding probe, context sweeps, and scoped runtime correctness/build checks described above.

benchmarks/v2.0/METHODOLOGY.md CHANGED Viewed

@@ -42,3 +42,10 @@ The source archive combines RC2 core code with the v1.1.1 portable SDK helpers a
 ## Controlled non-thinking coding result
 The release headline 42.28–42.31 tok/s comes from two retained repetitions: 57 prompt tokens and 520 generated tokens, an explicitly closed thinking block, greedy seed 123, fixed-six MTP, 16K context and the performance CPU governor. It is a served coding-probe rate. Source rows, raw hashes, actual commands, prompt and generation settings are in controlled-coding-42tps.json. It is not a HumanEval average or a speedup against a different workload.

 ## Controlled non-thinking coding result
 The release headline 42.28–42.31 tok/s comes from two retained repetitions: 57 prompt tokens and 520 generated tokens, an explicitly closed thinking block, greedy seed 123, fixed-six MTP, 16K context and the performance CPU governor. It is a served coding-probe rate. Source rows, raw hashes, actual commands, prompt and generation settings are in controlled-coding-42tps.json. It is not a HumanEval average or a speedup against a different workload.
+## Draft-depth scope and Git source (documentation update, 2026-09-06)
+The controlled coding result uses the complete fixed-six environment, including `CIRU_MTP_SHORTLIST=32768` and `CIRU_MTP_TOPK10=1`. Draft depth alone does not reproduce that configuration. It is not evidence that six is optimal for every workload or specifically tuned for long contexts or batching. The context sweep has MTP off and cannot answer that question. Lower acceptance at greater depth can leave extra draft work unrecovered; a shorter draft may be faster for a particular interactive workload. See the [profile verification and comparison instructions](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/blob/main/docs/RUNNING.md#confirm-the-mtp-profile-and-choose-a-draft-depth).
+The [GitHub `v2.0` tag](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/tree/v2.0) now identifies the exact original source archive tree, directly following v1.1.1. File contents, executable modes and symlink targets were checked for all 3,537 entries. [Machine-readable source identity](git-source.json) records the commit and archive checksum. The original archive, release weights and Hugging Face `v2.0` revision are unchanged. This documentation update adds no new inference measurements.

benchmarks/v2.0/git-source.json ADDED Viewed

	@@ -0,0 +1,18 @@

+{
+  "schema_version": 1,
+  "recorded_date": "2026-09-06",
+  "release": "v2.0",
+  "github_repository": "https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4",
+  "github_tag": "v2.0",
+  "github_commit": "3e21240ec7935b18fd39e1f07fd80f8d905ba968",
+  "github_parent_commit": "764ee491d4bc765cb8414d9bb17c24a5b364e097",
+  "huggingface_release_revision": "ecaaf44465cc667b8ce349b6512e1fe1a08bf3b6",
+  "source_archive": "runtime/v2.0/ciru-runtime-v2.0-source.tar.gz",
+  "source_archive_sha256": "938bd98629fc25014973082eef5a3ea4d45dd5acb6732b608e05e0ecde08f044",
+  "archive_root": "ciru-runtime-v2.0",
+  "verified_git_entries": 3537,
+  "verification": "Every archived file content, executable mode and symlink target matches the Git tag tree after removing the enclosing archive directory.",
+  "runtime_or_weights_changed_by_git_publication": false,
+  "documentation_policy": "Later documentation clarifications are on main. The original source archive and Hugging Face v2.0 revision are preserved.",
+  "huggingface_release_tag_object": "aa3ed2b0d0429034740c13c783d463e0f6872783"
+}

v2.0-checksums.sha256 CHANGED Viewed

@@ -1,15 +1,16 @@
 c16fda8809ad10f0d25d2243f4f83e76c29b80e0ee1622a4357a6f7235df97b4  .gitattributes
-8893e9d64d533b2de92d3efffa72b1d2417267f86dabca46814ad2a01c0ef83b  README.md
 3a46a7fd054305b2c1a931682015d93094312fb3b76c8ef83a2adb350b0f584c  assets/qwen38-flash-ciru-strix-iu4.jpg
 e260f72cba14e4c4e523794345f7de001444626240ab620d96dc44ee51beee90  assets/v2-bf16-fidelity.png
 88dd940fda0790ec6f474dc0a99019703114df253758ee14136c9686e93ea404  assets/v2-context-sweep.png
-ffb976a91e9bd793f580d90b1db6adecf7e29d07b5d6a45958a2817d5f805414  benchmarks/v2.0/METHODOLOGY.md
 4caca1c5e03ede5a9518a379e6ff24d161847454fe5a9a498caedc11ee4cb14f  benchmarks/v2.0/bf16-fidelity.csv
 01d34892371352271ddb98ac0a46cbfd0c3a8509abfcf7a0e826d8074a067462  benchmarks/v2.0/bf16-fidelity.json
 cfb6b00e9a98525b9abee62ce37431677d5dae33c08b00448aa65585edf4d9db  benchmarks/v2.0/ciru-provenance.json
 4c7299b4d220b84d7e982f5cdc54dd7e1ba3c9a10becb6bcc17cf146a12a9b82  benchmarks/v2.0/comparison.json
 a16df59acb7bfbbf436ea0f0356cd5e3760f8b2e82470dc65187008f2e336e26  benchmarks/v2.0/competitor-artifacts.json
 f665c5b67e162ee08a465e17702704cd972a43e2f4268a1ac030d21046091308  benchmarks/v2.0/controlled-coding-42tps.json
 fc4578f2148007209a4307ce34e8b0bb8fc72b28386e7b4e8506ee0bdf8b77b8  benchmarks/v2.0/mtp-off-context-sweep.csv
 f0dbce4afd4e6e6440675f8c1b1d4ba42aa19c89cb47f3db714304b3969baf5a  benchmarks/v2.0/performance-store-verify.json
 e16134cefe5935ec166920086d5d2bfa35f240fd6e351fe6739696afa610391b  benchmarks/v2.0/quality-store-corrected-run.json

 c16fda8809ad10f0d25d2243f4f83e76c29b80e0ee1622a4357a6f7235df97b4  .gitattributes
+c1640924dd034fd2731875a737a8fb0c9d579d44eb6e3df39503133f5cfb2fde  README.md
 3a46a7fd054305b2c1a931682015d93094312fb3b76c8ef83a2adb350b0f584c  assets/qwen38-flash-ciru-strix-iu4.jpg
 e260f72cba14e4c4e523794345f7de001444626240ab620d96dc44ee51beee90  assets/v2-bf16-fidelity.png
 88dd940fda0790ec6f474dc0a99019703114df253758ee14136c9686e93ea404  assets/v2-context-sweep.png
+47c8872e4cfddd8a8507312464c2546041916aecf6325526e1e064234e9f17a6  benchmarks/v2.0/METHODOLOGY.md
 4caca1c5e03ede5a9518a379e6ff24d161847454fe5a9a498caedc11ee4cb14f  benchmarks/v2.0/bf16-fidelity.csv
 01d34892371352271ddb98ac0a46cbfd0c3a8509abfcf7a0e826d8074a067462  benchmarks/v2.0/bf16-fidelity.json
 cfb6b00e9a98525b9abee62ce37431677d5dae33c08b00448aa65585edf4d9db  benchmarks/v2.0/ciru-provenance.json
 4c7299b4d220b84d7e982f5cdc54dd7e1ba3c9a10becb6bcc17cf146a12a9b82  benchmarks/v2.0/comparison.json
 a16df59acb7bfbbf436ea0f0356cd5e3760f8b2e82470dc65187008f2e336e26  benchmarks/v2.0/competitor-artifacts.json
 f665c5b67e162ee08a465e17702704cd972a43e2f4268a1ac030d21046091308  benchmarks/v2.0/controlled-coding-42tps.json
+2477cf6ef0e57b7cc0ca3ce47839842ad675dcf93c4bb90feb958f1aa8cf79f4  benchmarks/v2.0/git-source.json
 fc4578f2148007209a4307ce34e8b0bb8fc72b28386e7b4e8506ee0bdf8b77b8  benchmarks/v2.0/mtp-off-context-sweep.csv
 f0dbce4afd4e6e6440675f8c1b1d4ba42aa19c89cb47f3db714304b3969baf5a  benchmarks/v2.0/performance-store-verify.json
 e16134cefe5935ec166920086d5d2bfa35f240fd6e351fe6739696afa610391b  benchmarks/v2.0/quality-store-corrected-run.json