Instructions to use frontier-infra/jebadiah-4b-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use frontier-infra/jebadiah-4b-v2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Use Docker

docker model run hf.co/frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use frontier-infra/jebadiah-4b-v2-GGUF with Ollama:
```
ollama run hf.co/frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M
```
Unsloth Desktop

How to use frontier-infra/jebadiah-4b-v2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use frontier-infra/jebadiah-4b-v2-GGUF with Docker Model Runner:
```
docker model run hf.co/frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M
```

Lemonade

How to use frontier-infra/jebadiah-4b-v2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.jebadiah-4b-v2-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use frontier-infra/jebadiah-4b-v2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use frontier-infra/jebadiah-4b-v2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "frontier-infra/jebadiah-4b-v2-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

jebadiah-4b-v2-GGUF / README.md

jbrashear

Add scripts/decide_ollama.py and a Use it in Ollama section; fix the hf download commands

0268e43 verified about 1 hour ago

preview code

Raw

History Blame Contribute Delete

9.57 kB

	---
	license: apache-2.0
	base_model: frontier-infra/jebadiah-4b-v2
	base_model_relation: quantized
	library_name: gguf
	language:
	- en
	tags:
	- gguf
	- llama.cpp
	- decision-model
	- system-one
	- typed-decisions
	- calibrated-probabilities
	- ainode
	---

	# Jebadiah 4B v2 GGUF

	GGUF builds of [Jebadiah 4B v2](https://huggingface.co/frontier-infra/jebadiah-4b-v2) for [llama.cpp](https://github.com/ggml-org/llama.cpp), which runs on NVIDIA, AMD and
	Apple GPUs and on plain CPUs. Jebadiah answers a typed question (choice, noul or score) with a probability for
	every option, read from one forward pass. Nothing is generated. Code, trainer and evals:
	[getainode/jebadiah](https://github.com/getainode/jebadiah).

	Results and docs

	- Project site, with every result and how to run the models: [jebadiah.ai](https://jebadiah.ai).
	- For the family: [Decision Index 0.2.1](https://huggingface.co/spaces/multimodalart/jev-decision-index): the 27B scores 54.67, #5 of 67 open models (as of 2026-09-26). This is the board's own number; the maintainer validated my run and put it on the leaderboard. [Run record](https://huggingface.co/datasets/frontier-infra/jebadiah-decision-index-results/tree/main/runs/jebadiah-27b-1c0d794f).
	- JevBench v1.4.2: on its 231 public items, run through its own harness, Jebadiah 27B scores 0.866, the same as Jev 1.13.0; Jebadiah 9B v2 scores 0.818. This is my own run on the public items, not the official board, which also uses sealed items. [Details and caveats](https://github.com/getainode/jebadiah#where-we-stand-on-jevbench). The 4B has not been run on either.
	- All sizes: the [Hugging Face collection](https://huggingface.co/collections/frontier-infra/jebadiah-open-system-one-decision-models-6ab80765ddd3fa0b3eba5213), mirrored on [ModelScope](https://www.modelscope.ai/profile/JasonBrashear).

	Which one should I use? For local use, start with [Jebadiah 9B v2 GGUF](https://huggingface.co/frontier-infra/jebadiah-9b-v2-GGUF). On Apple silicon, use an MLX build: [27B](https://huggingface.co/frontier-infra/jebadiah-27b-MLX), [9B v2](https://huggingface.co/frontier-infra/jebadiah-9b-v2-MLX) or [4B v2](https://huggingface.co/frontier-infra/jebadiah-4b-v2-MLX). For vLLM or fine-tuning, use the full weights: [27B](https://huggingface.co/frontier-infra/jebadiah-27b), [9B v2](https://huggingface.co/frontier-infra/jebadiah-9b-v2) or [4B v2](https://huggingface.co/frontier-infra/jebadiah-4b-v2).

	## Files

	Every file was checked on the 260 held-out questions the merged weights were checked on, and compared with
	the merged bf16 weights and with the training run's own eval records.

	\| File \| Size \| Same answer as bf16 \| Same as the run \| choice + noul \| score \| Prob. diff median / max \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| `jebadiah-4b-v2-Q8_0.gguf` \| 4.6 GB \| 256 / 260 \| 257 / 260 \| 171 / 173 \| 86 / 87 \| 0.004 / 0.053 \|
	\| `jebadiah-4b-v2-Q5_K_M.gguf` \| 3.2 GB \| 239 / 260 \| 240 / 260 \| 166 / 173 \| 74 / 87 \| 0.016 / 0.119 \|
	\| `jebadiah-4b-v2-Q4_K_M.gguf` \| 2.8 GB \| 236 / 260 \| 237 / 260 \| 164 / 173 \| 73 / 87 \| 0.022 / 0.355 \|
	\| bf16 weights \| \| \| 259 / 260 \| 172 / 173 \| 87 / 87 \| 0.002 / 0.019 \|

	Which one: `Q8_0` if it fits (it changed 4 answers here); `Q4_K_M` when memory is short. A file
	needs about its own size in GPU or unified memory, plus about 1 GB for a 4k context.

	`Q5_K_M` changes 21 of 260 answers against bf16 (13 on score questions) and moves probabilities more (median 0.016, max 0.12). Use it only when a larger build does not fit.
	`Q4_K_M` changes 24 of 260 answers against bf16 (14 on score questions) and moves probabilities more (median 0.022, max 0.36). Use it only when a larger build does not fit.

	## Run it

	The answer is the log probability of each option label ("A", "B", ...) at the answer position, which
	llama-server's `/completion` returns. The script renders the prompt exactly as AINode does, sends the raw
	text (so the server's own chat template is never used), renormalises over the labels and applies
	`temperatures.json` (choice 1.1167, noul 1.3319, score 1.1974). You need a llama.cpp that knows the `qwen35` architecture: we checked
	v0.5.0 (older builds refuse the file).

	```bash
	hf download frontier-infra/jebadiah-4b-v2-GGUF --include "Q8_0.gguf" --include "scripts/" --include ".json" --include ".jinja" --include "*.txt" --local-dir jebadiah-4b-v2-GGUF
	cd jebadiah-4b-v2-GGUF
	llama-server -m jebadiah-4b-v2-Q8_0.gguf -c 4096 -np 1 --port 8080
	pip install transformers # the tokenizer only, no torch
	python scripts/decide_gguf.py --server http://127.0.0.1:8080 --request scripts/example-request.json
	```

	`--no-temperatures` returns the raw probabilities. For Ollama and LM Studio, see the next sections.

	On `example-request.json` (jebadiah-4b-v2-Q8_0.gguf):

	```json
	{
	"route": {"type": "choice", "choice": "billing", "confidence": 0.397366, "probabilities": {"billing": 0.598244, "support": 0.082792, "sales": 0.318963}},
	"urgent": {"type": "noul", "noul": 0.153519}
	}
	```

	## Use it in Ollama

	Jeb works in Ollama through its API, not `ollama run`: the chat window shows text, while a decision needs the
	probability of every option label with thinking off. `scripts/decide_ollama.py` takes the same arguments and prints
	the same output as `decide_gguf.py`. It sends the rendered prompt to `/api/generate` with `"raw": true` (so Ollama's
	own template is never used) and `"think": false`, asks for one token with the top 20 log probabilities, and stops
	with an error if Ollama's prompt token count differs from the local tokenizer's.

	```bash
	ollama pull hf.co/frontier-infra/jebadiah-4b-v2-GGUF:Q8_0
	hf download frontier-infra/jebadiah-4b-v2-GGUF --include "scripts/" --include ".json" --include ".jinja" --include ".txt" --local-dir jebadiah-4b-v2-GGUF
	pip install transformers # the tokenizer only, no torch
	python jebadiah-4b-v2-GGUF/scripts/decide_ollama.py --model hf.co/frontier-infra/jebadiah-4b-v2-GGUF:Q8_0 --request jebadiah-4b-v2-GGUF/scripts/example-request.json
	```

	Tested with Ollama 0.34.4 and `jebadiah-4b-v2-Q8_0` pulled from this repository: 256 of 260 answers the same as bf16,
	and the same answer as llama-server on the same file on 260 of 260. On the questions with 20 options or
	fewer the probabilities match llama-server to 0.0002 at most, so `example-request.json` prints the numbers above.

	At most 20 options per question. Ollama returns at most the top 20 log probabilities, the same cap as LM Studio
	and AINode's own route. On the 70 Banking77 questions (77 options) the pick matched llama-server on
	70 of 70, but the probabilities moved, so do not rely on them past 20
	options.

	## Use it in LM Studio

	Jeb works in LM Studio through its local server, not the chat window: chat runs with thinking on and shows
	text, while a decision needs the probability of every option label. `scripts/decide_lmstudio.py` takes the same
	arguments and prints the same output as `decide_gguf.py`. It sends AINode's messages to LM Studio's
	`/v1/chat/completions` with thinking off (`"reasoning_effort": "none"`), where LM Studio renders the same prompt
	text the llama-server path sends, and reads the option labels from the top log probabilities that come back. It
	stops with an error if LM Studio's prompt length differs from the local tokenizer's.

	1. In LM Studio, search for `jebadiah-4b-v2` and download `jebadiah-4b-v2-Q8_0.gguf` from this repository.
	2. Open the Developer tab, start the server and load the model. Note the identifier LM Studio shows for it
	(for example `jebadiah-4b-v2`).
	3. In a terminal:

	```bash
	hf download frontier-infra/jebadiah-4b-v2-GGUF --include "scripts/" --include ".json" --include ".jinja" --include ".txt" --local-dir jebadiah-4b-v2-GGUF
	pip install transformers # the tokenizer only, no torch
	python jebadiah-4b-v2-GGUF/scripts/decide_lmstudio.py --model jebadiah-4b-v2 --request jebadiah-4b-v2-GGUF/scripts/example-request.json
	```

	If Require Authentication is on in LM Studio's server settings, create a token there and
	`export LM_API_TOKEN=...` first.

	Tested on the 9B only: LM Studio 0.4.21 with `jebadiah-9b-v2-Q8_0`, 257 of 260 answers the same as bf16, and the
	same answer as llama-server on the same file on 260 of 260. This 4B build uses the same script and the same prompt, but it has not been run in LM Studio.

	At most 20 options per question. LM Studio returns only the top 20 log probabilities, the same cap AINode's own
	route has. On the 77-option Banking77 questions the pick was still right, but the probabilities moved by up to
	0.16, so do not rely on them past 20 options.

	If LM Studio shows a "Vision" tag, ignore it. Some GGUF repositories, including third-party quants of Jeb, ship a vision file (mmproj) from the Qwen base, and LM Studio labels the model Vision because of it. Jeb was not trained on images, and it does not make decisions in the chat window. It decides only through `decide_gguf.py` or `decide_lmstudio.py`, which send the typed question with thinking off and read the probabilities. The mmproj is not needed.

	## How it was measured

	Jevals PubMedQA, Banking77 (77 options) and HelpSteer2, plus Nimble: the merge check's fixed sample (seed
	20260925), the run's option order and temperatures. "Same answer" is the top option; "prob. diff" is the
	largest change on any option against the run's CUDA record. llama-server ran on Metal (M3 Ultra) with the same tokens as the Python renderer on every prompt. Records: `eval/agreement-*.json`.

	## License

	Apache-2.0, as the base model. Made in Texas.