frontier-infra/jebadiah-27b-GGUF

Jebadiah 27B GGUF

GGUF builds of Jebadiah 27B for llama.cpp, which runs on NVIDIA, AMD and Apple GPUs and on plain CPUs. Jebadiah answers a typed question (choice, noul or score) with a probability for every option, read from one forward pass. Nothing is generated. Code, trainer and evals: getainode/jebadiah.

Files

Every file was checked on the 260 held-out questions the merged weights were checked on, and compared with the merged bf16 weights and with the training run's own eval records.

File	Size	Same answer as bf16	Same as the run	choice + noul	score	Prob. diff median / max
jebadiah-27b-Q8_0.gguf	29.0 GB	260 / 260	259 / 260	173 / 173	86 / 87	0.002 / 0.025
jebadiah-27b-Q5_K_M.gguf	19.5 GB	255 / 260	256 / 260	172 / 173	84 / 87	0.007 / 0.060
jebadiah-27b-Q4_K_M.gguf	16.8 GB	258 / 260	259 / 260	172 / 173	87 / 87	0.013 / 0.127
bf16 weights	55.6 GB		259 / 260	173 / 173	86 / 87	0.002 / 0.019

Which one: Q8_0 if it fits (it changed no answer here); Q4_K_M when memory is short. A file needs about its own size in GPU or unified memory, plus about 1 GB for a 4k context.

Run it

The answer is the log probability of each option label ("A", "B", ...) at the answer position, which llama-server's /completion returns. The script renders the prompt exactly as AINode does, sends the raw text (so the server's own chat template is never used), renormalises over the labels and applies temperatures.json (choice 1.2321, noul 1.297, score 0.7558). You need a llama.cpp that knows the qwen35 architecture: we checked v0.5.0 (older builds refuse the file).

hf download frontier-infra/jebadiah-27b-GGUF --include "*Q8_0.gguf" "scripts/*" "*.json" "*.jinja" "*.txt" --local-dir jebadiah-27b-GGUF
cd jebadiah-27b-GGUF
llama-server -m jebadiah-27b-Q8_0.gguf -c 4096 -np 1 --port 8080
pip install transformers        # the tokenizer only, no torch
python scripts/decide_gguf.py --server http://127.0.0.1:8080 --request scripts/example-request.json

--no-temperatures returns the raw probabilities. For LM Studio, see the next section. Ollama was not checked: a decision needs the log probability of every option label at one position; if your runtime cannot return those, use llama-server.

On example-request.json (jebadiah-27b-Q8_0.gguf):

{
 "route": {"type": "choice", "choice": "billing", "confidence": 0.647004, "probabilities": {"billing": 0.764669, "support": 0.034001, "sales": 0.20133}},
 "urgent": {"type": "noul", "noul": 0.209376}
}

Use it in LM Studio

Jeb works in LM Studio through its local server, not the chat window: chat runs with thinking on and shows text, while a decision needs the probability of every option label. scripts/decide_lmstudio.py takes the same arguments and prints the same output as decide_gguf.py. It sends AINode's messages to LM Studio's /v1/chat/completions with thinking off ("reasoning_effort": "none"), where LM Studio renders the same prompt text the llama-server path sends, and reads the option labels from the top log probabilities that come back. It stops with an error if LM Studio's prompt length differs from the local tokenizer's.

1. In LM Studio, search for jebadiah-27b and download jebadiah-27b-Q8_0.gguf from this repository.
2. Open the Developer tab, start the server and load the model. Note the identifier LM Studio shows for it (for example jebadiah-27b).
3. In a terminal:

hf download frontier-infra/jebadiah-27b-GGUF --include "scripts/*" "*.json" "*.jinja" "*.txt" --local-dir jebadiah-27b-GGUF
pip install transformers        # the tokenizer only, no torch
python jebadiah-27b-GGUF/scripts/decide_lmstudio.py --model jebadiah-27b --request jebadiah-27b-GGUF/scripts/example-request.json

If Require Authentication is on in LM Studio's server settings, create a token there and export LM_API_TOKEN=... first.

Tested on the 9B only: LM Studio 0.4.21 with jebadiah-9b-v2-Q8_0, 257 of 260 answers the same as bf16, and the same answer as llama-server on the same file on 260 of 260. This 27B build uses the same script and the same prompt, but it has not been run in LM Studio.

At most 20 options per question. LM Studio returns only the top 20 log probabilities, the same cap AINode's own route has. On the 77-option Banking77 questions the pick was still right, but the probabilities moved by up to 0.16, so do not rely on them past 20 options.

If LM Studio shows a "Vision" tag, ignore it. Some GGUF repositories, including third-party quants of Jeb, ship a vision file (mmproj) from the Qwen base, and LM Studio labels the model Vision because of it. Jeb was not trained on images, and it does not make decisions in the chat window. It decides only through decide_gguf.py or decide_lmstudio.py, which send the typed question with thinking off and read the probabilities. The mmproj is not needed.

How it was measured

Jevals PubMedQA, Banking77 (77 options) and HelpSteer2, plus Nimble: the merge check's fixed sample (seed 20260925), the run's option order and the run's temperatures, so the probability differences compare like with like (the shipped temperatures.json since 1c0d794f uses 0.7558 for score; top picks do not depend on it). "Same answer" is the top option; "prob. diff" is the largest change on any option against the run's CUDA record. llama-server ran on Metal (M3 Ultra) with the same tokens as the Python renderer on every prompt. Records: eval/agreement-*.json.

License

Apache-2.0, as the base model. Made in Texas.