Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxMini-125M-role:Q4_K_M# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxMini-125M-role:Q4_K_MUse pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Thox-ai/ThoxMini-125M-role:Q4_K_M# Run inference directly in the terminal:
./llama-cli -hf Thox-ai/ThoxMini-125M-role:Q4_K_MBuild from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Thox-ai/ThoxMini-125M-role:Q4_K_M# Run inference directly in the terminal:
./build/bin/llama-cli -hf Thox-ai/ThoxMini-125M-role:Q4_K_MUse Docker
docker model run hf.co/Thox-ai/ThoxMini-125M-role:Q4_K_MThoxMini-125M-role
The on-device model for ThoxMini (Pi Zero 2 W, arm64/NEON, 512 MB).
Thox-ai/thox-micro-125m role-tuned so it can actually be used as an assistant.
Why this exists
The base is a base LM with no instruct tuning. Measured on the exact file that had been dropped to the image lane:
Hello, who are you?→<mup> PR snapd#7621 closed: tests: check that the test fails with an error <Created by mvo5>
That is raw training-scrape leaking. It could not ship as an assistant. After tuning, the same prompt returns a coherent deferral.
Measured behaviour
Prompts held out of the training corpus:
| prompt | answer |
|---|---|
| who funded thox | I do not have that on the device. Ask the host. |
| where is the headquarters | I do not have that on the device. Ask the host. |
| how much did the company raise | I do not have that on the device. Ask the host. |
| what is the weather | I do not have that on the device. Ask the host. |
Zero fabrications, and no scrape leakage. In-corpus commands work:
status → "Key online. Storage sealed. No pending transfers."
Two known defects — read before shipping
1. It says "the key", not "the mini." The role corpus was authored for
ThoxKey-S3 and reused verbatim, so device-specific strings are wrong for this
board: do you send my data anywhere → "No. I run on the key. Nothing
leaves it." Correct in substance, wrong in noun. Fix is a ThoxMini variant of
datasets/device/thoxkey-role.jsonl, not a retrain of the recipe.
2. It over-defers on identity. Hello, who are you? returns "I cannot
answer that locally. The host has it." It should name itself. The corpus trains
deferral hard and contains no identity turn for this device; adding one fixes it.
Neither is a fabrication, which is the failure mode that actually blocks shipping. Both are corpus gaps.
Training
| base | Thox-ai/thox-micro-125m (Llama arch, 12 layers, vocab 50304) |
| corpus | 1,360 device-role turns + 205 v5 chat turns = 1,565 rows |
| recipe | full fine-tune, 3 epochs, lr 5e-5, bf16, max_len 512 |
| hardware | local RTX 4060 Ti, ~1 min, $0 |
| final | loss ~0.90, mean token accuracy 0.86 |
v5 and not v4 deliberately. v4 carries tag: correction rows that quote a
falsehood so the assistant can deny it; those rows taught fabrication rather than
preventing it — a 7B trained on them invented "a Partner at Insight Partners"
for a real named person. v5 quarantines them.
EOS is appended explicitly during training. Omitting it is a defect this factory already paid for: adapters trained without it emitted EOS zero times across 12 gate answers and ran to the token cap every time.
- Downloads last month
- 452
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf Thox-ai/ThoxMini-125M-role:Q4_K_M# Run inference directly in the terminal: llama cli -hf Thox-ai/ThoxMini-125M-role:Q4_K_M