Instructions to use mertkayacs/Tholos-2B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use mertkayacs/Tholos-2B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mertkayacs/Tholos-2B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use mertkayacs/Tholos-2B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mertkayacs/Tholos-2B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mertkayacs/Tholos-2B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mertkayacs/Tholos-2B-GGUF:Q4_K_M

Ollama
How to use mertkayacs/Tholos-2B-GGUF with Ollama:
```
ollama run hf.co/mertkayacs/Tholos-2B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use mertkayacs/Tholos-2B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mertkayacs/Tholos-2B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mertkayacs/Tholos-2B-GGUF with Docker Model Runner:
```
docker model run hf.co/mertkayacs/Tholos-2B-GGUF:Q4_K_M
```

Lemonade

How to use mertkayacs/Tholos-2B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mertkayacs/Tholos-2B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Tholos-2B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use mertkayacs/Tholos-2B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mertkayacs/Tholos-2B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mertkayacs/Tholos-2B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mertkayacs/Tholos-2B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

mertkayacs commited on 1 day ago

Commit

bdba8dc

verified ·

1 Parent(s): 3268f89

Clearer card for first-time visitors

Browse files

Files changed (1) hide show

README.md +30 -37

README.md CHANGED Viewed

@@ -8,62 +8,55 @@ language:
 - en
 tags:
 - gguf
-- llama.cpp
 - ollama
-- agents
 - tool-use
 ---
-# Tholos-2B-GGUF
-**A 2B agent model built by senior AI engineer Mert Kaya:** it passes 137 of 160 Tholos-Bench scenarios on one Kaggle T4 GPU (Q4_K_M, llama.cpp with a JSON schema), 25 more than the model it was trained from ([evaluation](https://huggingface.co/mertkayacs/Tholos-2B#evaluation)).
-GGUF builds of [Tholos-2B](https://huggingface.co/mertkayacs/Tholos-2B), MiniCPM5-2B fine-tuned to be the agent in
-[Tholos](https://github.com/mertkayacs/tholos). The main card has the step format, the training data and the
-benchmark results. This page has the files and the commands.
-| File | Quantization | Size | sha256 |
-|---|---|---|---|
-| Tholos-2B-Q4_K_M.gguf | Q4_K_M | 1.56 GB | `65f4700e0e107d52f5a80c3c513b3387e8cfcef9cb7545f0a2c0a2293d1eadc3` |
-| Tholos-2B-Q8_0.gguf | Q8_0 | 2.68 GB | `5e57b416fb515260276413a9fadc3a67ef3d81f6fceb427cefe475850df1c476` |
-The repo's `SHA256SUMS` file lists the same sums. Q4_K_M is the file our benchmark runs use.
-## llama.cpp
 ```sh
-llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -t 4 -a tholos-2b \
-  --host 127.0.0.1 --port 8080
 ```
-Swap `Q4_K_M` for `Q8_0` to load the larger file. Send a `json_schema` response format with each request so the
-server constrains decoding; the [main card](https://huggingface.co/mertkayacs/Tholos-2B#use-with-llamacpp) has a
-complete request.
-## Ollama
 ```sh
-ollama pull hf.co/mertkayacs/Tholos-2B-GGUF:Q4_K_M
 ```
-The repo carries a `template` and a `params` file. The template renders prompts the way the model was trained, with
-an empty think block before each answer, and matches the training render byte for byte. The params file sets the
-stop tokens and a 16,384-token context (Ollama's default is 4,096).
-Without a template file Ollama picks one automatically. On the base model's official Q4_K_M file that choice left
-out the empty think block, and the file passed 48 of 160 Tholos-Bench scenarios with it and 96 of 160 with our
-template.
-In Tholos, press Detect in Settings and add the model. Tholos asks Ollama for JSON mode and sends
-`reasoning_effort: "none"` on its own.
-## License
-Apache-2.0, the license of the base model MiniCPM5-2B. See the main card for the training data and the terms that
-apply to it.
----
-<a href="https://eschatialabs.com"><img src="https://raw.githubusercontent.com/mertkayacs/jevalt/main/docs/assets/eschatia-labs.png" alt="Eschatia Labs" width="120"></a>
-An [Eschatia Labs](https://eschatialabs.com) project. [Built by Mert Kaya](https://mertkayacs.com).

 - en
 tags:
 - gguf
 - ollama
+- llama.cpp
+- small-language-model
+- local-llm
+- on-device
+- ai-agent
 - tool-use
+- structured-output
+datasets:
+- mertkayacs/tholos-trajectories
 ---
+# Tholos-2B GGUF: local AI agents with Ollama and llama.cpp
+GGUF builds of Tholos-2B, a small language model for AI agents that choose tools for tables, notes and tasks. **Tholos-2B passes 137/160 scenarios; MiniCPM5-2B passes 112/160** on Tholos-Bench with Q4_K_M, llama.cpp JSON schema decoding and one Kaggle T4. Only Tholos-2B was fine-tuned for this task format.
+## Run with Ollama
 ```sh
+ollama pull hf.co/mertkayacs/Tholos-2B-GGUF:Q4_K_M && uv tool install git+https://github.com/mertkayacs/tholos && tholos
 ```
+Open `http://127.0.0.1:7070`, select **Detect** in **Settings**, then add the model. The repo's `template` and `params` preserve the trained prompt format, stop tokens and 16,384-token context.
+Direct Ollama clients should request `json_object` mode and `reasoning_effort: "none"`, then validate each step. Tholos does this for you and applies approval rules.
+## Run with llama.cpp
 ```sh
+llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -t 4 -a tholos-2b --host 127.0.0.1 --port 8080
 ```
+Use `json_schema` response format. The [complete request](https://huggingface.co/mertkayacs/Tholos-2B#run-with-llamacpp) shows the tool definitions, step schema and response loop.
+## Files
+| Quantization | File size |
+|---|---|
+| [Q4_K_M](https://huggingface.co/mertkayacs/Tholos-2B-GGUF/resolve/main/Tholos-2B-Q4_K_M.gguf) | 1.56 GB |
+| [Q8_0](https://huggingface.co/mertkayacs/Tholos-2B-GGUF/resolve/main/Tholos-2B-Q8_0.gguf) | 2.68 GB |
+Use Q4_K_M to match the benchmark. [SHA256SUMS](https://huggingface.co/mertkayacs/Tholos-2B-GGUF/blob/main/SHA256SUMS) records the checksums. File size is not total runtime RAM.
+## Results and limits
+With Q4_K_M on a CPU, Tholos-2B passes **134/160** with llama.cpp JSON schema, against **117/160 for MiniCPM5-2B**. With Ollama JSON mode and the repo template, the counts are **136/160** and **96/160**. Compare results within one setup.
+English only. The model learned Tholos's thirteen tools; other formats are unmeasured. Validate calls, treat external text as untrusted, and keep approvals enabled for consequential actions.
+Apache-2.0. [Full-precision weights, training and evaluation](https://huggingface.co/mertkayacs/Tholos-2B) | [Code](https://github.com/mertkayacs/tholos) | [Project](https://tholos.mertkayacs.com).
+An [Eschatia Labs](https://eschatialabs.com) project. [Mert Kaya](https://mertkayacs.com).