Instructions to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Hob-forge/Qwen3.5-4B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hob-forge/Qwen3.5-4B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Ollama
How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with Ollama:
```
ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-4B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Hob-forge/Qwen3.5-4B-Instruct-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.5-4B-Instruct-GGUF / RUNNING.md

RhinoWithAcape

RUNNING.md — every step executed

cfe9f1c verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

4.8 kB

Running Qwen3.5-4B-Instruct (Hob Forge Edition) — zero to first tool call

Every step below was executed on real hardware before this shipped. Two paths: A for any machine with ollama (easiest), B for llama.cpp directly (most control). Numbers in the card's tables were measured with path B on a 12GB RTX 5070; an 8GB card runs everything here — see the memory table in the card.

Path A — ollama (any OS, 5 minutes)

Install ollama: https://ollama.com/download (one installer, all platforms).
Pull and run this edition directly from HF:
```
ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
```
First run downloads 2.6GB. You're chatting when the >>> appears.

Thinking mode: the model reasons out loud by default. For clean answers via the API:

curl http://localhost:11434/api/chat -d '{
  "model": "hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M",
  "messages": [{"role":"user","content":"Why is the sky blue? One sentence."}],
  "think": false, "stream": false
}'

Note think sits at the top level of the body — not inside options. (Scar #1: we lost an afternoon to that once.)

Context size: ollama defaults small. For the long context this arch is great at: ollama run … then /set parameter num_ctx 16384 — or bake it into a Modelfile. Check what you actually got: the server log prints the KV allocation.

Path B — llama.cpp (measured-numbers path)

Get a 2026 build — this is a hybrid-attention architecture; builds older than ~March 2026 will fail with unknown-architecture errors (Scar #2: a December build converted this model into a file that crashed newer runtimes — toolchain vintage matters in both directions):
```
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j   # drop -DGGML_CUDA=ON for CPU
```

Download a quant (Q4_K_M recommended — see the card's which-file table):

hf download Hob-forge/Qwen3.5-4B-Instruct-GGUF Qwen3.5-4B-Instruct-Q4_K_M.gguf --local-dir .

Chat, single-turn, GPU:
```
./build/bin/llama-cli -m Qwen3.5-4B-Instruct-Q4_K_M.gguf \
  -st -p "Explain mmap in one paragraph." -ngl 99 -c 8192
```
- -st (single-turn) matters: without it llama-cli may enter interactive conversation mode and appear to "hang" waiting at a > prompt (Scar #3 — we watched a script wait four hours for someone to type).
- 8GB card: this fits whole (-ngl 99). If you're sharing the GPU, -ngl 20 splits layers to CPU gracefully.

Serve an OpenAI-compatible API:

./build/bin/llama-server -m Qwen3.5-4B-Instruct-Q4_K_M.gguf -ngl 99 -c 16384 --port 8080

First tool call (the part most cards skip)

The template supports native tool calling. Against llama-server:

curl http://localhost:8080/v1/chat/completions -d '{
  "model": "qwen3.5-4b",
  "messages": [{"role":"user","content":"What is 37.2% of 8412? Use the calculator."}],
  "tools": [{"type":"function","function":{"name":"calculator",
    "description":"Evaluate a math expression",
    "parameters":{"type":"object","properties":{"expression":{"type":"string"}},
    "required":["expression"]}}}]
}'

Expected: a tool_calls entry with {"expression":"8412*0.372"}-style arguments. We ran exactly this before shipping. If you get prose instead of a tool call, your runtime is too old to render this template's tool block — see step B1.

Sampling that works (verified)

Mode	temp	top_p	top_k
Thinking (default)	0.6	0.95	20
Non-thinking	0.7	0.8	20

Troubleshooting (our scars, your shortcuts)

Symptom	Cause & fix
`unknown architecture` / load error	llama.cpp too old — build ≥ March 2026 (B1).
`blk.32 … not found` on a self-converted file	You converted with a text-only load that dropped the MTP block — convert from the full snapshot, or use our files.
Appears to hang at a `>`	Interactive mode — add `-st`, give `-p`.
Painfully slow on CPU-only	The hybrid DeltaNet layers' CPU path is immature; this model wants a GPU. CPU works for testing, not serving.
`<think>` text in answers	By design. `think:false` (ollama, top-level) / `enable_thinking=False` (transformers) / parse it out.
IQ4_XS slower than Q4_K_M despite being smaller	Correct and measured (45 vs 138 t/s on RTX) — i-quant dequant cost. Use Q4_K_M unless the 200MB matters.
Garbled/endless output	Check eos: template expects `<

Something not covered? Open a discussion on the repo — we actually answer. — Hob Forge