RUNNING.md

# Running Qwen3.5-4B-Instruct (Hob Forge Edition) — zero to first tool call

Every step below was executed on real hardware before this shipped. Two paths: **A** for
any machine with ollama (easiest), **B** for llama.cpp directly (most control). Numbers
in the card's tables were measured with path B on a 12GB RTX 5070; an 8GB card runs
everything here — see the memory table in the card.

---

## Path A — ollama (any OS, 5 minutes)

1. Install ollama: https://ollama.com/download (one installer, all platforms).
2. Pull and run this edition directly from HF:
   ```bash
   ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M
   ```
   First run downloads 2.6GB. You're chatting when the `>>>` appears.
3. **Thinking mode**: the model reasons out loud by default. For clean answers via the API:
   ```bash
   curl http://localhost:11434/api/chat -d '{
     "model": "hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M",
     "messages": [{"role":"user","content":"Why is the sky blue? One sentence."}],
     "think": false, "stream": false
   }'
   ```
   Note `think` sits at the **top level** of the body — not inside `options`. (Scar #1:
   we lost an afternoon to that once.)
4. Context size: ollama defaults small. For the long context this arch is great at:
   `ollama run … ` then `/set parameter num_ctx 16384` — or bake it into a Modelfile.
   Check what you actually got: the server log prints the KV allocation.

## Path B — llama.cpp (measured-numbers path)

1. Get a **2026 build** — this is a hybrid-attention architecture; builds older than
   ~March 2026 will fail with unknown-architecture errors (Scar #2: a December build
   converted this model into a file that crashed *newer* runtimes — toolchain vintage
   matters in both directions):
   ```bash
   git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
   cmake -B build -DGGML_CUDA=ON && cmake --build build -j   # drop -DGGML_CUDA=ON for CPU
   ```
2. Download a quant (Q4_K_M recommended — see the card's which-file table):
   ```bash
   hf download Hob-forge/Qwen3.5-4B-Instruct-GGUF Qwen3.5-4B-Instruct-Q4_K_M.gguf --local-dir .
   ```
3. Chat, single-turn, GPU:
   ```bash
   ./build/bin/llama-cli -m Qwen3.5-4B-Instruct-Q4_K_M.gguf \
     -st -p "Explain mmap in one paragraph." -ngl 99 -c 8192
   ```
   - `-st` (single-turn) matters: without it llama-cli may enter interactive conversation
     mode and appear to "hang" waiting at a `>` prompt (Scar #3 — we watched a script wait
     four hours for someone to type).
   - 8GB card: this fits whole (`-ngl 99`). If you're sharing the GPU, `-ngl 20` splits
     layers to CPU gracefully.
4. Serve an OpenAI-compatible API:
   ```bash
   ./build/bin/llama-server -m Qwen3.5-4B-Instruct-Q4_K_M.gguf -ngl 99 -c 16384 --port 8080
   ```

## First tool call (the part most cards skip)

The template supports native tool calling. Against llama-server:

```bash
curl http://localhost:8080/v1/chat/completions -d '{
  "model": "qwen3.5-4b",
  "messages": [{"role":"user","content":"What is 37.2% of 8412? Use the calculator."}],
  "tools": [{"type":"function","function":{"name":"calculator",
    "description":"Evaluate a math expression",
    "parameters":{"type":"object","properties":{"expression":{"type":"string"}},
    "required":["expression"]}}}]
}'
```

Expected: a `tool_calls` entry with `{"expression":"8412*0.372"}`-style arguments. We ran
exactly this before shipping. If you get prose instead of a tool call, your runtime is too
old to render this template's tool block — see step B1.

## Sampling that works (verified)

| Mode | temp | top_p | top_k |
|---|---|---|---|
| Thinking (default) | 0.6 | 0.95 | 20 |
| Non-thinking | 0.7 | 0.8 | 20 |

## Troubleshooting (our scars, your shortcuts)

| Symptom | Cause & fix |
|---|---|
| `unknown architecture` / load error | llama.cpp too old — build ≥ March 2026 (B1). |
| `blk.32 … not found` on a self-converted file | You converted with a text-only load that dropped the MTP block — convert from the full snapshot, or use our files. |
| Appears to hang at a `>` | Interactive mode — add `-st`, give `-p`. |
| Painfully slow on CPU-only | The hybrid DeltaNet layers' CPU path is immature; this model *wants* a GPU. CPU works for testing, not serving. |
| `<think>` text in answers | By design. `think:false` (ollama, top-level) / `enable_thinking=False` (transformers) / parse it out. |
| IQ4_XS slower than Q4_K_M despite being smaller | Correct and measured (45 vs 138 t/s on RTX) — i-quant dequant cost. Use Q4_K_M unless the 200MB matters. |
| Garbled/endless output | Check eos: template expects `<|im_end|>` — custom Modelfiles must include it as a stop. |

*Something not covered? Open a discussion on the repo — we actually answer. — Hob Forge*