---
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
base_model_relation: quantized
language:
- en
library_name: gguf
pipeline_tag: text-generation
tags:
- hob-forge
- edition
- small-gpu
- gguf
- qwen3.5
- llama.cpp
- ollama
- lm-studio
- imatrix
- iq4_xs
- q4_k_m
- q5_k_m
- q6_k
- q8_0
- 8gb
- cpu
- laptop
- instruct
- conversational
---
# Qwen3.5-4B-Instruct — Hob Forge Edition (GGUF)

**Qwen's excellent 4B, set up correctly for small GPUs and modest RAM — with everything
measured, nothing projected.** This is a Hob Forge *edition*: we didn't train this model
(all credit to [Qwen](https://huggingface.co/Qwen/Qwen3.5-4B), Apache-2.0); we quantized it
with a current toolchain, verified the chat template and tool-calling actually work,
measured speed / memory / quality-loss for every file, and wrote the run guide we wished
existed. If it's in a table below, we ran it on real hardware.

**The headline you won't find on other GGUF pages**: this is a hybrid-attention
architecture (8 full-attention layers + linear-attention DeltaNet + a 1-layer MTP block) —
its **KV cache is ~5× smaller than a dense 4B**. Measured: 8K context costs **256 MiB**;
32K costs **1 GiB**. Long context on tiny hardware is this model's superpower.

## Which file? (measured on RTX 5070, `-ngl 99`, llama.cpp b368b24c)

| File | Size | Gen speed tg128 | Perplexity (wikitext-2, 120 chunks) | Note |
|---|---|---|---|---|
| **Q4_K_M** ⭐ | 2.58 GiB | **138 t/s** | 10.030 ±0.157 | **recommended default** (imatrix) |
| Q5_K_M | 2.93 GiB | 131 t/s | 10.009 ±0.158 | quality step up |
| Q6_K | 3.31 GiB | 120 t/s | 9.887 ±0.155 | near-lossless |
| Q8_0 | 4.28 GiB | 102 t/s | 9.832 ±0.154 | reference quality |
| IQ4_XS | 2.39 GiB | **45 t/s** ⚠ | 10.094 ±0.159 | smallest, but i-quant dequant is ~3× slower on RTX cards — only pick this if the last 200MB matters more than speed |
| F16 | 8.42 GiB | — | 9.871 ±0.155 | conversion source |

F16 baseline shows the whole ladder loses almost nothing: Q4_K_M sits +0.16 PPL from F16,
inside ~1σ. imatrix (for Q4_K_M and IQ4_XS) computed from a 2MB wikitext-2-train slice,
250 chunks — the calibration file ships in this repo (`calib/`).

## Memory budget (measured, not projected)

Architecture-fixed overheads (identical for every quant): KV cache + 50 MiB recurrent-state
+ ~70–96 MiB compute buffer.

| Context | KV cache | Total @ Q4_K_M | Total @ Q8_0 | Fits 4GB? | Fits 8GB? |
|---|---|---|---|---|---|
| 4K | 128 MiB | ~2.8 GiB | ~4.5 GiB | ✅ Q4 | ✅ |
| 8K | 256 MiB | ~2.9 GiB | ~4.7 GiB | ✅ Q4 | ✅ |
| 16K | 512 MiB | ~3.2 GiB | ~4.9 GiB | ✅ Q4 | ✅ |
| 32K | 1 GiB | ~3.7 GiB | ~5.4 GiB | ⚠ tight | ✅ |

Yes — **32K context, whole model, under 4GB** at Q4_K_M. Every number from `llama.cpp`'s
own allocator logs on our hardware.

## Run it

```bash
# ollama
ollama run hf.co/Hob-forge/Qwen3.5-4B-Instruct-GGUF:Q4_K_M

# llama.cpp (recent build required — this arch needs 2026 support)
llama-cli -m Qwen3.5-4B-Instruct-Q4_K_M.gguf -st -p "your prompt" -ngl 99 -c 8192
```

**Sampling (Qwen's recommendations, verified here)**: thinking mode `temp 0.6, top_p 0.95,
top_k 20`; non-thinking `temp 0.7, top_p 0.8, top_k 20`. **This model thinks by default** —
it emits reasoning before answering. Turn it off: `think:false` (top-level, ollama API) /
`enable_thinking=False` (transformers) / strip `<think>` blocks client-side for llama.cpp.

Full walkthrough — zero-to-first-tool-call on an 8GB-class GPU and a standard desktop,
with the troubleshooting we earned building this — in [`RUNNING.md`](./RUNNING.md).

## Provenance & method

- Base: `Qwen/Qwen3.5-4B` (Apache-2.0) — untouched weights, full multimodal snapshot converted text-only.
- Toolchain: llama.cpp convert (build 2026-08-15) + quantize (build 2026-08-12). Chat template
  verified by rendering tests (system/no-system/tools × both); tool-calling exercised with a real call.
- Evals here measure **our files** (quantization quality), not the model's intelligence —
  for capability benchmarks see Qwen's card. PPL runs used identical chunks across all quants.
- No training data involved; nothing to decontaminate. No abliteration anywhere in lineage.

## Limits

A 4B is a 4B: strong for its size at chat, coding assistance, and tool use; not a frontier
model. IQ4_XS speed caveat above. Vision components of the base are not included (text-only
GGUFs). MTP block included in F16 but speculative decoding needs runtime support.

---
*Hob Forge — the small-GPU and modest-RAM champion. Measured budgets, honest tables, real
support. If something in this card doesn't reproduce on your machine, open a discussion —
we answer.*