--- license: apache-2.0 base_model: Qwen/Qwen3.5-9B library_name: gguf pipeline_tag: text-generation tags: [qwen3, coding, code, math, cybersecurity, reasoning, thinking, gguf, llama.cpp, local-llm] --- # ⚡ Qwen3.5-9B-Claude-4.8 (GGUF) — âœĻ ### 🧠 More accuracy, **20% less rambling** — a sharp local model for **code, math & general tasks** > **Runs on modest hardware.** With **~8-9 GB** of VRAM *or* unified memory free, you get a private, offline > reasoning assistant that **thinks less and lands more**. 🚀 > This is the **v1 edition** — tuned to reason efficiently, cut redundant chain-of-thought, and still hit the > correct final answer. All local, all yours. 💚 ### ðŸŽŊ What it is A focused fine-tune of **Qwen3.5-9B** with trace-inversed **CoT from Opus4.8** (Dataset not published anywhere) specialized for **coding, mathematics, and cybersecurity reasoning**. The headline trait: it reaches the goal with roughly **20% shorter thinking traces** than the base model while **maintaining final-answer accuracy** — less wandering, faster tokens-to-solution, lower latency. ðŸ§ âšĄ --- ## âœĻ Highlights - ðŸŠķ **~20% shorter reasoning** vs. base model, with accuracy held — tighter CoT, faster answers. - ðŸ’ŧ **Improved coding** — cleaner, more runnable solutions across common languages. - ðŸ§Ū **Strong math** — multi-step problems with the reasoning shown, then a clear final answer. - ðŸ›Ąïļ **Security-aware** — geared toward defensive concepts, code review, and CTF-style learning. - ðŸ“Ķ **One quant, well-tuned:** ships as **Q4_K_M** — the size/quality sweet spot. --- ## ðŸ“Ķ Download (GGUF quant) | Quant | Size | Vibe | |------|------|------| | ðŸ”ĩ **Q4_K_M** | **~5.6 GB** | the sweet spot 👌 (recommended — balanced quality & footprint) | > ðŸ’Ą Only Q4_K_M is published for now. Want another quant (Q5_K_M, Q8_0, f16)? Open a discussion and I'll consider it. --- ## ðŸ§Ū "Will it fit?" — context cheat-sheet Rough estimates ðŸĪ“ (assumes `q8_0` KV cache + ~1.5 GB overhead; switch to `q4_0` KV cache for ≈2× more context). | Your VRAM / unified mem | ðŸ”ĩ Q4_K_M (~8-9G) | |---|---| | **8 GB** | ~24K ctx | | **12 GB** | ~64K | | **16 GB** | ~100K | | **24 GB** | comfortable headroom | > ðŸ’Ą Apple Silicon / integrated GPUs with **unified memory** work too — same idea, just slower than a dGPU. --- ## 🚀 How to run it ### Option A — llama.cpp (recommended) ðŸĶ™ 1. Download `â€Ķ-Q4_K_M.gguf` and `llama-server` from [llama.cpp](https://github.com/ggml-org/llama.cpp). > ⚠ïļ Use a **recent llama.cpp** build for current Qwen3 architectures. 2. Run a server: \```bash llama-server \ -m ./qwen3.5-9b-reasoner-Q4_K_M.gguf \ --ctx-size 16384 \ --n-gpu-layers 99 \ -fa on \ --cache-type-k q8_0 --cache-type-v q8_0 \ --temp 0.7 --top-p 0.8 --top-k 20 \ --host 0.0.0.0 --port 8080 \``` 3. Connect to your agent and chat. 🎉 ### Option B — one-click apps ðŸ–ąïļ Works in **LM Studio**, **Jan**, **Ollama**, etc. — import the GGUF, pick the quant, go. ðŸū ### 🧠 Thinking mode This model reasons before answering. Keep thinking enabled (the default chat template handles it). Suggested sampling: `temp 0.7, top_p 0.95, top_k 20`. For deterministic code/math, try greedy (`temp 0`). --- ## ⚠ïļ Good to know - **Not safety-aligned for production.** This is a specialized reasoning fine-tune — add your own guardrails, input/output filtering, and review before any production or user-facing deployment. Use responsibly. 🙏 [Not uncensored] - Strongest in **code, math, and security reasoning**; double-check general-knowledge facts and figures. - English-centric. --- ## 🙏 Acknowledgements Special thanks to: - The Qwen team for the strong Qwen3.5 base model. - Unsloth for efficient fine-tuning frameworks. --- ## 📚 Base & License - **License:** Apache 2.0 - **Base model:** [`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B)