manalejandro/Qwen2.5-Coder-1-5B-Instruct-f16-q4_k_m

Qwen2.5-Coder-1.5B-Instruct (Q4_K_M GGUF)

A compact GGUF quantization of Qwen/Qwen2.5-Coder-1.5B-Instruct — a code-focused 1.5B instruction model.

This card covers the single file:

• Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf

Property	Value
File	Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf
Size	940.37 MiB (986.05 MB)
Quantization	Q4_K_M (≈ 4.85 bits/weight)
Parameters	1.54B
Architecture	Qwen2
Context length	32768 tokens
GGUF version	3 (latest)

Why this quantization

Q4_K_M is the sweet spot for this model on low-VRAM hardware: ~0.94 GB of weights fit any 4 GB-class GPU (e.g. GTX 1650) with plenty of headroom for the KV cache. It keeps the model fluent for code generation, debugging and algorithm explanations while being a fraction of the F16 size (2.9 GB).

Unlike reasoning (R1-style) models, Qwen2.5-Coder answers directly — no mandatory chain-of-thought — so it is fast and responsive in interactive agents and never falls into a "thinking loop".

It was produced from the F16 source with llama.cpp, and the advertised context was capped at 32768 tokens so the KV cache fits a 4 GB GPU when fully offloaded (-ngl 999).

Usage

llama.cpp (CLI)

llama-cli \
  -m Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf \
  -p "Write a Python function to reverse a linked list." \
  -n 256 \
  -ngl 999 \
  -c 32768

llama-server (OpenAI-compatible API)

llama-server \
  -m Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf \
  -ngl 999 \
  -c 32768 \
  --port 8080
# curl http://localhost:8080/v1/chat/completions ...

Docker Model Runner (docker model)

docker model package \
  --gguf Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf \
  --context-size 32768 \
  qwen-coder
docker model run qwen-coder "Write a Python function to reverse a linked list."

Ollama

ollama create qwen-coder -f Modelfile   # FROM ./Qwen2.5-Coder-1.5B-Instruct-f16-q4_k_m.gguf
ollama run qwen-coder

Quality expectations

Q4_K_M on a 1.5B coder model handles common programming tasks well: function writing, debugging, algorithm explanations, and simple refactors. Larger or highly nuanced codebases may exceed its capacity (1.5B is a small model). Quality is noticeably higher than IQ-type quantizations of the same size; for the same VRAM footprint prefer Q4_K_M over Q4_0/IQ3_XXS.

Credits & license

• Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct (Apache 2.0)
• Source GGUF: bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF (MIT)
• Quantized with llama.cpp
• This file is released under the Apache 2.0 license.