PollardWeights/Qwen2.5-14B-Instruct-Pollard

Qwen2.5-14B-Instruct — Pollard

Pollard shrank this model: 27.5 GB (f16) → 3.65 GB — 87% smaller, 7.5× down, and under half the size of NVFP4 (7.8 GB) or Q4_K_M (9 GB).

The 1-bit-class flagship (IQ1_KT, 2.12 bpw), still beating uniform 1-bit on every metric. Need more quality? The IQ3_S / IQ4_XS / Q6_K rungs are larger, higher-fidelity options.

format	this model's size
f16	27.5 GB
Q8_0	~15.7 GB
Q6_K	~12.2 GB
Q4_K_M / NVFP4	~9.0 / ~7.8 GB
PollardMix (this file)	3.65 GB

A memory-fit, mixed-precision build of Qwen2.5-14B-Instruct made with Pollard Weights. The expert/FFN body is crushed to 1-bit trellis (IQ1_KT); attention, the residual writers (ffn_down), embeddings, output and the first/last blocks are protected. The result sits in the 1-bit size class but reads much closer to 2-bit quality.

This is a standard GGUF — runs in stock llama.cpp / ik_llama.cpp, Ollama, and LM Studio.

The numbers (WikiText-2 raw, ctx 2048, 145 chunks)

build	PPL	size	bpw	Mean KLD	Median KLD	top-1
uniform IQ2_KT (2-bit ceiling)	6.92	4.30 GB	2.50	0.353	0.128	76.85%
PollardMix (this model)	8.27	3.65 GB	2.12	0.552	0.253	70.79%
uniform IQ1_KT (1-bit baseline)	9.65	3.37 GB	1.94	0.714	0.363	66.41%

PollardMix beats the uniform 1-bit trellis quant on every metric — PPL −14%, Mean KLD −23%, Median KLD −30%, top-1 +4.4 pts — at +8.5% size, and stays under the 2-bit ceiling. It also passes a chat-coherence gate (explanation, code, reasoning, creative continuation all coherent).

Allocation (the surgery)

tensor role	atom
expert / FFN body (gate, up)	IQ1_KT	crushed
attention k, v	IQ1_KT	crushed
attention q, output	IQ2_KT	protected
ffn_down (residual writer)	IQ2_KT	protected
first-2 / last-2 blocks	IQ2_KT	protected
token embeddings	Q4_K	kept
output head	Q6_K	kept
norms	F32	kept

Usage

# llama.cpp / ik_llama.cpp
llama-cli -m Qwen2.5-14B-Instruct-Pollard-IQ1_KT.gguf -p "Explain why the sky is blue." --temp 0.7 --repeat-penalty 1.15

# Ollama
ollama run hf.co/PollardWeights/Qwen2.5-14B-Instruct-Pollard

The *_KT trellis quants are an ik_llama.cpp format (QTIP-style); build ik_llama.cpp for best speed, though the file loads in stock llama.cpp too.

Errata (honest scope)

• Chat at this tier benefits from --repeat-penalty 1.15; stock sampling can loop.
• Quality win vs uniform 1-bit (better PPL/KL/top-1, +0.28 GB), not smaller-and-better; it is under the 2-bit ceiling, not equal to it.
• KL measured vs a Q8_0 reference (near-lossless; the 27.5 GB f16 exceeds a 16 GB card), consistent to within ~0.0005 of true f16.
• Single machine; replication invited.

Built with Pollard Weights — frontier models, small hardware, no compromise.