⚠️ NEEDS POST TRAINING. A research preview, published as the record of a

compression method and its measurements rather than as a finished assistant. Further healing and instruction tuning runs are planned over time, and these checkpoints will improve as those land.

Qwen3.8-Whittle-16B

🔬 RESEARCH PREVIEW. This model is a compression-research artifact, evaluated

with field measurements rather than academic benchmarks. Expect rough edges, use the serving settings below, and do not rely on it for factual reference or production systems.

🛑 NON-STARTER FOR AGENT AND LONG MULTI-TURN USE: repetition looping.

In extended generations, multi-turn conversations, and agent loops (coding assistants, tool use) the model can fall into repetition loops badly enough to be unusable, even with DRY sampling enabled. Single-turn use with the recommended serving flags works well. Further training rounds targeting looping, multi-turn, and agent formats are actively in progress; this card will be updated as fixes land and pass real-world testing.

✅ v2 WEIGHTS: use the -v2- GGUFs. v2 fixed code fencing (8/8 on the fencing

suite; v1 emitted code outside markdown blocks) and reduced single-turn long-form loops to 1/6 with mitigations disabled. Regression held: battery 35/39, recognition 5/7. v2 adds three.js fluency (trained on the official MIT examples + manual) at the cost of one A100 hour on a 2.2M-token clean mix; recipe and mix in training/. Serve with: --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 --repeat-penalty 1.15 --repeat-last-n 512. The v1 files remain for reproducibility.

A 27B whittled down to 16.8B with a logit lens and a pricing table, then healed with one A100 evening. It now outscores every intermediate cut, including ones 4B larger, at 20 tokens/second on two consumer 8GB GPUs.

Whittle is Qwen3.8-27B minus 20 of its 64 layers (whole interval-blocks, chosen by measured boundary cost) and minus 25% of every remaining MLP's width (weakest neurons by ‖down_column‖ × activation-std). That is a −10.1B parameter cut performed with zero training, followed by a single 11M-token QLoRA heal on a fully clean-lineage mix (synthetic fact frames, programmatic short-think arithmetic, hand-written code drills, public-domain long pages).

	params	file	battery (39 greedy tasks)	speed (RTX 4060 + 3050)
Qwen3.8-27B (base)	26.9B	n/a	not measurable on ref. hardware	n/a
48-layer cut	20.8B	12.9GB	33/39	5 t/s
un-repaired Whittle	16.8B	10.1GB	25/39	20.5 t/s
Whittle (healed, this)	16.8B	10.1GB	36/39	18.5–20.9 t/s

The heal fixed 11 battery items and broke zero. Long-tail recall recovered (recognition probe 2/7 → 5/7), arithmetic precedence and code completion returned to textbook form, and the boiling point of water, the fragile fact that failed every intermediate variant, answers correctly. Full measurement history, every pricing run, and all scripts: see the companion research repo Qwen3.8-p44w75-16.8B-unrepaired and its research/ folder. Both heal adapters live in this repo under adapters/ (v1-heal applies to the un-repaired base; v2-fencing-threejs applies on top of v1).

Running it

The GGUFs run on stock llama.cpp (any build with Qwen3.5-series support). Recommended serving command. The anti-loop sampling flags are part of the recipe, not optional garnish:

llama-server -m Qwen3.8-Whittle-16B-v2-Q4_K_M.gguf -ngl 99 -c 8192 --jinja \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 \
  --repeat-penalty 1.15 --repeat-last-n 512 \
  --temp 0.7 --top-p 0.95 --min-p 0.05

The Q4_K_M fits entirely in 16GB of VRAM (or 2×8GB split). This is a thinking model: it reasons in <think> before answering, so give it generous max_tokens (500+), and keep --jinja (included above) if your client uses tool calling. Context can be raised well past 8192 cheaply: only 11 of the 44 layers are full attention, the rest carry fixed-size recurrent state.

Known limitations

• Open-ended creative prompts (e.g. "write a haiku") can exhaust the thinking budget in deliberation. Concrete instructions ("exactly 3 short lines, no preamble") work. This is the one measured failure the heal did not close; it is targeted in the next round.
• Evaluated with a 39-prompt greedy battery plus probes, not academic benchmarks. The numbers above are honest field measurements, comparable within this table.
• The heal mix is small and targeted. Knowledge breadth beyond what a 16.8B carries is not magically restored: this is an efficient model, not a 27B in disguise.

How it was made (short version)

1. Sound the model: stream per-layer FP8 shards through an 8GB GPU, record each layer's identity cosine and a logit-lens sounding at every boundary.
2. Price the cuts: block-drops must keep the GDN:attention interval (GGUF expressibility); price every candidate at the boundary, then at task level. Findings: drop damage is non-additive, width damage compounds, and layers 32–35 hold arithmetic, not knowledge.
3. Cut: 20 layers + 25% of MLP width, zero training, single-width so stock tooling serves it.
4. Heal: QLoRA r=64 on every linear (GDN projections included), 110 steps, cosine annealed to completion, clean-lineage data only.

Full methodology in the research log (research/RESEARCH_activation_space_layer_merging.md).

Honest state of the model, and what it would take to finish it

Web development capability took real damage in the compression: current measurements show the shape cut collapsed targeted web-token retrieval roughly 28-fold at the boundary, and we are still isolating how much came from the depth cut versus the width prune (restoration pricing runs are in progress). The heals so far were single A100 hours and evenings; they recovered facts, arithmetic, and code fencing, but turning this research preview into an actually usable daily model (agent-capable, loop-free in long multi-turn use, restored web-dev depth) needs sustained post-training that is beyond a grocery-money compute budget. The methods and data are ready; the A100 hours are the missing ingredient.

Support this work

Independent research on consumer hardware. Every donation becomes A100 hours, and every A100 hour ends up as a public model or a public measurement. If you want the usable version of this model to exist, this is the lever. ☕ ko-fi.com/davida81328

Acknowledgements

Base model by the Qwen team (Apache 2.0). Whittled and healed by David Aylward with Claude (Fable 5, Anthropic) as co-author. The instruments, pricing runs, builds, training and evaluations were executed by Claude under David's direction, including several load-bearing ideas of David's: the parallel-composition merge operator, the slice-stack-merge width reduction, and the recognition-vs-recall damage probe.