whittle-next — GGUF builds (research preview)

⚠️ Research artifact. A 19.8B qwen4exp-architecture model built by weight surgery on Whittle-tri-14.7B (Qwen3.8-27B depth-compressed 64 -> 32 layers by parallel-compose merging, FFN width uncut, Apache-2.0), then repaired by SFT. It now holds a conversation, follows the chat template, writes fenced code, and stops cleanly — but it is factually thin and its arithmetic is approximate. Treat it as an architecture demonstrator, not an assistant.

If this research is useful to you: ko-fi.com/davida81328 ☕

Serving settings — these are REQUIRED, not suggestions

llama-server -m whittle-next-qwen4exp-sft-PLE4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja \
  --override-kv qwen4exp.expert_used_count=int:104

Request body: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05.

Two settings do almost all the work, and both were measured on this build:

• Sampling, not greedy. At temperature 0 a paragraph-length answer degenerates (4-gram repetition 0.885 — "the ocean is a combination of water and water…"). At the settings above the same prompt scores 0.000 and ends with finish_reason=stop. Greedy decoding is the single largest cause of looping in this model.
• k=104 active experts (the --override-kv above). Raising k from the trained 58 to 104 is a config-only change that fixed list termination, restored task engagement (a "build a page" request went from a fabricated URL to real fenced HTML), and removed intra-list repetition — with zero gradient steps.

Reasoning is optional: pass chat_template_kwargs: {"enable_thinking": false} for short factual turns. With thinking on, allow ≥700 tokens — the think block is verbose.

The files

file	what it is	recommended
whittle-next-qwen4exp-sft-PLE4B-Q4_K_M.gguf	SFT + woken hyper-connections + trained shared-expert gates + 4B n-gram memory	✅ yes
whittle-next-qwen4exp-router104-PLE4B-Q4_K_M.gguf	the above plus jointly-trained routers at k=104 — better offline metrics, worse behaviour (see below)	experimental
whittle-next-qwen4exp-HC-Q4_K_M.gguf	hyper-connections only, no n-gram memory	ablation
whittle-next-qwen4exp-HC-PLE4B-f16.gguf	f16, n-gram memory, pre-SFT	ablation

Measured behaviour (identical 6-probe harness, k=104, ai-2 / 3×RTX 3060)

build	mode	clean stops	looping answers
SFT (recommended)	thinking off	5/6	1/6
SFT (recommended)	thinking on	4/6	1/6
router104	thinking off	4/6	2/6
router104	thinking on	2/6	— over-thinks, ran out of budget

With the recommended sampling the remaining loop disappears: longform, explanation, code and list probes all returned finish_reason=stop with 4-gram repetition 0.000 (one short story at 0.38).

Why router104 is not the default, despite better numbers. Training the routers jointly with the shared-expert gates, hyper-connections and n-gram projections — and at the k they serve — produced the best offline metrics this project has recorded (held-out CE 4.1466 → 3.9745, fact battery 4/5 → 5/5). But served, it over-thinks and repeats more. The training-time gate was selecting on cross-entropy and a short greedy battery, neither of which measures paragraph-length generation; repetition on that gate rose 0.057 → 0.093 over the same window while CE improved. The router result is real and reproducible — it is a training-objective lesson, not a serving win.

How conversational is it, really?

Mechanically conversational; not yet substantively reliable. It takes a turn, answers, and stops — and the content underneath is often wrong. Verified single-turn probes (recommended build, k=104, serving sampler): Paris ✅, a complete valid fenced HTML page ✅, a coherent non-repeating paragraph ✅ — against "the sky is blue because sunlight shines through the clouds" ❌, 17+25 = 32 ❌, and "list exactly 5 fruits" sometimes answered "1, 2, 3, 4, and 5 are fruits" ❌ (it hears the format and misses the substance).

Untested: every probe is single-turn. Multi-turn context retention — arguably the real test of "conversational" — has not been measured, and we make no claim about it.

The failure mode has moved from broken generation to a small model with damaged knowledge.

Honest limitations

• Facts are unreliable. "Why is the sky blue" gets a confident wrong answer. The 4B n-gram memory measurably helps in-domain perplexity (86.6 → 33.2 on wikitext-2 at ctx 512) but does not make the model knowledgeable.
• Arithmetic is approximate: 17 + 25 returns 32, 20 or 49 depending on build and mode. The model estimates where the parent computed.
• It will invent URLs and citations rather than decline. Do not trust any link it produces.
• Long code generations can still decay under greedy decoding; use the sampling settings.

Architecture

19.775B parameters, 32 layers × 5120, 3:1 GDN:full-attention, 240 experts (k=104 recommended, 58 trained), 4 hyper-connection residual streams, per-layer n-gram memory over a 6.25M-row × 640 table (≈4B parameters, host-offloadable). Requires a llama.cpp with qwen4exp support; the GGUFs declare output_gate_type: silu, which transformers' qwen4_exp config now supports natively — the Qwen3.5-derived GDN weights need a SiLU output gate, not the sigmoid a Flash-Next model uses.

David Aylward (logic65) & Claude (Anthropic) — designed, debugged and trained together.