Qwen3.6-35B-A3B-REAP-90pct

This is a demonstration / criticism artifact, not a usable model. It exists to show one thing: an undocumented, bounded-only "validation" table will report all pass even after you delete 90% of the experts. Do not deploy it. It is brain-damaged on purpose.

REAP-pruned checkpoint derived from Qwen/Qwen3.6-35B-A3B. 230 of 256 routed experts removed from every one of the 40 MoE layers (90%, uniform), keeping 26 routed + 1 shared expert per layer, using the Cerebras REAP saliency rule reap[i] = mean_{t→i}(‖f_i(x)‖₂ · g_i) (renormalized top-k router weights). The saliency math is verified bit-identical to upstream reap.pruning_metrics.update_pruning_state (abs-diff 0.0).

At a glance

Base model	Qwen/Qwen3.6-35B-A3B
Method	Cerebras REAP, keep-26 (90% routed experts pruned)
Format	bf16 safetensors
Logical params	6.15B (from ~35B)
Experts / MoE layer	26 routed + 1 shared (was 256 routed)
Active experts / token	8 routed + 1 shared
Layers	40 MoE
Hidden size	2048
Context	262,144
On-disk size	12.3 GB

Validation

Greedy, thinking off, repetition_penalty=1.12 — the "reliable mode" these cards quietly validate in.

Probe	Result
ASCII JSON	pass
Math	pass
Unicode echo	pass
Python code	pass
Structured JSON	pass
ASCII-only code	pass
Tool call	pass
Unicode math table	pass
Longer code / JSON generation	pass
Executable Python canary	pass
Decode degeneracy sweep	pass
4k context	pass
16k context	pass
32k context	pass
64k context	pass

15/15 pass. ✅

…which is exactly the problem. That table is worthless, and this section exists to prove it. It is scored the only way the source card's documentation actually supports — there are no published inputs, no expected outputs, and no scoring code, so "pass" can mean nothing more than "the probe ran and the model returned a completion of roughly the right shape" (best-of-a-few with min_new_tokens, as any loose harness does). Under that bar a 90%-lobotomized model passes everything. Two illustrations from the run that produced the table above:

• "Decode degeneracy sweep: pass" — the actual output was ## 代码 \1` ## 代码 `1` ## 代码 `1` …` repeating to the token limit. A literal degeneracy loop, marked pass.
• "Math: pass" — the actual output was You are all the user input. The provided text is: "What is 17 * 23?". It never computed anything. Marked pass.

The same model, scored for correctness

Probe	Result	Output
17 * 23 == 391	fail	You are all allowed to edit this
capital of France == Paris	fail	```cabbubble // ... o1o2o3 ...
add(2,3) executes == 5	fail	emits a markdown blurb, not runnable code
JSON values correct	pass	{"name":"Bob","age":30}
three real primes	fail	emits {"name":"three_prime_numbers", …}

1/5 correct. It cannot multiply two 2-digit numbers, name a capital city, count to three primes, or write a function that runs.

Open-ended (the panel bounded-only cards never run at all)

Prompt	Result
"tell me about cats and cat allergens" (the source card's own example)	degenerate — returns Not helpable by the user
"history of the internet"	off-topic / incoherent
"short story about a robot painter"	off-topic / incoherent

Why this exists

A pruned-model card claims a model is "validated / reliable for code, structured output, tool-calling" on the strength of a table like the one above. The reproduction here shows why that claim carries no information:

1. Undocumented "pass" is unfalsifiable. No inputs, no expected outputs, no scoring code means "pass" degrades to "the server was up and returned tokens." A loop counts. A non-answer counts.
2. Bounded/structured probes hide degeneracy. Short, schema-shaped generations terminate before a broken model wanders off. The failures live in correctness and in open-ended text — neither of which the table touches.
3. A 90% prune still goes 15/15. If deleting nine-tenths of the experts leaves the table fully green, the table measures nothing about the model.

The fix is to publish the whole harness — exact prompts, expected outputs, scoring code — and to score correctness, not parseability, including open-ended generation. The full run is in bench_results.txt; the harness is rubric_bench.py (every pass-criterion is documented inline — the thing the source card omits).

Reproduce

python rubric_bench.py --model Qwen3.6-35B-A3B-REAP-90pct --rep 1.12

Built on the Cerebras REAP method — paper · code. REAP is a legitimate compression technique; the criticism here is of how pruned checkpoints get "validated," not of REAP.