Joakimpalm-Zen/Muse-Glimmer-30B-surgery-report

There Are No Experts Hiding in Your Dense Model, and What We Found Instead

Nine preregistered runs on one frozen 30B model, on one 24 GB GPU slice. The fashionable hypothesis died in three runs. What the autopsy taught us produced two laws, three measurement instruments that were lying to us, and two shipped artifacts.

---

1. The question, and the stakes

Muse-Glimmer-30B is a dense decoder: every token pays for every FFN channel in every layer. Mixture-of-experts models pay for a fraction. The obvious question, asked many times in many labs: is the dense model secretly sparse? Is there latent expert structure a runtime could route between, so a 30B pays like an 8B?

The stakes are local inference. FFN weights dominate the bytes a memory-bound decode must stream. If conditional structure exists and is coarse enough to skip whole quantized blocks, a commodity runtime gets a large speedup with no new hardware. We build such a runtime (xyntetik-runner), so every claim in this study had a customer waiting and a reason to be honest: we would have had to live with the answer.

House rules, fixed before run 1 and unchanged through run 9: every experiment preregisters its hypotheses, expectations and kill criteria in a commit BEFORE any result exists; corrections are recorded in place, never edited away; and the fidelity bar for any "the model is preserved" claim is margin-qualified top-1 agreement of at least 97% AND mean KLD of at most 0.05 against the frozen parent on a held-out split.

2. The preregistration ledger

The credibility of what follows is this table, not our prose. Every row was committed before its measurement.

preregistered	outcome
Clustering beats random partitions, or there is no structure	HELD, 4 layers, up to 3.9x
Router beats equal-FLOP static, or routing adds nothing	FAILED frozen, then root-caused: the objective, not the idea
Block-256 recovers >=70% of per-channel advantage	FAILED at every layer and ordering
C2: single-layer splice breaks token identity at 50%	WRONG in the good direction: 96.9% / 0.009
Full-model compounding within 2x of additive	WRONG: sub-linear, 0.26x at 52 layers
Operating point in 70-95% active	WRONG in the good direction: 40% passes
Block-oracle inherits absorption (H-cond)	WRONG: granularity, not conditioning, buys absorption
Router compounding sub-linear (R1)	WRONG: 1.65x super-linear; being RIGHT buys absorption
Healed frontier reaches 6-8% of decoder (E2)	HELD: 6.34%
Quant+surgery stack additively -> a predicted artifact (X1c)	HELD: predicted 0.0484, measured 0.0481
Trained conversion beats static per layer (G2)	HELD at all 13 layers
Data scaling rescues the 7.92% cut (D2)	FAILED by 0.26%, saturation measured

Six of these preregistrations were wrong. Every one is recorded with its correction beside it in the run reports. Three were wrong in the optimistic direction, which is precisely why preregistration matters: results this good, predicted this badly, would otherwise smell of selection.

3. Act one: the extraction hypothesis dies well (runs 1-3)

Run 1 captured activations across the 19,968-channel SwiGLU FFNs and asked whether co-activation structure exists. It does: clustered partitions beat equal-FLOP random partitions at every layer measured. But the decisive control was static pruning at equal FLOPs, and it beat even the GROUP ORACLE, because grouping destroys the 396x spike-shaped channel importance. The live discovery: a per-token per-channel oracle reaches cosine 0.994+ at HALF the channels. The redundancy is real and enormous. It is just not expert-shaped.

Run 2 measured the shape precisely. The granularity sweep: oracle recovery of the per-channel advantage is flat and near zero from block width 2496 all the way down to 64, then jumps to 100% at width one. Grouping even FOUR adjacent channels costs 5.6x. At full model scale the per-channel oracle passes our bar while skipping 65% of FFN reads; the block-oracle, which upper-bounds every possible block router, fails at every compliant fraction. The hardware-shaped container does not exist in frozen weights.

Run 3 killed the last hope, routing, and produced the study's sharpest sentence. Per-layer routers recover 46-59% of the oracle advantage; spliced through the model they recover about 10%, and their errors compound SUPER-linearly, like static pruning's, not sub-linearly, like the oracle's. Root cause: at ~20% chance-corrected selection accuracy a router's error is mostly systematic, and systematic error correlates across depth. Conditionality alone does not buy absorption; being right does.

Act one's summary, measured and final: extraction of MoE structure from this frozen dense model is dead at every granularity above per-channel, and per-channel is unroutable.

4. The compounding taxonomy

The arc's most general finding fell out of act one and hardened through act three. How per-layer errors combine at model scale depends on the KIND of error:

error type	measured / naive additive	behaviour
per-channel oracle (right, input-conditioned)	0.26x	sub-linear: the residual stream absorbs it
quantization noise x surgery error	0.97-1.00x	independent: they simply add
trained block modules (mostly right)	~1.65x	super-linear
routers at ~20% accuracy	1.65x	super-linear
static pruning (systematic)	2.2x	super-linear: avalanches

Two consequences we now treat as method:

Single-layer measurements are screens, never rankings. Run 2's "depth-varying static schedule" recommendation, made from single-layer data, was falsified at model scale by a factor of 79. Run 7's per-layer conversion wins of 2.2-30.3% bought one extra passing layer at model scale. The wall between local quality and model-scale viability moved 7 to 13 to 12-13 across three schemes of strictly rising sophistication. The compounding regime, not the per-layer number, decides.

Independence is the useful surprise. Quantization noise and surgery error are the first genuinely independent error pair this study measured: additive within 3.5% across five configurations, two surgery levels and two quantization types. Independence means DESIGN-BY-ARITHMETIC: we predicted a shipped artifact's fidelity to 0.6% before building it, and later correctly declined to build one the arithmetic said would fail.

5. Act two: surgery, and the absorbability law (runs 4-6)

If conditional structure cannot be exploited at runtime, remove capacity outright. Run 4 mapped the price of deleting each layer and each sublayer. Removable layers cluster in bands; no PAIR of whole layers survives; FFN sublayers are 2.4x more parameter-efficient to remove than attention. Free result, no retraining: FFN sublayers of two layers, 3.17% of decoder parameters, passing the bar. Run 5 added the honest asterisk: an apparent perplexity IMPROVEMENT from surgery was one domain masking a trade, and the claim was withdrawn in place.

Run 6 asked what repair buys. Recipe: for each removed FFN, train the first surviving FFN below it against the frozen parent. Cost: 2.5M tokens and ~10 minutes per site on the 24 GB slice. Healing cut divergence 3.3-4.6x, and the healed 4.75% cut beat the un-healed 3.17% standing result on BOTH bar terms. The frontier reached 6.34% of decoder parameters, and the next step failed, honestly bracketing the method.

Underneath sat a law. Measured across six repair sites spanning the network: absorbability falls monotonically with depth. A shallow FFN's function is ~77% recoverable by its neighbour; a deep one's, ~9%. Not a property of the cut's size; a property of its depth. It also explained run 6's exploratory falsification: consolidating three adjacent shallow FFNs into one heals BETTER than anything else measured (6.5x) and still loses, because clustered removal's starting damage grows faster than absorbability improves. Depth separation survives healing, now for a measured reason.

6. Act three: conversion works locally and walls globally (runs 7-8)

The frozen-weights verdict left one door: TRAINED reorganization. Run 7: replace an FFN with a hardware-shaped module (40% shared, 78 blocks of 256 channels, top-8 routed from the input before any weight read, 50% active), initialize from the layer's own weights, train briefly. It beats equal-FLOP static at all 13 layers tried, by 2.2-30.3%. And the router problem simply dissolves at this granularity: 0.60-0.71 chance-corrected overlap, within ~2% of its own partition's oracle ceiling, at 0.078% of the FFN's FLOPs.

Run 8 then closed the two levers left inside the design. Five partitions, including no-clustering and re-clustering at the cut width, land within 1.7% of each other: the partition is saturated. The hard depth band (L40-44) survives an equal-FLOP control at higher activity: it is a regime, not an artifact. And the decisive depth-adaptive splice, assembled by a rule fixed before measurement: 11 of 52 layers convertible under the bar, 10.6% of FFN reads saved. A real mechanism, and a better CONSTANT once more, never a better exponent. The track was closed by its own numbers, with both internal levers measured at their ceilings.

7. The data axis, measured and closed (run 9)

Healing trained on 135k unique tokens seen ~18 times, and its train loss diverged from validation by 10x. The obvious lever: more unique data. Run 9 isolated uniqueness at fixed compute: the memorization gap collapsed from 9.95x to 1.21x (the diagnosis was right) and the held-out gain was 5-11%, saturating by 4x the original corpus; 14.4x adds 1.9% more. The 7.92% frontier cut missed the bar by 0.26% and the arithmetic says data cannot close it. The lever is real, small, and now closed.

Run 9's larger contribution is methodological. Document disjointness is not decontamination. The extension corpus was disjoint from evaluation BY CONSTRUCTION, and still carried 11.5% of the eval split's 64-token windows, through near-duplicate source documents and repeated instruction preambles. A window-level audit caught it; enforcing it cost a domain and six sequences, dropped rather than weakening the filter. The final corpus overlaps the evaluation in zero windows. Every healed number in this study is clean because of that audit, and any study fine-tuning near its own evaluation should assume its corpus is contaminated until a window-level audit says otherwise.

8. Three instruments that were lying

Findings we now apply to every experiment, each discovered because a preregistered control disagreed with a plausible number:

1. Utilisation entropy certified a chance-level router. A maximally balanced router and a maximally uninformative one are indistinguishable on entropy. Report selection overlap against an oracle, never entropy.
2. Straight-through top-k reconstruction trained to chance where supervised BCE and rank-MSE on oracle labels worked at ~9x lower cost, in the extraction setting.
3. Aggregate perplexity hid a domain trade (run 5), and document-level dedup hid window-level contamination (run 9). Aggregates and coarse filters flatter; per-domain and per-window views tell the truth.

9. What shipped

Two artifacts, both passing the bar, both with byte-substitution provenance records and checksums, both written by the runner's own quantizer, both public on Hugging Face:

• 14.61 GB Q4_K, 4.75% of the decoder removed. Its fidelity was predicted from the independence law before the file existed: predicted 0.0484, measured 0.0481, bar 0.05.
• 27.58 GB Q8_0, 6.34% removed (healed at the measured-best data dose on the audited corpus): 0.03883 KLD, 98.34% margin-qualified top-1.

Plus the negative space, which we consider equally shipped: a frozen-extraction NO-GO ceiling-measured at every scale, a conversion wall triple-confirmed, a data axis with a measured saturation point, and six corrected preregistrations that keep all of the above honest.

10. Limits, and what would change our minds

One model family, one size, one corpus mix, one 24 GB GPU slice. The fidelity bar is parent-agreement, deliberately strict and deliberately narrow: it certifies that the smaller model behaves like the original, not that either is good at your task. Healing beyond rung one (LoRA over remaining layers, wider repair windows) is unmeasured. The conversion wall is a statement about ~2.5M-token-per-site training budgets; a serious upcycling run with real compute is a different question, and the laws above are the evidence a proposal for one would cite. N=52 behaviour of any trained-module scheme was never measured and is not implied anywhere in this document.

11. Post-publication addendum: we re-audited our own numbers (2026-08-29)

Section 7 reported that a window-level audit kept the run-9 extension corpus clean. Applying the same audit retroactively to the BASE healing corpus found what document-level construction had hidden: 283 of the eval split's 43,855 windows (0.65%, one domain) had always been present in healing data, undetected across five runs. The correction protocol in section 1 applied: closed reports stand, state files carry the correction, and every affected public number was re-measured.

All three artifacts pass on the decontaminated subset (Q4_K 0.04903 / 97.70%; Q8_0 0.03901 / 98.34%; BF16 frontier 0.03873 / 98.38%), and the published full-split numbers reproduce from the files on disk to five decimals. The honest sharpening: the Q4_K artifact is a thin-margin pass on every examined axis (clean headroom 1.37 times the noise floor); its siblings hold roughly 15 times.

The re-score also produced this study's fifth measurement-instrument finding, and its clearest: naive deltas can cancel; only a control decomposition tells the truth. The clean-minus-full deltas looked like noise (+0.0002 to +0.0009). They were two real effects with opposite signs: removing the leaked windows changed the domain mix (a hard sequence lost positions, lowering the mean by about 0.0012) while removing the leak itself raised it (+0.00122, +0.00140, +0.00138 across three independent artifacts, spread 0.00018). Healed models really do perform better on positions whose content they saw in healing data, by about 3.5% relative here. We predicted the deltas would sit inside the noise floor; that prediction was wrong, is recorded as wrong, and the control is the only reason we can tell.

We publish this not because any claim failed but because the audit that found it was pointed at our own successes. A bar this study asks others to trust has to be one we keep re-testing ourselves.

---

Reproducibility: every run preregistered its hypotheses and kill criteria in a commit before any result existed, and every correction is recorded in place; that run-by-run record is maintained in the project's research repository and is being prepared for publication. Both artifacts below carry byte-level provenance records and checksums. The runtime, its quantizer, and the determinism contract they operate under are public at github.com/Joakimpalm-Zen/xyntetik-runner.

The artifacts: Muse-Glimmer-30B-Surgical-Q4_K-GGUF (14.61 GB, 4.75% removed) and Muse-Glimmer-30B-Surgical-Q8_0-GGUF (27.58 GB, 6.34% removed).

12. The benchmark annex: we tested our own bar, and it failed (2026-08-29)

Section 10 said what would change our minds. One night of preregistered measurement did, on a question we had been treating as settled: does the house fidelity bar (margin-qualified top-1 >=97%, mean KLD <=0.05 versus the frozen parent) imply benchmark equivalence? We assumed a passing table would confirm it. Instead:

config x benchmark	preregistered margin	outcome	verdict
healed 4.75% BF16 x HellaSwag	±0.5 pts	−0.916, CI [−1.230, −0.583]	FAIL
healed 6.34% BF16 x HellaSwag	±0.5 pts	−1.524, CI [−1.872, −1.151]	FAIL
4.75% Q4_K x HellaSwag	±0.5 pts	not run	BLOCKED: no continuation-loglikelihood serving path
6.34% Q8_0 x any	—	not run	BLOCKED: 27.58 GB exceeds the 24 GB measurement GPU
any x MMLU-Pro	±0.5 pts	not run	BLOCKED (priced): 49.2 h/config
any x IFEval	±1.5 pts	stopped 182/541	BLOCKED (priced): 9.5 h/config

Method: lm-evaluation-harness 0.4.9, stock task configs (hashes in the run records), both models scored through the identical loglikelihood path on identical items, per-item pairing, exact binomial CIs on the discordant pairs, margins fixed before sampling. Determinism anchor: the parent scored twice over 200 documents produced bit-identical per-document loglikelihoods. Contamination audit: 1 item in 22,615, excluded.

What the two measured rows established:

1. A model that passes the fidelity bar comfortably (KLD 0.03056, margin-qualified top-1 98.64%) fails a ±0.5 pt paired equivalence test at full n. The bar and benchmark equivalence are different quantities. Nobody should infer one from the other on our cards, and after this measurement, nobody has to take our word for it in either direction.
2. The deficit is directional. On discordant items the surgical models lose roughly 2:1 (191/99 and 269/116). Symmetric noise would sit near even.
3. The KLD-to-benchmark mapping is monotone and super-linear: 30.0 points lost per unit KLD at 4.75% removed, 39.5 at 6.34%, established by a direct paired test between the two configs (−0.607, CI excludes zero), with a saturation check showing 17.99 points of headroom to any ceiling. Deeper surgery costs more per unit of measured divergence, so scaling surgery on fidelity numbers alone under-predicts the benchmark price.

Four preregistered expectations were recorded wrong in the run records, including the work order's own "all configs pass" and its "sub-linear" lean. The one lean that held predicted failure.

Both artifact cards now carry their benchmark row next to their fidelity table. The fidelity claims stand as measurements of what they measure: distribution tracking against the parent. The equivalence reading of them is withdrawn, because we tested it and it is false.