Upload folder using huggingface_hub

README.md

@@ -0,0 +1,70 @@
+---
+license: mit
+base_model:
+- deepseek-ai/DeepSeek-V4-Flash-0731
+tags:
+- deepseek
+- deepseek-v4
+- abliterated
+- uncensored
+- heretic
+- fp8
+---
+
+# DeepSeek-V4-Flash-0731-BahamutRU-t265-abliterated
+
+`deepseek-ai/DeepSeek-V4-Flash-0731` with the refusal direction ablated, using the rank-1
+adapter from
+[`BahamutRU/DeepSeek-V4-Flash-0731-heretic-abliterated-v2-GGUF-lora`](https://huggingface.co/BahamutRU/DeepSeek-V4-Flash-0731-heretic-abliterated-v2-GGUF-lora)
+baked directly into the FP8 weights.
+
+**There is no LoRA at runtime.** The edit is merged into the tensors; load it like any other
+checkpoint.
+
+## What was changed
+
+57 target tensors (114 counting block scales):
+
+* `layers.{11..42}.attn.wo_b` — attention output projections
+* `layers.{18..42}.ffn.shared_experts.w2` — shared-expert down-projections
+
+Everything else — all 72,203 other indexed tensors, including every routed expert —
+is byte-identical to the official DeepSeek release. The 48 original shards are unmodified;
+the edited tensors live in a single overlay file that the index redirects to.
+
+## Method
+
+Directional ablation is inherently rank-1: `W <- W - lam * r (r^T W)` is an outer product, so
+the GGUF adapter is the edit in closed form. Verified against the base weights before
+applying — `cos(lora_a, r^T W)` is -0.995 / -0.999 / -0.943 at layers 30 / 20 / 42, with
+implied lambda 4.81 / 2.67 / 2.39. All 32 attention layers share one direction
+(min |cos| 0.994), matching the adapter's declared global direction scope.
+
+The delta is applied verbatim rather than re-derived as a projection, since the cosine is
+near but not exactly -1 — reproducing what the adapter author measured rather than an
+approximation of it.
+
+Dequantise FP8 e4m3 (ue8m0 128x128 block scales) -> add `lora_b (x) lora_a` -> requantise,
+raising a block's exponent where the larger delta would otherwise clip. Zero elements were
+clamped; peak overshoot 1.095x.
+
+Lambda ranges 0.72–4.88 across layers, peaking at layer 31. Above 2.0 the refusal component
+is inverted rather than merely removed.
+
+## Not included
+
+The adapter also ablates routed-expert down-projections (256 experts x 31 layers). Those are
+FP4 and were left untouched here, so this is a partial application of `t265`.
+
+## Credit
+
+* Base model: [`deepseek-ai/DeepSeek-V4-Flash-0731`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) (MIT)
+* Adapter: [`BahamutRU/DeepSeek-V4-Flash-0731-heretic-abliterated-v2-GGUF-lora`](https://huggingface.co/BahamutRU/DeepSeek-V4-Flash-0731-heretic-abliterated-v2-GGUF-lora) (MIT),
+  produced with heretic-gguf; reported 12 refusals on 140 harmful prompts at
+  KL 0.065 on harmless prompts
+* Ablation method: Heretic, by Philipp Emanuel Weidmann
+
+## Warning
+
+Refusal behaviour has been deliberately removed. This model will attempt whatever it is
+asked. Safety judgement is entirely the operator's responsibility.

bahamut_lora_report.json

@@ -0,0 +1,1127 @@
+{
+  "source_lora": "/tmp/claude-1000/-home-shaakz-spark-vllm-docker/61ff45c0-8219-4a7f-8e77-3a157b4e0796/scratchpad/lora/ds4-flash-heretic-f4-t265-lora.gguf",
+  "adapter_metadata": {
+    "general.architecture": "deepseek4",
+    "general.type": "adapter",
+    "adapter.type": "lora",
+    "adapter.lora.alpha": 1.0,
+    "adapter.heretic.study_name": "followup4",
+    "adapter.heretic.trial_number": 265.0,
+    "adapter.heretic.trial_index": 266.0,
+    "adapter.heretic.direction_scope": "global",
+    "adapter.heretic.direction_index": 20.5830999729783,
+    "adapter.heretic.row_normalization": "pre",
+    "adapter.heretic.expert_mode": "routing",
+    "adapter.heretic.components": "attn.o_proj,mlp.down_proj,mlp.down_proj.shared",
+    "adapter.heretic.seed": 46.0,
+    "adapter.heretic.llama_cpp_commit": "c5a4a0bb832fcdb44487996150b3141490fdff69",
+    "adapter.heretic.heretic_gguf_commit": "ec383b79288b30076043352418a757660353f462",
+    "adapter.heretic.score.keyword_rate": 0.08571428571428572,
+    "adapter.heretic.score.kl_divergence": 0.065054476087532,
+    "adapter.heretic.param.direction_index": 20.5830999729783,
+    "adapter.heretic.param.attn.o_proj.max_weight": 4.9612056333275385,
+    "adapter.heretic.param.attn.o_proj.max_weight_position": 30.620748194589943,
+    "adapter.heretic.param.attn.o_proj.min_weight": 0.1370855237172443,
+    "adapter.heretic.param.attn.o_proj.min_weight_distance": 19.83092882362658,
+    "adapter.heretic.param.mlp.down_proj.max_weight": 0.5468963693418676,
+    "adapter.heretic.param.mlp.down_proj.max_weight_position": 28.323665138976576,
+    "adapter.heretic.param.mlp.down_proj.min_weight": 0.6153807383029815,
+    "adapter.heretic.param.mlp.down_proj.min_weight_distance": 16.72124692757414,
+    "adapter.heretic.param.mlp.down_proj.shared.max_weight": 1.46427906043093,
+    "adapter.heretic.param.mlp.down_proj.shared.max_weight_position": 33.110047960113455,
+    "adapter.heretic.param.mlp.down_proj.shared.min_weight": 0.4630487032209349,
+    "adapter.heretic.param.mlp.down_proj.shared.min_weight_distance": 15.96458629664681
+  },
+  "overflow_policy": "bump",
+  "targets_edited": 57,
+  "tensors_edited": 114,
+  "tensors_total": 72317,
+  "routed_experts_skipped": 31,
+  "overlay_bytes": 1283547800,
+  "overlay_sha256": "eec6e22df119894f61ac6160b8d31baa2c160762bde14b208f3145f90e0ebe43",
+  "per_tensor": {
+    "layers.11.attn.wo_b": {
+      "bytes_changed": 6982746,
+      "bytes_total": 33554432,
+      "frac_changed": 0.2081020474433899,
+      "edit_frobenius": 1.6163755609591064,
+      "requant_noise_frobenius": 1.2469109658619615,
+      "noise_to_signal": 0.7714240402905398,
+      "rel_edit_size": 0.011494108418509737,
+      "blocks_total": 2048,
+      "blocks_overflowed": 180,
+      "blocks_exponent_bumped": 180,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0130713411739893,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": 0.5936989599920331,
+      "implied_lambda": 0.724756601547405,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 1.0000000000000009
+    },
+    "layers.12.attn.wo_b": {
+      "bytes_changed": 7765544,
+      "bytes_total": 33554432,
+      "frac_changed": 0.231431245803833,
+      "edit_frobenius": 2.3270999961065155,
+      "requant_noise_frobenius": 1.596174109532731,
+      "noise_to_signal": 0.6859069709953587,
+      "rel_edit_size": 0.016811390805902913,
+      "blocks_total": 2048,
+      "blocks_overflowed": 118,
+      "blocks_exponent_bumped": 118,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0161055667059762,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": 0.3406463799527072,
+      "implied_lambda": 0.9417943162503716,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9998341836443077
+    },
+    "layers.13.attn.wo_b": {
+      "bytes_changed": 9016661,
+      "bytes_total": 33554432,
+      "frac_changed": 0.2687174379825592,
+      "edit_frobenius": 2.754973578613904,
+      "requant_noise_frobenius": 1.7703267932866071,
+      "noise_to_signal": 0.6425930205026876,
+      "rel_edit_size": 0.0199190797880549,
+      "blocks_total": 2048,
+      "blocks_overflowed": 147,
+      "blocks_exponent_bumped": 147,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.035516279084342,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.12559338724405145,
+      "implied_lambda": 1.1567984354971084,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9998349518301644
+    },
+    "layers.14.attn.wo_b": {
+      "bytes_changed": 11059948,
+      "bytes_total": 33554432,
+      "frac_changed": 0.329612135887146,
+      "edit_frobenius": 3.659549376991329,
+      "requant_noise_frobenius": 2.101980022942672,
+      "noise_to_signal": 0.5743822002125297,
+      "rel_edit_size": 0.02537203130776057,
+      "blocks_total": 2048,
+      "blocks_overflowed": 205,
+      "blocks_exponent_bumped": 205,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0248443058558874,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.1383144387349788,
+      "implied_lambda": 1.3723043996983415,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9997344215951071
+    },
+    "layers.15.attn.wo_b": {
+      "bytes_changed": 11902718,
+      "bytes_total": 33554432,
+      "frac_changed": 0.354728639125824,
+      "edit_frobenius": 4.371804271454447,
+      "requant_noise_frobenius": 2.271675696376331,
+      "noise_to_signal": 0.5196197165571119,
+      "rel_edit_size": 0.030791614290519343,
+      "blocks_total": 2048,
+      "blocks_overflowed": 176,
+      "blocks_exponent_bumped": 176,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0262259500367301,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -0.3946642114701824,
+      "implied_lambda": 1.5893777896929666,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9998318373731446
+    },
+    "layers.16.attn.wo_b": {
+      "bytes_changed": 13212439,
+      "bytes_total": 33554432,
+      "frac_changed": 0.39376136660575867,
+      "edit_frobenius": 4.961044028720494,
+      "requant_noise_frobenius": 2.467762390825155,
+      "noise_to_signal": 0.4974280366267213,
+      "rel_edit_size": 0.03341503327022309,
+      "blocks_total": 2048,
+      "blocks_overflowed": 239,
+      "blocks_exponent_bumped": 239,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0479112352643694,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.6050254187301797,
+      "implied_lambda": 1.8016849224215368,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9995303432012204
+    },
+    "layers.17.attn.wo_b": {
+      "bytes_changed": 13890916,
+      "bytes_total": 33554432,
+      "frac_changed": 0.413981556892395,
+      "edit_frobenius": 5.926419106654851,
+      "requant_noise_frobenius": 2.6486595516894504,
+      "noise_to_signal": 0.4469241044250882,
+      "rel_edit_size": 0.03989043882641731,
+      "blocks_total": 2048,
+      "blocks_overflowed": 197,
+      "blocks_exponent_bumped": 197,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0408262865883964,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -0.8716704971876319,
+      "implied_lambda": 2.021738815377262,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9998367826062821
+    },
+    "layers.18.attn.wo_b": {
+      "bytes_changed": 14443066,
+      "bytes_total": 33554432,
+      "frac_changed": 0.430436909198761,
+      "edit_frobenius": 6.0275349654128325,
+      "requant_noise_frobenius": 2.6633320112939303,
+      "noise_to_signal": 0.4418608977926544,
+      "rel_edit_size": 0.040389539336487604,
+      "blocks_total": 2048,
+      "blocks_overflowed": 244,
+      "blocks_exponent_bumped": 244,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0493996483939034,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -1.0713661253726863,
+      "implied_lambda": 2.2367814791438216,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.999607697470402
+    },
+    "layers.18.ffn.shared_experts.w2": {
+      "bytes_changed": 1334434,
+      "bytes_total": 8388608,
+      "frac_changed": 0.15907692909240723,
+      "edit_frobenius": 0.8241208395489388,
+      "requant_noise_frobenius": 0.5199400573409119,
+      "noise_to_signal": 0.6309026933786647,
+      "rel_edit_size": 0.014591993823137351,
+      "blocks_total": 512,
+      "blocks_overflowed": 14,
+      "blocks_exponent_bumped": 14,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0278013774326868,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.47797319716419645,
+      "implied_lambda": 0.7163241962489749,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9986220860724971
+    },
+    "layers.19.attn.wo_b": {
+      "bytes_changed": 15422841,
+      "bytes_total": 33554432,
+      "frac_changed": 0.45963647961616516,
+      "edit_frobenius": 7.19128664658266,
+      "requant_noise_frobenius": 2.8132897525848297,
+      "noise_to_signal": 0.3912081232253092,
+      "rel_edit_size": 0.04866233628446717,
+      "blocks_total": 2048,
+      "blocks_overflowed": 213,
+      "blocks_exponent_bumped": 213,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0446210759026664,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -1.3320893590381089,
+      "implied_lambda": 2.4524767350778425,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9998181636056176
+    },
+    "layers.19.ffn.shared_experts.w2": {
+      "bytes_changed": 1060290,
+      "bytes_total": 8388608,
+      "frac_changed": 0.12639641761779785,
+      "edit_frobenius": 0.62588112100712,
+      "requant_noise_frobenius": 0.48074474506811005,
+      "noise_to_signal": 0.7681087173464066,
+      "rel_edit_size": 0.011060982554224514,
+      "blocks_total": 512,
+      "blocks_overflowed": 4,
+      "blocks_exponent_bumped": 4,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0013445727527142,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": 0.5675623740573843,
+      "implied_lambda": 0.7672812262657969,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9994336740716218
+    },
+    "layers.20.attn.wo_b": {
+      "bytes_changed": 16220930,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4834213852882385,
+      "edit_frobenius": 7.591125406607625,
+      "requant_noise_frobenius": 2.902955209792607,
+      "noise_to_signal": 0.38241433967956273,
+      "rel_edit_size": 0.050601827951904904,
+      "blocks_total": 2048,
+      "blocks_overflowed": 261,
+      "blocks_exponent_bumped": 261,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0591952800750732,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -1.5480655394481093,
+      "implied_lambda": 2.6679563577222725,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9997642795598237
+    },
+    "layers.20.ffn.shared_experts.w2": {
+      "bytes_changed": 1523755,
+      "bytes_total": 8388608,
+      "frac_changed": 0.18164575099945068,
+      "edit_frobenius": 1.0629658877536579,
+      "requant_noise_frobenius": 0.5918994589307581,
+      "noise_to_signal": 0.5568376800704358,
+      "rel_edit_size": 0.01851909099207693,
+      "blocks_total": 512,
+      "blocks_overflowed": 13,
+      "blocks_exponent_bumped": 13,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0062717710222517,
+      "bytes_sign_normalized": 3,
+      "direction_signed_gain": 0.3658628727064897,
+      "implied_lambda": 0.804729303963677,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9980637254804885
+    },
+    "layers.21.attn.wo_b": {
+      "bytes_changed": 15709437,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4681777060031891,
+      "edit_frobenius": 7.43775394575627,
+      "requant_noise_frobenius": 2.7959129188846585,
+      "noise_to_signal": 0.37590822972571064,
+      "rel_edit_size": 0.05160121561504187,
+      "blocks_total": 2048,
+      "blocks_overflowed": 201,
+      "blocks_exponent_bumped": 201,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0918829781668526,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -1.7601038536334308,
+      "implied_lambda": 2.8859418519404723,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9997532398271372
+    },
+    "layers.21.ffn.shared_experts.w2": {
+      "bytes_changed": 1702164,
+      "bytes_total": 8388608,
+      "frac_changed": 0.20291376113891602,
+      "edit_frobenius": 1.0798928489893282,
+      "requant_noise_frobenius": 0.6026427243525119,
+      "noise_to_signal": 0.5580578896475935,
+      "rel_edit_size": 0.019104689568558646,
+      "blocks_total": 512,
+      "blocks_overflowed": 22,
+      "blocks_exponent_bumped": 22,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.012504254068647,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.30598970224370126,
+      "implied_lambda": 0.8647336589449667,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9987977179365468
+    },
+    "layers.22.attn.wo_b": {
+      "bytes_changed": 15096310,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4499050974845886,
+      "edit_frobenius": 6.611610241388012,
+      "requant_noise_frobenius": 2.765308971364292,
+      "noise_to_signal": 0.4182504519177095,
+      "rel_edit_size": 0.044475373620491275,
+      "blocks_total": 2048,
+      "blocks_overflowed": 230,
+      "blocks_exponent_bumped": 230,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0525879859924316,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -1.9122114275967985,
+      "implied_lambda": 3.098075222558122,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9997506822710533
+    },
+    "layers.22.ffn.shared_experts.w2": {
+      "bytes_changed": 1667020,
+      "bytes_total": 8388608,
+      "frac_changed": 0.19872426986694336,
+      "edit_frobenius": 1.0147245666255624,
+      "requant_noise_frobenius": 0.6213595935532713,
+      "noise_to_signal": 0.6123431066812395,
+      "rel_edit_size": 0.017888060596338094,
+      "blocks_total": 512,
+      "blocks_overflowed": 15,
+      "blocks_exponent_bumped": 15,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0164795688220434,
+      "bytes_sign_normalized": 3,
+      "direction_signed_gain": 0.31890764853556336,
+      "implied_lambda": 0.9052052081283634,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9977350397895159
+    },
+    "layers.23.attn.wo_b": {
+      "bytes_changed": 16531540,
+      "bytes_total": 33554432,
+      "frac_changed": 0.49267828464508057,
+      "edit_frobenius": 7.688841910319432,
+      "requant_noise_frobenius": 2.9005189819643236,
+      "noise_to_signal": 0.3772374326062612,
+      "rel_edit_size": 0.051605174281738886,
+      "blocks_total": 2048,
+      "blocks_overflowed": 278,
+      "blocks_exponent_bumped": 278,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0442489555903844,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.171193818528343,
+      "implied_lambda": 3.314938790943368,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.999791075159068
+    },
+    "layers.23.ffn.shared_experts.w2": {
+      "bytes_changed": 1771583,
+      "bytes_total": 8388608,
+      "frac_changed": 0.2111891508102417,
+      "edit_frobenius": 1.017111135227058,
+      "requant_noise_frobenius": 0.5815432778102404,
+      "noise_to_signal": 0.5717598182428881,
+      "rel_edit_size": 0.02092547285287721,
+      "blocks_total": 512,
+      "blocks_overflowed": 9,
+      "blocks_exponent_bumped": 9,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.00869848046984,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": 0.217160837264009,
+      "implied_lambda": 0.9664482830144527,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9990637433091791
+    },
+    "layers.24.attn.wo_b": {
+      "bytes_changed": 16399499,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4887431561946869,
+      "edit_frobenius": 8.751616259544965,
+      "requant_noise_frobenius": 3.1067181556985473,
+      "noise_to_signal": 0.35498793177890997,
+      "rel_edit_size": 0.05633599048758558,
+      "blocks_total": 2048,
+      "blocks_overflowed": 199,
+      "blocks_exponent_bumped": 199,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0667373112269811,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.4033631937324946,
+      "implied_lambda": 3.529953159899262,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9997401958304736
+    },
+    "layers.24.ffn.shared_experts.w2": {
+      "bytes_changed": 1861932,
+      "bytes_total": 8388608,
+      "frac_changed": 0.22195959091186523,
+      "edit_frobenius": 1.0585071776150932,
+      "requant_noise_frobenius": 0.5999802179832215,
+      "noise_to_signal": 0.5668173354620306,
+      "rel_edit_size": 0.022343562478275125,
+      "blocks_total": 512,
+      "blocks_overflowed": 9,
+      "blocks_exponent_bumped": 9,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0081126179013933,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.16909087339523166,
+      "implied_lambda": 1.0143733044509409,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9990484854977735
+    },
+    "layers.25.attn.wo_b": {
+      "bytes_changed": 16639629,
+      "bytes_total": 33554432,
+      "frac_changed": 0.49589958786964417,
+      "edit_frobenius": 8.17998352004807,
+      "requant_noise_frobenius": 3.0388481166680195,
+      "noise_to_signal": 0.3714980732198543,
+      "rel_edit_size": 0.052761573194127886,
+      "blocks_total": 2048,
+      "blocks_overflowed": 270,
+      "blocks_exponent_bumped": 270,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0612385613577706,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -2.5782994699331923,
+      "implied_lambda": 3.7433026178746656,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.999446934854876
+    },
+    "layers.25.ffn.shared_experts.w2": {
+      "bytes_changed": 1910309,
+      "bytes_total": 8388608,
+      "frac_changed": 0.22772657871246338,
+      "edit_frobenius": 1.0332831536175129,
+      "requant_noise_frobenius": 0.5591037460409136,
+      "noise_to_signal": 0.5410944174242052,
+      "rel_edit_size": 0.024478900709787373,
+      "blocks_total": 512,
+      "blocks_overflowed": 3,
+      "blocks_exponent_bumped": 3,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0020909681916237,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.10612396865541897,
+      "implied_lambda": 1.062380373247505,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9991408652316012
+    },
+    "layers.26.attn.wo_b": {
+      "bytes_changed": 16677848,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4970386028289795,
+      "edit_frobenius": 9.212484160290288,
+      "requant_noise_frobenius": 2.7923703855222315,
+      "noise_to_signal": 0.30310721157693077,
+      "rel_edit_size": 0.06939004068065988,
+      "blocks_total": 2048,
+      "blocks_overflowed": 62,
+      "blocks_exponent_bumped": 62,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0326173646109444,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.874205518175861,
+      "implied_lambda": 3.962686152296236,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9996972474731345
+    },
+    "layers.26.ffn.shared_experts.w2": {
+      "bytes_changed": 1938131,
+      "bytes_total": 8388608,
+      "frac_changed": 0.23104321956634521,
+      "edit_frobenius": 1.145319095654184,
+      "requant_noise_frobenius": 0.576963831336244,
+      "noise_to_signal": 0.5037581522262958,
+      "rel_edit_size": 0.026706828529869598,
+      "blocks_total": 512,
+      "blocks_overflowed": 1,
+      "blocks_exponent_bumped": 1,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0026224383286066,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.0419321168413412,
+      "implied_lambda": 1.1067772674927863,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9991041233757195
+    },
+    "layers.27.attn.wo_b": {
+      "bytes_changed": 17980848,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5358710289001465,
+      "edit_frobenius": 9.641388941545667,
+      "requant_noise_frobenius": 3.144685039846803,
+      "noise_to_signal": 0.3261651468385488,
+      "rel_edit_size": 0.06276439165376453,
+      "blocks_total": 2048,
+      "blocks_overflowed": 303,
+      "blocks_exponent_bumped": 303,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0709215913500105,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -3.0639117055878162,
+      "implied_lambda": 4.178888966678869,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9996705921303283
+    },
+    "layers.27.ffn.shared_experts.w2": {
+      "bytes_changed": 2003530,
+      "bytes_total": 8388608,
+      "frac_changed": 0.23883938789367676,
+      "edit_frobenius": 1.1268074261257592,
+      "requant_noise_frobenius": 0.5898487144706669,
+      "noise_to_signal": 0.5234689626591401,
+      "rel_edit_size": 0.02608851718842635,
+      "blocks_total": 512,
+      "blocks_overflowed": 3,
+      "blocks_exponent_bumped": 3,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.001253630965948,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": 0.0045473728287045295,
+      "implied_lambda": 1.1616479068517933,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9989738546733914
+    },
+    "layers.28.attn.wo_b": {
+      "bytes_changed": 18709236,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5575786828994751,
+      "edit_frobenius": 11.604410212123584,
+      "requant_noise_frobenius": 3.1742743925496497,
+      "noise_to_signal": 0.27354034668934407,
+      "rel_edit_size": 0.07894580028132334,
+      "blocks_total": 2048,
+      "blocks_overflowed": 191,
+      "blocks_exponent_bumped": 191,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0716850757598877,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -3.308199654514721,
+      "implied_lambda": 4.387676929820198,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9995008240918627
+    },
+    "layers.28.ffn.shared_experts.w2": {
+      "bytes_changed": 2062998,
+      "bytes_total": 8388608,
+      "frac_changed": 0.24592852592468262,
+      "edit_frobenius": 1.1736386857069288,
+      "requant_noise_frobenius": 0.59395056568626,
+      "noise_to_signal": 0.5060761654499317,
+      "rel_edit_size": 0.027600593808088898,
+      "blocks_total": 512,
+      "blocks_overflowed": 2,
+      "blocks_exponent_bumped": 2,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.000062923545816,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -0.04928820992556719,
+      "implied_lambda": 1.2081826807296816,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9987562671393035
+    },
+    "layers.29.attn.wo_b": {
+      "bytes_changed": 19242117,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5734597742557526,
+      "edit_frobenius": 11.8765645210171,
+      "requant_noise_frobenius": 3.197694867333839,
+      "noise_to_signal": 0.2692440950979645,
+      "rel_edit_size": 0.08076509761721308,
+      "blocks_total": 2048,
+      "blocks_overflowed": 240,
+      "blocks_exponent_bumped": 240,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.089263984135219,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -3.537298999542481,
+      "implied_lambda": 4.607599209093972,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.999715749320466
+    },
+    "layers.29.ffn.shared_experts.w2": {
+      "bytes_changed": 2125974,
+      "bytes_total": 8388608,
+      "frac_changed": 0.2534358501434326,
+      "edit_frobenius": 1.243760202306456,
+      "requant_noise_frobenius": 0.5978597807735655,
+      "noise_to_signal": 0.48068733801329333,
+      "rel_edit_size": 0.02875591374912105,
+      "blocks_total": 512,
+      "blocks_overflowed": 7,
+      "blocks_exponent_bumped": 7,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.008077425616128,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.09414836159415675,
+      "implied_lambda": 1.25095397326936,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.997931073766525
+    },
+    "layers.30.attn.wo_b": {
+      "bytes_changed": 18191560,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5421507358551025,
+      "edit_frobenius": 11.246904531894943,
+      "requant_noise_frobenius": 3.2758744813187266,
+      "noise_to_signal": 0.2912689862378327,
+      "rel_edit_size": 0.0728666446031667,
+      "blocks_total": 2048,
+      "blocks_overflowed": 205,
+      "blocks_exponent_bumped": 205,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0591203144618444,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -3.6933408719121466,
+      "implied_lambda": 4.80816872738677,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9993289469068709
+    },
+    "layers.30.ffn.shared_experts.w2": {
+      "bytes_changed": 2113132,
+      "bytes_total": 8388608,
+      "frac_changed": 0.2519049644470215,
+      "edit_frobenius": 1.25190610545891,
+      "requant_noise_frobenius": 0.6198610078290615,
+      "noise_to_signal": 0.4951337844956349,
+      "rel_edit_size": 0.028284831069709852,
+      "blocks_total": 512,
+      "blocks_overflowed": 5,
+      "blocks_exponent_bumped": 5,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0018477945455484,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -0.08861118174286227,
+      "implied_lambda": 1.2815997806413477,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9967686906069841
+    },
+    "layers.31.attn.wo_b": {
+      "bytes_changed": 17487524,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5211688280105591,
+      "edit_frobenius": 10.279497205060528,
+      "requant_noise_frobenius": 2.9977682164737365,
+      "noise_to_signal": 0.29162595763905214,
+      "rel_edit_size": 0.07272491564681303,
+      "blocks_total": 2048,
+      "blocks_overflowed": 113,
+      "blocks_exponent_bumped": 113,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0721096992492676,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -3.7822024960370055,
+      "implied_lambda": 4.876822822356775,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9997523575898866
+    },
+    "layers.31.ffn.shared_experts.w2": {
+      "bytes_changed": 1987596,
+      "bytes_total": 8388608,
+      "frac_changed": 0.2369399070739746,
+      "edit_frobenius": 1.0323141189331566,
+      "requant_noise_frobenius": 0.619633452554879,
+      "noise_to_signal": 0.6002373126459204,
+      "rel_edit_size": 0.022100061872461392,
+      "blocks_total": 512,
+      "blocks_overflowed": 8,
+      "blocks_exponent_bumped": 8,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0098342980657304,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.02811960592523809,
+      "implied_lambda": 1.3259823254545386,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9975144731503831
+    },
+    "layers.32.attn.wo_b": {
+      "bytes_changed": 17088662,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5092818140983582,
+      "edit_frobenius": 9.743917498597328,
+      "requant_noise_frobenius": 2.8489450250370107,
+      "noise_to_signal": 0.29238189110767066,
+      "rel_edit_size": 0.07251056854802733,
+      "blocks_total": 2048,
+      "blocks_overflowed": 77,
+      "blocks_exponent_bumped": 77,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.082754135131836,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -3.5571029352285684,
+      "implied_lambda": 4.655419855295536,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9996174827689057
+    },
+    "layers.32.ffn.shared_experts.w2": {
+      "bytes_changed": 2072727,
+      "bytes_total": 8388608,
+      "frac_changed": 0.24708831310272217,
+      "edit_frobenius": 1.1828366099933507,
+      "requant_noise_frobenius": 0.6189560723830073,
+      "noise_to_signal": 0.5232811253504293,
+      "rel_edit_size": 0.02708144574519221,
+      "blocks_total": 512,
+      "blocks_overflowed": 1,
+      "blocks_exponent_bumped": 1,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0004083407776696,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -0.18452515261595698,
+      "implied_lambda": 1.401764792405694,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9982342027080382
+    },
+    "layers.33.attn.wo_b": {
+      "bytes_changed": 18975033,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5655000507831573,
+      "edit_frobenius": 11.01691400935028,
+      "requant_noise_frobenius": 3.1886538824339423,
+      "noise_to_signal": 0.28943258336478495,
+      "rel_edit_size": 0.07411354561753018,
+      "blocks_total": 2048,
+      "blocks_overflowed": 266,
+      "blocks_exponent_bumped": 266,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.09482707296099,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -3.349426292095493,
+      "implied_lambda": 4.439213200635072,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9995793415264033
+    },
+    "layers.33.ffn.shared_experts.w2": {
+      "bytes_changed": 2046619,
+      "bytes_total": 8388608,
+      "frac_changed": 0.24397599697113037,
+      "edit_frobenius": 1.1072572703193044,
+      "requant_noise_frobenius": 0.5981928829222891,
+      "noise_to_signal": 0.5402474194184209,
+      "rel_edit_size": 0.02545208206191668,
+      "blocks_total": 512,
+      "blocks_overflowed": 4,
+      "blocks_exponent_bumped": 4,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.002774067223072,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.15461250924880499,
+      "implied_lambda": 1.4121771904978886,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9976203536499172
+    },
+    "layers.34.attn.wo_b": {
+      "bytes_changed": 17137795,
+      "bytes_total": 33554432,
+      "frac_changed": 0.5107460916042328,
+      "edit_frobenius": 9.436438454278544,
+      "requant_noise_frobenius": 2.8855276082493715,
+      "noise_to_signal": 0.3057856650292732,
+      "rel_edit_size": 0.06856185523901816,
+      "blocks_total": 2048,
+      "blocks_overflowed": 127,
+      "blocks_exponent_bumped": 127,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0377494948250907,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -3.1245768608489612,
+      "implied_lambda": 4.224978169866536,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9996257867934409
+    },
+    "layers.34.ffn.shared_experts.w2": {
+      "bytes_changed": 2089934,
+      "bytes_total": 8388608,
+      "frac_changed": 0.24913954734802246,
+      "edit_frobenius": 1.1609943460781942,
+      "requant_noise_frobenius": 0.6459668555313915,
+      "noise_to_signal": 0.5563910433443963,
+      "rel_edit_size": 0.025378269745363797,
+      "blocks_total": 512,
+      "blocks_overflowed": 3,
+      "blocks_exponent_bumped": 3,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0106481143406458,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.1695215919817807,
+      "implied_lambda": 1.4164211037847874,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9984429686528752
+    },
+    "layers.35.attn.wo_b": {
+      "bytes_changed": 17389942,
+      "bytes_total": 33554432,
+      "frac_changed": 0.518260657787323,
+      "edit_frobenius": 10.199916029722235,
+      "requant_noise_frobenius": 2.9108759677965494,
+      "noise_to_signal": 0.2853823462187677,
+      "rel_edit_size": 0.07514085396713704,
+      "blocks_total": 2048,
+      "blocks_overflowed": 71,
+      "blocks_exponent_bumped": 71,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0509160246167863,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.938894361582444,
+      "implied_lambda": 4.012171182185959,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9996858395340702
+    },
+    "layers.35.ffn.shared_experts.w2": {
+      "bytes_changed": 2075671,
+      "bytes_total": 8388608,
+      "frac_changed": 0.24743926525115967,
+      "edit_frobenius": 1.1526127646737774,
+      "requant_noise_frobenius": 0.6952605737796049,
+      "noise_to_signal": 0.60320395113478,
+      "rel_edit_size": 0.021978175727267114,
+      "blocks_total": 512,
+      "blocks_overflowed": 21,
+      "blocks_exponent_bumped": 21,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0186630657741003,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -0.07372654308020132,
+      "implied_lambda": 1.3619306791926444,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9982638748979263
+    },
+    "layers.36.attn.wo_b": {
+      "bytes_changed": 16620711,
+      "bytes_total": 33554432,
+      "frac_changed": 0.49533578753471375,
+      "edit_frobenius": 9.092804062023685,
+      "requant_noise_frobenius": 3.1336922029103693,
+      "noise_to_signal": 0.3446343043944288,
+      "rel_edit_size": 0.05855878548016699,
+      "blocks_total": 2048,
+      "blocks_overflowed": 192,
+      "blocks_exponent_bumped": 192,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0553676060267858,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -2.63177415001905,
+      "implied_lambda": 3.774708620782787,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.998884323510219
+    },
+    "layers.36.ffn.shared_experts.w2": {
+      "bytes_changed": 2077455,
+      "bytes_total": 8388608,
+      "frac_changed": 0.24765193462371826,
+      "edit_frobenius": 1.190315472487199,
+      "requant_noise_frobenius": 0.7017125153802115,
+      "noise_to_signal": 0.5895180996966819,
+      "rel_edit_size": 0.02286564176353482,
+      "blocks_total": 512,
+      "blocks_overflowed": 14,
+      "blocks_exponent_bumped": 14,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0096035259110587,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -0.062177539415835065,
+      "implied_lambda": 1.3188187795310393,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9988857976909455
+    },
+    "layers.37.attn.wo_b": {
+      "bytes_changed": 15029492,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4479137659072876,
+      "edit_frobenius": 7.325423660881461,
+      "requant_noise_frobenius": 2.6164905309759066,
+      "noise_to_signal": 0.3571794140655978,
+      "rel_edit_size": 0.05579046085221563,
+      "blocks_total": 2048,
+      "blocks_overflowed": 62,
+      "blocks_exponent_bumped": 62,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.036022731236049,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.4469363433573337,
+      "implied_lambda": 3.5819199534220703,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9994718382990545
+    },
+    "layers.37.ffn.shared_experts.w2": {
+      "bytes_changed": 1958545,
+      "bytes_total": 8388608,
+      "frac_changed": 0.23347675800323486,
+      "edit_frobenius": 1.063049618488462,
+      "requant_noise_frobenius": 0.6232969306183691,
+      "noise_to_signal": 0.5863291042845469,
+      "rel_edit_size": 0.022246377085073054,
+      "blocks_total": 512,
+      "blocks_overflowed": 10,
+      "blocks_exponent_bumped": 10,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.006502091884613,
+      "bytes_sign_normalized": 2,
+      "direction_signed_gain": 0.009712036884405967,
+      "implied_lambda": 1.2575035831127501,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9969125769396714
+    },
+    "layers.38.attn.wo_b": {
+      "bytes_changed": 14809092,
+      "bytes_total": 33554432,
+      "frac_changed": 0.44134533405303955,
+      "edit_frobenius": 7.485419318953432,
+      "requant_noise_frobenius": 2.836590856775567,
+      "noise_to_signal": 0.3789488251637669,
+      "rel_edit_size": 0.051394315685424144,
+      "blocks_total": 2048,
+      "blocks_overflowed": 108,
+      "blocks_exponent_bumped": 108,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.052781241280692,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.194864443484076,
+      "implied_lambda": 3.353864813996831,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9991849822972525
+    },
+    "layers.38.ffn.shared_experts.w2": {
+      "bytes_changed": 2016508,
+      "bytes_total": 8388608,
+      "frac_changed": 0.2403864860534668,
+      "edit_frobenius": 1.0564467927525543,
+      "requant_noise_frobenius": 0.6742896915984219,
+      "noise_to_signal": 0.6382618568433261,
+      "rel_edit_size": 0.019784231288010563,
+      "blocks_total": 512,
+      "blocks_overflowed": 21,
+      "blocks_exponent_bumped": 21,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0179848074913025,
+      "bytes_sign_normalized": 2,
+      "direction_signed_gain": 0.10045343488693066,
+      "implied_lambda": 1.2066749056404076,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9978241442220397
+    },
+    "layers.39.attn.wo_b": {
+      "bytes_changed": 14601573,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4351607859134674,
+      "edit_frobenius": 6.802934301447028,
+      "requant_noise_frobenius": 2.5910244269500815,
+      "noise_to_signal": 0.3808686534572227,
+      "rel_edit_size": 0.050979913548129085,
+      "blocks_total": 2048,
+      "blocks_overflowed": 85,
+      "blocks_exponent_bumped": 85,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0430685452052526,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -2.003105766061649,
+      "implied_lambda": 3.1492078241889643,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9993217221203019
+    },
+    "layers.39.ffn.shared_experts.w2": {
+      "bytes_changed": 1862050,
+      "bytes_total": 8388608,
+      "frac_changed": 0.22197365760803223,
+      "edit_frobenius": 0.9695079115353092,
+      "requant_noise_frobenius": 0.6539559220664095,
+      "noise_to_signal": 0.6745235539448123,
+      "rel_edit_size": 0.017715372998773647,
+      "blocks_total": 512,
+      "blocks_overflowed": 18,
+      "blocks_exponent_bumped": 18,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0152531266212463,
+      "bytes_sign_normalized": 3,
+      "direction_signed_gain": 0.17845170591059442,
+      "implied_lambda": 1.170313112431278,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.99707964818421
+    },
+    "layers.40.attn.wo_b": {
+      "bytes_changed": 15133190,
+      "bytes_total": 33554432,
+      "frac_changed": 0.4510042071342468,
+      "edit_frobenius": 6.688942940451778,
+      "requant_noise_frobenius": 2.782606081028505,
+      "noise_to_signal": 0.4160008697638203,
+      "rel_edit_size": 0.04480636069854791,
+      "blocks_total": 2048,
+      "blocks_overflowed": 235,
+      "blocks_exponent_bumped": 235,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.042454719543457,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -1.7532165736985565,
+      "implied_lambda": 2.9315307777632795,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9995236371174596
+    },
+    "layers.40.ffn.shared_experts.w2": {
+      "bytes_changed": 1711222,
+      "bytes_total": 8388608,
+      "frac_changed": 0.203993558883667,
+      "edit_frobenius": 0.8694976688066625,
+      "requant_noise_frobenius": 0.5844799133230961,
+      "noise_to_signal": 0.6722041177238146,
+      "rel_edit_size": 0.017367577600107334,
+      "blocks_total": 512,
+      "blocks_overflowed": 12,
+      "blocks_exponent_bumped": 12,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.021938647542681,
+      "bytes_sign_normalized": 3,
+      "direction_signed_gain": 0.26171898203358795,
+      "implied_lambda": 1.0787694091813762,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9957066147463894
+    },
+    "layers.41.attn.wo_b": {
+      "bytes_changed": 13706495,
+      "bytes_total": 33554432,
+      "frac_changed": 0.40848538279533386,
+      "edit_frobenius": 5.601032844763589,
+      "requant_noise_frobenius": 2.500010316290731,
+      "noise_to_signal": 0.44634809071473186,
+      "rel_edit_size": 0.0401051035044363,
+      "blocks_total": 2048,
+      "blocks_overflowed": 172,
+      "blocks_exponent_bumped": 172,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0567749568394251,
+      "bytes_sign_normalized": 0,
+      "direction_signed_gain": -1.4634845030486305,
+      "implied_lambda": 2.6917725718725776,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9982084252625694
+    },
+    "layers.41.ffn.shared_experts.w2": {
+      "bytes_changed": 1611403,
+      "bytes_total": 8388608,
+      "frac_changed": 0.19209420680999756,
+      "edit_frobenius": 0.8590381533024782,
+      "requant_noise_frobenius": 0.6048316236706351,
+      "noise_to_signal": 0.7040800473709184,
+      "rel_edit_size": 0.015632470697350236,
+      "blocks_total": 512,
+      "blocks_overflowed": 9,
+      "blocks_exponent_bumped": 9,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0149183869361877,
+      "bytes_sign_normalized": 3,
+      "direction_signed_gain": 0.37189512932920954,
+      "implied_lambda": 0.9967010475885506,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9946728125936386
+    },
+    "layers.42.attn.wo_b": {
+      "bytes_changed": 12774715,
+      "bytes_total": 33554432,
+      "frac_changed": 0.3807161748409271,
+      "edit_frobenius": 4.935452983077352,
+      "requant_noise_frobenius": 2.251669158118195,
+      "noise_to_signal": 0.4562234035738367,
+      "rel_edit_size": 0.03848292557176007,
+      "blocks_total": 2048,
+      "blocks_overflowed": 117,
+      "blocks_exponent_bumped": 117,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.083071572440011,
+      "bytes_sign_normalized": 1,
+      "direction_signed_gain": -1.0702047153768344,
+      "implied_lambda": 2.385347963897475,
+      "elements_total": 33554432,
+      "cos_to_first_direction": 0.9942245732980467
+    },
+    "layers.42.ffn.shared_experts.w2": {
+      "bytes_changed": 1684392,
+      "bytes_total": 8388608,
+      "frac_changed": 0.20079517364501953,
+      "edit_frobenius": 1.0070149788084721,
+      "requant_noise_frobenius": 0.6890782837369094,
+      "noise_to_signal": 0.6842780874543155,
+      "rel_edit_size": 0.015173230063210735,
+      "blocks_total": 512,
+      "blocks_overflowed": 19,
+      "blocks_exponent_bumped": 19,
+      "elements_clamped": 0,
+      "max_overshoot_ratio": 1.0086848395211356,
+      "bytes_sign_normalized": 5,
+      "direction_signed_gain": 0.7967170780984925,
+      "implied_lambda": 0.5861125947006848,
+      "elements_total": 8388608,
+      "cos_to_first_direction": 0.9738879114026664
+    }
+  }
+}

config.json

@@ -0,0 +1,71 @@
+{
+  "architectures": [
+    "DeepseekV4ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "expert_dtype": "fp4",
+  "hc_eps": 1e-06,
+  "hc_mult": 4,
+  "hc_sinkhorn_iters": 20,
+  "head_dim": 512,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "index_head_dim": 128,
+  "index_n_heads": 64,
+  "index_topk": 512,
+  "initializer_range": 0.02,
+  "max_position_embeddings": 1048576,
+  "model_type": "deepseek_v4",
+  "moe_intermediate_size": 2048,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 43,
+  "num_hash_layers": 3,
+  "num_key_value_heads": 1,
+  "num_nextn_predict_layers": 1,
+  "o_groups": 8,
+  "o_lora_rank": 1024,
+  "q_lora_rank": 1024,
+  "qk_rope_head_dim": 64,
+  "quantization_config": {
+    "activation_scheme": "dynamic",
+    "fmt": "e4m3",
+    "quant_method": "fp8",
+    "scale_fmt": "ue8m0",
+    "weight_block_size": [
+      128,
+      128
+    ]
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 16,
+    "original_max_position_embeddings": 65536,
+    "type": "yarn"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 1.5,
+  "scoring_func": "sqrtsoftplus",
+  "sliding_window": 128,
+  "swiglu_limit": 10.0,
+  "tie_word_embeddings": false,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "compress_rope_theta": 160000,
+  "compress_ratios": [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 0, 0, 0],
+  "dspark_block_size": 5,
+  "dspark_noise_token_id": 128799,
+  "dspark_target_layer_ids": [40, 41, 42],
+  "dspark_markov_rank": 256
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "do_sample": true,
+  "temperature": 1.0,
+  "top_p": 1.0,
+  "transformers_version": "4.46.3"
+}

model-00001-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3668ba4cccf1ca6a7eb84e888fb92c1cdc7204d472ba9db771e6fd3abf6b874
+size 1059061856

model-00002-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77b26c939a0e25b3113c8d6bb04e1901a748bd4a7d2589e3bfdaabdf1e9bba14
+size 3566321192

model-00003-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:412abf4c906faadc221ef0cb50f90fe20bde8454a08ad4dc2364b6b79e7fda5c
+size 3566321192

model-00004-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9610f56bc587fb0ff9a8b68a60299482ee8c433fe5b5587e4257aca98add4a2e
+size 3596229272

model-00005-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f87a5ac7b8becc31f9c3169afd3a6f33fb82b4af9e21022e3755a10bc28f0180
+size 3568768976

model-00006-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a4f3764e3fc772b9fba67f0a44ef68e18f178b6f00faa80b75db549e51894cd
+size 3590024776

model-00007-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df81bb80e27a689e01fa579eebd6499f86e0b6105f7fea18961aa5eebbbee9bc
+size 3568768976

model-00008-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:224968d2b27f8669365ec08657a768dfec40da0585f85f302a31495931f6a526
+size 3590024776

model-00009-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04d69ef1071fff8721c62968c200a5583122b59b015e9ef9b2978bfed271b2b7
+size 3568768976

model-00010-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:627145f4ebeb1cc3f5bdd03416b8cb7370b3c96974853cbeb8e5516ad5713e49
+size 3590024776

model-00011-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4b8e601dcbebe902e0102e7b098b670a121cb8b9564dd719fc41d782c8416e0
+size 3568768976

model-00012-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64ed4e5f6126ba029c462c9d5fca0fc907c5f855b4ba01194d79560f6db16e42
+size 3590026352

model-00013-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dfe199d07c07ddd141c2c0136a2237f1161250a1a03ebe8deaabac93440da1d
+size 3568770544

model-00014-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45db2f540f825f92453c50335e49aede58cca56bc578d1787c12a0fbca6593e5
+size 3590026352

model-00015-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5810381a0f05b7381c002d299ed6ac19e42eba8070dd17e2703546944d84f292
+size 3568770544

model-00016-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0530b7024771b0ce2df9b40bcc2232578f3300178487ec216863b0b2835617b
+size 3590026352

model-00017-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed11130247118b185ade893c0109bad896dd394cb1e066ce4fce044176261d94
+size 3568770544

model-00018-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e393fea96da2a3414ef089354fc32e1c8891954de40958c84d2c2ecf80365b25
+size 3590026352

model-00019-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a74ca4d3e8e82ce20c458bb8b1900110b753793ad4c58d08f38995f719c616f7
+size 3568770544

model-00020-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f556769926e60309e8defe45ab59fc8b26ae460d30c190cd746a3d78c11e2c2
+size 3590026352

model-00021-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1671cce7f90d781f796b5ca6bf32dd1aeb740abcb2735e41ffd28f62485ce005
+size 3568770544

model-00022-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:decd67a4bd97a75fa36861d2ad3067afeefa6a04a20da997fe6c19f171e70132
+size 3590026352

model-00023-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c61a3e179cdbee19bfb8cbc4e111928ce2f1e1f0f4729d7c0cd5634354a4689d
+size 3568770544

model-00024-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc27aeb4233534f6f7781dcfe57127a3908ae10fc025c5d86dc0682057f8b2fe
+size 3590026352

model-00025-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a66b6b8d5821b68f5b511e4f91e12025cd07d0fa6d0b71e722d825a2d6d878ca
+size 3568770544

model-00026-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:657b89314fbaf6eee4acce24b3baf7e5fd2c5986a96ad85b08d90539cde869fe
+size 3590026352

model-00027-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb01f21a0da0446b0bdf25a127ab19a6b06006acd8735f06a9ebfe34423fd7f5
+size 3568770544

model-00028-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2fd5cbbb639f16e673bc484e5cca16b52a58bf2ab4bd62592e0c5408712ad7c
+size 3590026352

model-00029-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ec2fdf900275daeac0980490c5c731cc7868b151ce1de5698f48418de4fa5f0
+size 3568770544

model-00030-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ed3c317bf967d32133ad3a068ee4c56aae9784bc8b7da694482437f37dc1782
+size 3590026352

model-00031-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5078c3fca3e6370043606ead7856e0b8fe67a9aab52c415769f29934c4d7f5d
+size 3568770544

model-00032-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:163653848f002718d3deaa6ce48885483fc1f2e12e50e44a47477c73ccd91393
+size 3590026352

model-00033-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2cffd43f2a5f491f4691f8694e6bc08239158e143ae7063dc04f0eb0259214a
+size 3568770544

model-00034-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f94945121474cfdb6a9ab175914d3811ffbf08e6cc54082e14e473c755d18d8
+size 3590026352

model-00035-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cb6a316989f7c7385e3ec2bd42ffe766ee126c70ef3466742849982ee1b0f0f
+size 3568770544

model-00036-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e6761421fe944c2143eb897b983085891b421c148c6c17fb5cd8eaa9bdaa497
+size 3590026352

model-00037-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a59d662f1143596d56c452a1230b717ce43edf678207398c573a2503b0f72c91
+size 3568770544

model-00038-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:137fa617a74ba8e73fd76bb1010c7a85d791aaa150006cb66faa04a83e9e730f
+size 3590026352

model-00039-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a29af1aa519d7ce726235ea2c2b38146d756290cda7a82d90c4d4438155b53e4
+size 3568770544

model-00040-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bc93d8a7d1987dc86b14e22b1d8f42ec31da92c56edd9f312daf43f33a6a206
+size 3590026352

model-00041-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd312e7fdd6cb5796df356a7f0314f124851dc149991e9fb02c5bed45cc4ba05
+size 3568770544

model-00042-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d19bf368083c9a183cb0849f316ec17b62f859ca824c0586e779657efb6e6a6
+size 3590026352

model-00043-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7103842ceb70848f9804f55c193d6a57f43174a587cba42b61b5c1bc4e1303d
+size 3568770544

model-00044-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:422d3889fa20c238b7f97464c14df0bcf3328f189c294f41a3a334421dc560c7
+size 3590026352

model-00045-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5be6aed7b84fc87ec42b5d24ba0b0d67f253a3906fcd99c13f4f7be5958fc00
+size 1059332516

model-00046-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5db924ca907e0d93acd975bd5079c3662717f9ac709f23d079bd8f816d29d9dd
+size 3610455184