--- license: apache-2.0 base_model: - google/gemma-4-31B-it - ReadyArt/gemma-4-31B-it-scotoma-2 - bgg1996/Melinoe-Gemma4-31B-VL-heretic - sophosympatheia/Glistening-Gem-31B-v2.1 - sophosympatheia/Glistening-Gem-31B-v2.0 - zerofata/G4-MeroMero-v2-31B - Ateron/Gemma-4-Dark-Thoughts-V2-31B base_model_relation: merge tags: - merge - mergekit - roleplay - character-rp - creative writing - storytelling - conversational - low-refusal - nsfw - 31B - gemma-4 ---

SPLIT

31B 路 Gemma 4 路 Merge
“Remember the rules.”
FOUR MODELS
TWO PHASES
ONE MIND
EXPERIMENTAL
PROMPT ADHERENT
ROLEPLAY MERGE
◖◗馃數 or 馃敶
Basically the same, they share a brain, yet differ on their speaking vein.
SPLIT-31B◖  you are herePick this one if you want instruction adherence for things like trackers or header and footer blocks. Expect a more structured output, and staying on track for long sessions.
SPLIT-UNTIED-31B◗  the other headPick this one if you want more creativity, a darker lean and more lewdness, but only if you're ready to trade a little prompt adherence for it.
⬛GGUF quants
Thank you to anyone providing quants!
STATIC IMATRIX · SOON
☰The merge
Two della_linear passes over gemma-4-31B-it, on zerofata’s gemma-4-support fork.
I

Comprehension

Of The Residual Stream

The theory: Adjust q_proj + k_proj for the attention mechanism, and alter gate_proj + up_proj for the knowledge routing.

Model
Purpose
Scope
Melinoe-VL-heretic
persona tracking, world-state stability
q/k 0.45 → 0.30 by depth 路 spine q/k 0.32
gate/up 0.28 → 0.18
Glistening-Gem v2.1
format adherence, in-character
q/k 0.14 → 0.10 路 spine 0.0
gate/up 0.06
MeroMero v2
swipe variety
gate/up only 路 0.30 路 layers 6-47 minus spine
▸  Phase 1: della_linear ```yaml merge_method: della_linear base_model: ./google/gemma-4-31B-it dtype: bfloat16 tokenizer_source: base parameters: normalize: false epsilon: 0.05 lambda: 1.0 int8_mask: true models: - model: ./bgg1996/Melinoe-Gemma4-31B-VL-heretic parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 0.85 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.32 } - { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.32 } - { filter: model.language_model.layers.0.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.0.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.1.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.1.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.2.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.2.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.3.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.3.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.4.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.4.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.6.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.6.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.7.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.7.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.8.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.8.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.9.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.9.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.10.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.10.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.12.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.12.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.13.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.13.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.14.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.14.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.15.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.15.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.16.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.16.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.18.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.18.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.19.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.19.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.20.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.20.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.21.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.21.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.22.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.22.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.24.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.24.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.25.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.25.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.26.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.26.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.27.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.27.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.28.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.28.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.30.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.30.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.31.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.31.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.32.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.32.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.33.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.33.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.34.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.34.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.36.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.36.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.37.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.37.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.38.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.38.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.39.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.39.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.40.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.40.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.42.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.42.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.43.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.43.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.44.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.44.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.45.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.45.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.46.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.46.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.45 } - { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.45 } - { filter: model.language_model.layers.48.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.48.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.49.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.49.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.50.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.50.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.51.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.51.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.52.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.52.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.54.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.54.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.55.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.55.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.56.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.56.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.57.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.57.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.58.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.58.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.3 } - { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.3 } - { filter: model.language_model.layers.0.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.0.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.1.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.1.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.2.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.2.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.3.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.3.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.4.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.4.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.5.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.5.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.6.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.7.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.8.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.9.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.10.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.11.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.11.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.12.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.13.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.14.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.15.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.16.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.17.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.17.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.18.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.19.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.20.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.21.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.22.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.23.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.23.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.24.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.25.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.26.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.27.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.28.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.29.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.29.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.30.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.31.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.32.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.33.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.34.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.35.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.35.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.36.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.37.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.38.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.39.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.40.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.41.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.41.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.42.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.43.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.44.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.45.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.46.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.47.mlp.gate_proj, value: 0.28 } - { filter: model.language_model.layers.47.mlp.up_proj, value: 0.28 } - { filter: model.language_model.layers.48.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.48.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.49.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.49.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.50.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.50.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.51.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.51.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.52.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.52.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.53.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.53.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.54.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.54.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.55.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.55.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.56.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.56.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.57.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.57.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.58.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.58.mlp.up_proj, value: 0.18 } - { filter: model.language_model.layers.59.mlp.gate_proj, value: 0.18 } - { filter: model.language_model.layers.59.mlp.up_proj, value: 0.18 } - { value: 0.0 } - model: ./sophosympatheia/Glistening-Gem-31B-v2.1 parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 0.55 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.0 } - { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.0 } - { filter: model.language_model.layers.0.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.0.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.1.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.1.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.2.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.2.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.3.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.3.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.4.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.4.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.5.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.5.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.6.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.6.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.7.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.7.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.8.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.8.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.9.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.9.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.10.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.10.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.11.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.11.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.12.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.12.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.13.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.13.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.14.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.14.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.15.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.15.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.16.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.16.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.17.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.17.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.18.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.18.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.19.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.19.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.20.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.20.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.21.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.21.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.22.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.22.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.23.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.23.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.24.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.24.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.25.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.25.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.26.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.26.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.27.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.27.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.28.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.28.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.29.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.29.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.30.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.30.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.31.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.31.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.32.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.32.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.33.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.33.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.34.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.34.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.35.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.35.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.36.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.36.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.37.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.37.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.38.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.38.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.39.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.39.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.40.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.40.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.41.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.41.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.42.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.42.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.43.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.43.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.44.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.44.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.45.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.45.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.46.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.46.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.47.self_attn.q_proj, value: 0.14 } - { filter: model.language_model.layers.47.self_attn.k_proj, value: 0.14 } - { filter: model.language_model.layers.48.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.48.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.49.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.49.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.50.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.50.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.51.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.51.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.52.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.52.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.53.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.53.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.54.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.54.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.55.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.55.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.56.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.56.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.57.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.57.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.58.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.58.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.59.self_attn.q_proj, value: 0.1 } - { filter: model.language_model.layers.59.self_attn.k_proj, value: 0.1 } - { filter: model.language_model.layers.0.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.0.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.1.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.1.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.2.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.2.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.3.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.3.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.4.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.4.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.5.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.5.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.6.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.7.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.8.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.9.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.10.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.11.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.11.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.12.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.13.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.14.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.15.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.16.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.17.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.17.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.18.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.19.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.20.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.21.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.22.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.23.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.23.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.24.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.25.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.26.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.27.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.28.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.29.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.29.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.30.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.31.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.32.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.33.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.34.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.35.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.35.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.36.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.37.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.38.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.39.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.40.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.41.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.41.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.42.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.43.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.44.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.45.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.46.mlp.up_proj, value: 0.06 } - { filter: model.language_model.layers.47.mlp.gate_proj, value: 0.06 } - { filter: model.language_model.layers.47.mlp.up_proj, value: 0.06 } - { value: 0.0 } - model: ./zerofata/G4-MeroMero-v2-31B parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 0.60 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.5., value: 0.0 } - { filter: model.language_model.layers.11., value: 0.0 } - { filter: model.language_model.layers.17., value: 0.0 } - { filter: model.language_model.layers.23., value: 0.0 } - { filter: model.language_model.layers.29., value: 0.0 } - { filter: model.language_model.layers.35., value: 0.0 } - { filter: model.language_model.layers.41., value: 0.0 } - { filter: model.language_model.layers.47., value: 0.0 } - { filter: model.language_model.layers.53., value: 0.0 } - { filter: model.language_model.layers.59., value: 0.0 } - { filter: model.language_model.layers.6.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.6.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.7.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.7.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.8.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.8.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.9.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.9.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.10.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.10.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.12.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.12.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.13.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.13.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.14.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.14.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.15.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.15.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.16.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.16.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.18.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.18.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.19.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.19.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.20.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.20.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.21.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.21.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.22.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.22.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.24.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.24.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.25.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.25.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.26.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.26.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.27.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.27.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.28.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.28.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.30.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.30.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.31.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.31.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.32.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.32.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.33.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.33.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.34.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.34.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.36.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.36.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.37.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.37.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.38.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.38.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.39.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.39.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.40.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.40.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.42.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.42.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.43.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.43.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.44.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.44.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.45.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.45.mlp.up_proj, value: 0.3 } - { filter: model.language_model.layers.46.mlp.gate_proj, value: 0.3 } - { filter: model.language_model.layers.46.mlp.up_proj, value: 0.3 } - { value: 0.0 } ```
II

Voice

Of The Residual Stream

If that also checks out, then v_proj steers the expression of what q/k came up with, and o_proj + mlp.down_proj are the layers that coordinate how the output should be written.

Model
Purpose
Scope
scotoma-2
Anti-Gemma tics
weight 1.0 路 density 1.0 路 layers 7-41
alone on 7-11
Dark-Thoughts V2
darker lean, less sanitizing
v/o/down 路 0.50 (24-47) → 0.30 (48-53)
Glistening-Gem v2.0
self-repetition, coherence
v/o/down 路 0.22 路 layers 12-41
Melinoe-VL-heretic
willingness
v/o/down 路 0.30 路 layers 42-53
▸  Phase 2: della_linear ```yaml merge_method: della_linear base_model: ./Split-31B-read dtype: bfloat16 tokenizer_source: base parameters: normalize: false epsilon: 0.05 lambda: 1.0 int8_mask: true models: - model: ./ReadyArt/gemma-4-31B-it-scotoma-2 parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 1.0 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.7.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.7.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.8.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.8.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.9.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.9.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.10.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.10.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.11.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.11.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.12.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.12.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.13.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.13.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.14.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.14.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.15.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.15.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.16.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.16.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.17.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.17.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.18.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.18.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.19.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.19.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.20.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.20.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.21.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.21.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.22.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.22.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.23.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.23.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.24.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.24.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.25.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.25.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.26.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.26.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.27.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.27.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.28.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.28.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.29.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.29.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.30.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.30.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.31.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.31.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.32.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.32.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.33.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.33.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.34.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.34.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.35.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.35.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.36.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.36.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.37.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.37.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.38.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.38.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.39.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.39.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.40.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.40.mlp.down_proj, value: 1.0 } - { filter: model.language_model.layers.41.self_attn.o_proj, value: 1.0 } - { filter: model.language_model.layers.41.mlp.down_proj, value: 1.0 } - { value: 0.0 } - model: ./Ateron/Gemma-4-Dark-Thoughts-V2-31B parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 0.75 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.24.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.24.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.24.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.25.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.25.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.25.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.26.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.26.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.26.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.27.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.27.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.27.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.28.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.28.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.28.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.29.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.29.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.30.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.30.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.30.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.31.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.31.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.31.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.32.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.32.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.32.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.33.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.33.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.33.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.34.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.34.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.34.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.35.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.35.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.36.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.36.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.36.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.37.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.37.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.37.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.38.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.38.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.38.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.39.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.39.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.39.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.40.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.40.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.40.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.41.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.41.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.42.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.42.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.42.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.43.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.43.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.43.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.44.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.44.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.44.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.45.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.45.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.45.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.46.self_attn.v_proj, value: 0.5 } - { filter: model.language_model.layers.46.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.46.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.47.self_attn.o_proj, value: 0.5 } - { filter: model.language_model.layers.47.mlp.down_proj, value: 0.5 } - { filter: model.language_model.layers.48.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.48.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.48.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.49.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.49.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.49.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.50.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.50.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.50.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.51.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.51.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.51.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.52.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.52.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.52.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.53.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.53.mlp.down_proj, value: 0.3 } - { value: 0.0 } - model: ./sophosympatheia/Glistening-Gem-31B-v2.0 parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 0.60 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.12.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.12.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.12.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.13.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.13.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.13.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.14.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.14.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.14.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.15.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.15.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.15.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.16.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.16.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.16.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.17.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.17.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.18.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.18.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.18.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.19.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.19.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.19.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.20.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.20.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.20.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.21.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.21.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.21.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.22.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.22.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.22.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.23.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.23.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.24.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.24.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.24.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.25.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.25.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.25.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.26.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.26.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.26.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.27.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.27.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.27.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.28.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.28.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.28.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.29.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.29.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.30.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.30.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.30.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.31.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.31.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.31.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.32.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.32.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.32.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.33.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.33.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.33.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.34.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.34.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.34.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.35.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.35.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.36.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.36.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.36.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.37.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.37.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.37.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.38.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.38.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.38.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.39.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.39.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.39.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.40.self_attn.v_proj, value: 0.22 } - { filter: model.language_model.layers.40.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.40.mlp.down_proj, value: 0.22 } - { filter: model.language_model.layers.41.self_attn.o_proj, value: 0.22 } - { filter: model.language_model.layers.41.mlp.down_proj, value: 0.22 } - { value: 0.0 } - model: ./bgg1996/Melinoe-Gemma4-31B-VL-heretic parameters: density: - { filter: norm, value: 1.0 } - { filter: layer_scalar, value: 1.0 } - { value: 0.80 } weight: - { filter: embed_tokens, value: 0.0 } - { filter: lm_head, value: 0.0 } - { filter: embed_vision, value: 0.0 } - { filter: vision_tower, value: 0.0 } - { filter: multi_modal_projector, value: 0.0 } - { filter: model.language_model.layers.42.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.42.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.42.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.43.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.43.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.43.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.44.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.44.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.44.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.45.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.45.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.45.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.46.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.46.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.46.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.47.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.47.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.48.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.48.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.48.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.49.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.49.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.49.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.50.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.50.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.50.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.51.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.51.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.51.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.52.self_attn.v_proj, value: 0.3 } - { filter: model.language_model.layers.52.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.52.mlp.down_proj, value: 0.3 } - { filter: model.language_model.layers.53.self_attn.o_proj, value: 0.3 } - { filter: model.language_model.layers.53.mlp.down_proj, value: 0.3 } - { value: 0.0 } ```

◆  Afterthought

You can't stump over scotoma-2. Checking benchmarks like caliperbench objectively, almost none of the merges beat their best parent when it comes to adjective and adverb density. So, instead of just blending scotoma-2 in again, I compared the delta of it's layers to the base model and gave it a weight of 1.0 over those. No idea if that survived contact with the actual prose on the long run, though.

Samplers and Thinking. Samplers are... yeah, they are. I give you a serving suggestion below, but go with whatever works for you. Thinking also works, but some people always report that thinking breaks for some merges, while it doesn't for others, so I still don't know if it's truly merge, front-end or even back-end related. So; just assume it's a no-think model if it breaks for you and we're good.

☰Samplers
Setting
Value
Temperature
0.9
Min-P
0.05
Top-K / Top-P
off / 1.0
Repetition penalty
off
DRY multiplier
0.8
◖◗Usage Like Always

◆  Thinking. Put <|think|> at the start of the system prompt, or let your front-end handle the Gemma 4 template. Reasoning gives better recall capability, but you might lose a bit on ERP pacing.

◆  Give it a scene. A sophisticated card will always get more nuanced output of this model than a few lines, just saying.

◖◗Thankus Maximus
  • AesSedai / ReadyArt: scotoma-2 is just a banger, and you know it Aes.
  • bgg1996: Melinoe-VL-heretic, doing two different jobs here on two different tensor families, cool thing.
  • sophosympatheia: Glistening-Gem v2.0 and v2.1, kind of being the glue here.
  • zerofata: MeroMero v2, and the mergekit fork obviously, your tunes going into anything should also speak for itself.
  • Ateron: Dark-Thoughts V2. The rat guy that cursed me with the habit of merging. Thank you Mr. Rat, this was a good one.
  • Gryphe: for the untied head.
  • Charles Goddard/ArceeAI: For the mergekit and its continuation.
  • The BeaverAI community: for the feedback that once again lead to actually shaping this one significantly.
  • Google DeepMind & the open-source community: for the base model and everything that ended up inside this merge.
◖ · ◗