A merge is a graft. You take tissue from one model and thread it into the body of another, and the only real question is how deep you go and where. Go shallow and the host never notices. Go deep and it stops being the host.
v1.0 went too deep. Joyous took better than intended — the prose shifted more than I wanted, and the host's reasoning went soft at the edges. This is the same procedure performed with a finer instrument.
Not a docile organism. It latches onto whatever came before and is reluctant to let go — preceding context outweighs the character card, and once it has decided how a scene goes, it is not taking suggestions. Inside the right sampler settings it is very good. Outside them it is a fight. The settings are further down and they are not optional.
If you want something that simply behaves, the host this was grafted onto — Omega-Sapphira v1.3 — is pliant, forgiving about settings, and good at nearly everything.
The specimen
what is in it
The host is Omega-Sapphira v1.3, itself a graft: Sapphira's prose threaded into Omega's skeleton at controlled depth. Joyous is the third organism, introduced into the middle of that composite and kept out of its extremities.
Ancestry of the feed-forward weights through the 80-layer column. The dashed line is Joyous's share of attention, which never exceeds 5%. Shares are approximate — SLERP interpolates on a hypersphere, not linearly.
| Tissue | Omega | Sapphira | Joyous |
|---|---|---|---|
| MLP — voice, style, knowledge | 56.1% | 36.7% | 7.3% |
| self_attn — structure, coherence | 71.6% | 25.8% | 2.6% |
Depth of penetration
what changed from v1.0
The two curves were not pulled back evenly. Attention was cut roughly twice as hard as the feed-forward blocks, because the two symptoms come from different tissue: prose flavour is MLP, and coherence is attention. Cutting both by the same amount would have thrown away the part that was working.
| v1.0 | v1.1 | change | |
|---|---|---|---|
| Joyous in MLP (avg) | 12.9% | 7.3% | −43% |
| Joyous in MLP (peak) | 24.1% | 15.0% | −38% |
| Joyous in attention (avg) | 7.9% | 2.6% | −67% |
| Joyous in attention (peak) | 15.8% | 5.0% | −68% |
Measured against the host, Omega's share of attention was 73.4% in v1.3 before any of this. v1.0 diluted it to 68.1%. v1.1 returns it to 71.6% — within two points of the untouched original, while keeping a little over half of v1.0's prose graft.
Procedure
merge configuration
slices:
- sources:
# t=0 => Omega-Sapphira v1.3 (host), t=1 => Joyous (graft)
- model: cactopus/Omega-Sapphira-L3.3-70B-v1.3
layer_range: [0, 80]
- model: allura-org/Llama-3.3-70B-Joyous
layer_range: [0, 80]
merge_method: slerp
base_model: cactopus/Omega-Sapphira-L3.3-70B-v1.3
parameters:
t:
- filter: model.embed_tokens
value: 0.0
- filter: lm_head
value: 0.0
- filter: model.norm
value: 0.0
- filter: mlp
value: [0.02, 0.03, 0.05, 0.09, 0.13, 0.15, 0.12, 0.08, 0.04, 0.02, 0.01]
- filter: self_attn
value: [0.01, 0.015, 0.02, 0.03, 0.045, 0.05, 0.04, 0.025, 0.015, 0.01, 0.008]
- value: 0.0
dtype: bfloat16
tokenizer:
source: allura-org/Llama-3.3-70B-Joyous
chat_template: "llama3"Eleven control points at layers 0, 8, 16 … 80, interpolated linearly. Embeddings, both per-layer norms, the final norm and the output head are left entirely on the host — nothing at the boundaries of the network is touched, so the graft is confined to the interior.
Viability
how it behaves
A darkhorse, and a stubborn one. The graft worked — the improvement over v1.0 was immediate and not subtle. Testing is ongoing, but the temper described above is the main thing to know, and the settings below defang most of it.
Its failure mode is inertia. It weights the preceding context heavily — often at the expense of the character card, whose details it will quietly drop — and once it has settled on a reading of a scene it does not want to be talked out of it. The settings below mitigate most of that, but not all of it.
If you want something that behaves without being coaxed, take Omega-Sapphira v1.3 instead. It is the host this was grafted onto, it is far more forgiving about samplers, and it has been my daily driver for six months. This one is worth the trouble if you want the warmer prose and are willing to tune for it.
v1.0 is still up for anyone who wants the heavier graft and does not mind the softening it brings.
Culture conditions
running it
Llama 3 chat template. The tokenizer is taken from Joyous rather than from the host, since the host carries Omega's files — which omit add_bos_token, pad_token_id and generation_config.json. Vocabulary and merge rules are stock Llama 3.3 either way.
Samplers
This is the part that matters. Where the host will run on almost anything, this one wants a specific collar. What follows is where I landed after some experimentation — a jumping-off point rather than gospel, but the shape of it is not optional:
Temperature: 0.8 - 0.9
Min P: 0.02 - 0.025
TFS: 0.97 <- required
Top P: 1 (off, see below)
Top K / Top A: 0
Epsilon / Eta: 0
Repetition Penalty: 1.02Tail-free sampling is doing the heavy lifting. Fitting, on a thing with this many tails. It is the single setting that separates a model worth using from one that will spend four paragraphs refusing to notice that the scene has changed. Do not skip it.
Leave Top-P off. Running it alongside TFS appears to degrade output rather than tighten it further — the two are cutting at the same tail from different directions and the result is worse than either alone. Top-P at 1 means disabled.
Temperature above about 1 loosens the grip in the wrong way: you get more variance without more responsiveness, which reads as the model being erratic and stubborn at the same time. Below 0.95 the prose flattens out and you lose the reason to be running this over the host.
Host and symbionts
lineage
Omega-Sapphira v1.3 — the host. A depth-graded SLERP of ReadyArt's Omega Directive v2.1 and BruhzWater's Sapphira 0.2.
allura-org/Llama-3.3-70B-Joyous — the graft, brought in purely for prose register. It is the most repetition-prone of the three, which is exactly why it is kept thin and confined to the feed-forward blocks.
Sapphira in turn descends from Apocrypha 0.3 and Serpents-Tongue 0.3 over a cogito-v2-preview base, and Serpents-Tongue from Anubis 70B v1.1, Fallen-Llama v1, Hanami-x1, mhnnn-x1, MythoNemo and Eden 0.3. Very little here is original tissue.
Quantisations
available cultures
EXL3 · 4.25bpw · H8 — fits 40448 tokens of context on 48GB of VRAM.
GGUF: pending.
Handling
before you use it
Every model underneath this one is unaligned and built for adult fiction. It will engage with explicit and violent material without refusing, and it will not stop to check whether you meant it.
You are responsible for what you generate. You should be 18 or older.
Grafted with mergekit by Cactopus. Released under the Llama 3.3 Community License. Credit for the underlying work belongs to ReadyArt, BruhzWater, allura-org, and the upstream authors listed on their cards.
- Downloads last month
- 1,090