Omega-Sapphira-Joyous-L3.3-70B-v1.0
Three Llama 3.3 70B fine-tunes, blended by depth rather than by ratio.
Omega-Sapphira-Joyous-L3.3-70B-v1.1 is the follow-up to this merge, and is the one to start with. It performs the same graft with the Joyous share cut back sharply — roughly 43% less in the feed-forward blocks and 67% less in attention — because the amounts used here turned out to be too much. This page is kept for the record.
TLDR: Experimental. For a stable, well-behaveing merge use Omega-Sapphira-L3.3-70B-v1.3. Most merges pick one number and apply it everywhere. This one applies a different number at every layer, and a different number again depending on whether the tensor is an attention block or a feed-forward block. The result is a model whose ancestry shifts as you move up the stack.
Ancestry of the feed-forward weights across the 80-layer stack. Shares are approximate; SLERP interpolates on a hypersphere, not linearly.
What is actually in it
Averaged over all 80 layers:
| Sub-block | Omega | Sapphira | Joyous |
|---|---|---|---|
| MLP — voice, style, knowledge | 52.9% | 34.2% | 12.9% |
| self_attn — structure, coherence | 68.1% | 24.1% | 7.9% |
Omega supplies structure and holds a majority of the attention weights at every depth. Sapphira supplies most of the prose, peaking at 53.9% of the feed-forward weights around layer 48. Joyous is a wash over the middle of the network, peaking at 24.1% around layer 40 and falling to near nothing at either end.
How it was built
Two SLERP passes. The first merged Sapphira into Omega with depth-graded curves, producing an intermediate. The second layered Joyous over that intermediate with a second, gentler set of curves.
Embeddings, per-layer norms, the final norm, and the output head are all taken unmodified from the first-pass merge, so nothing at the boundaries of the network is interpolated twice.
slices:
- sources:
# t=0 => Omega-Sapphira v1.3, t=1 => Joyous
- model: Omega-Sapphira-L3.3-70B-v1.3
layer_range: [0, 80]
- model: allura-org/Llama-3.3-70B-Joyous
layer_range: [0, 80]
merge_method: slerp
base_model: Omega-Sapphira-L3.3-70B-v1.3
parameters:
t:
- filter: model.embed_tokens
value: 0.0
- filter: lm_head
value: 0.0
- filter: model.norm
value: 0.0
- filter: mlp
value: [0.058, 0.080, 0.110, 0.161, 0.219, 0.241, 0.183, 0.124, 0.080, 0.051, 0.029]
- filter: self_attn
value: [0.030, 0.038, 0.060, 0.090, 0.128, 0.158, 0.120, 0.083, 0.053, 0.030, 0.023]
- value: 0.0
dtype: bfloat16
tokenizer:
source: allura-org/Llama-3.3-70B-Joyous
chat_template: "llama3"The eleven values are control points at layers 0, 8, 16 … 80. mergekit interpolates linearly between them.
How it behaves
Based on my own testing: Joyous comes through more strongly than I expected from the numbers, and I find coherence a little softer than in the first-pass merge this was built on.
My reading is that the second pass travels further than the percentages imply — neither parent of that pass shares ancestry with Joyous, so SLERP covers a longer arc for a given t. If you want Omega-Sapphira's structural discipline intact, I would point you at v1.3 instead. This one trades some of that for a warmer prose register, which is a tradeoff I like on some days and not others.
Running it
Llama 3 chat template. The tokenizer comes from Joyous rather than from the merge base — it carries the stock Llama 3.3 vocabulary along with add_bos_token, a pad_token_id, and a generation_config.json, none of which are present in Omega's tokenizer files.
40448 tokens of context fits on a 3090 + 4090 pair — 48GB of VRAM total — at EXL3 4.25bpw.
Samplers
Two reference points. The left column is what Sapphira's author publishes for Sapphira itself; the right is what I actually run this merge at.
| Sapphira 0.2 (published) | This merge (in use) | |
|---|---|---|
| Context | — | 40448 |
| Temperature | 0.95 – 1.1 | 0.88 |
| Min P | 0.015 – 0.03 | 0.025 |
| Top P | 0.97 – 1 | 1 |
| Typical P | — | 1 |
| TFS | — | 1 |
| Top K / Top A | — | 0 |
| XTC threshold | 0.11 | off |
| XTC probability | 0.15 | off |
| Repetition penalty | — | 1.04 |
The gap between the columns is mostly my preference. I run temperature below Sapphira's range, leave XTC off entirely, and let a light repetition penalty do that work instead. That is my response to the softening described above — tightening the distribution pulls back some of what the merge gave away. If you prefer Sapphira's looser settings, expect the softening to be more noticeable.
Treat my column as a starting point rather than a recommendation. Merged models sit between their parents in probability space, so it is worth re-tuning Min-P and the repetition penalty for your own taste rather than carrying anything across unchanged. In my experience a surprising amount of what reads as merge character is sampler character.
Lineage
ReadyArt/L3.3-The-Omega-Directive-70B-Unslop-v2.1 — a QLoRA fine-tune of Steelskull/L3.3-Shakudo-70b on a regex-filtered dataset of roughly 39M tokens, built to strip formulaic phrasing and to keep the model from writing on the user's behalf.
BruhzWater/Sapphira-L3.3-70b-0.2 — a Multi-SLERP of Apocrypha 0.3 and Serpents-Tongue 0.3 over a cogito-v2-preview-llama-70B base. Serpents-Tongue itself draws on Anubis 70B v1.1, Fallen-Llama v1, Hanami-x1, mhnnn-x1, MythoNemo and Eden 0.3, so a good deal of the prose here has a long history behind it.
allura-org/Llama-3.3-70B-Joyous — brought in for its prose register. It is the most repetition-prone of the three, which is why its share is kept small and confined to the feed-forward blocks.
Quantisations
EXL3 and GGUF: pending.
Before you use it
All three parents are unaligned and intended for adult fiction. This merge inherits that and will engage with explicit and violent material without refusing. You are responsible for what you generate. You should be 18 or older.
Merged with mergekit by Cactopus. Released under the Llama 3.3 Community License. All credit for the underlying work belongs to ReadyArt, BruhzWater, allura-org, and the many upstream authors listed on their cards.
- Downloads last month
- 941