danish-lm-400m-sft-v29-v30-soup

Uniform 0.5/0.5 weight-space average of jensjepsen/danish-lm-400m-sft-v29-avg-top7 and jensjepsen/danish-lm-400m-sft-v30-avg-top7 — both derived from the same base jensjepsen/danish-lm-400m-base-ropext2048-v1, so weight interpolation is well-defined.

Base

• Base: jensjepsen/danish-lm-400m-base-ropext2048-v1 (2048-token context)
• Tokenizer: jensjepsen/danish-tokenizer

Recipe

Element-wise 50/50 mean of the two parents' model.safetensors, computed in float32 and cast back to the parents' original dtype at save time. No fine-tuning on top — just weight averaging. Script: scripts/avg_ckpts.py in the espllm repo.

Tried 0.7·v29 + 0.3·v30 on cit-gen: 30.28% vs balanced 30.83%. Balanced 50/50 was the winner; weighting toward the individually-stronger parent did NOT help.

Downstream — soup vs parents

Wins on all 5 freeform-generation evals:

eval	v29-avg7	v30-ep4-avg7	soup	Δ vs best parent
cit-gen (freeform Q, substring gold)	30.00	27.64	30.83	+0.83
GSM8K[da] gen (CoT → number)	17.31	17.54	18.30	+0.76
SciQ openq (freeform Q, substring)	11.70	10.50	12.50	+0.80
IFEval-DA prompt-strict	23.1	23.9	25.8	+1.9
IFEval-DA inst-strict	36.6	37.2	40.5	+3.3

Loses on 4-of-4 MC-letter emission evals (5th is noise near random):

eval	v29-avg7	v30-ep4-avg7	soup	Δ vs best parent
SciQ MC-letter	60.30	62.00	59.70	−2.30
Cit-MC	49.3	48.80	48.6	−0.7
PIQA (2-choice)	—	56.0	54.0	−2.0
ARC (4-5 choice)	—	27.76	26.82	−0.9
GPQA (near-random baseline)	—	21.72	24.75	+3.0 (noise)

Why the asymmetry

Free-form generation lets the parents' complementary knowledge compose over multi-token outputs — e.g. avg correctly answered "Thomas Vinterberg 2021 Oscar?" with "Druk" when v29 said "Den gode vilje" and v30 said "Pusher II", and correctly said "De jævne folk (bønder og tjenestefolk)" about 1800-tallet Venstre-vælgere when neither parent had the answer.

MC-letter emission has only ~4 valid single-token outputs. When the two parents' argmax disagrees on which letter, the averaged logits usually break in one specific direction — often losing v30's individual wins. Occasionally the averaged distribution's argmax lands on a third letter neither parent picked (surprising but rare).

When to use this soup

• Best default for assistant chat, IFEval, freeform QA, GSM-style reasoning.
• If your benchmark is MC-letter picking (SciQ MC, PIQA, ARC, cit-MC), use the individual parent instead:
  • PIQA / ARC / SciQ MC → v30-avg-top7
  • Cit-MC → v29-avg-top7

Not measured

• Textman ChrF++ — full 1000-item val on 1080 Ti takes ~2.6h; skipped locally. Pattern predicts soup wins by 1-2pp.
• Weighted sweeps beyond 0.7/0.3 — worth exploring, but 50/50 was the best of what we tried.

Related

• v29-avg-top7 — task-expansion mix, 16 sources
• v30-avg-top7 (ep 4) — v29 mix + 3 STEM datasets, 4-epoch
• base ropext2048-v1 — long-context base both parents were fine-tuned from