50/50 weight-space avg

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+language:
+- da
+base_model: jensjepsen/danish-lm-400m-base-ropext2048-v1
+tags:
+- danish
+- sft
+- long-context
+- "2048"
+- model-soup
+---
+
+# danish-lm-400m-sft-v29-v30-soup
+
+Uniform 0.5/0.5 weight-space average of
+[`jensjepsen/danish-lm-400m-sft-v29-avg-top7`](https://huggingface.co/jensjepsen/danish-lm-400m-sft-v29-avg-top7)
+and
+[`jensjepsen/danish-lm-400m-sft-v30-avg-top7`](https://huggingface.co/jensjepsen/danish-lm-400m-sft-v30-avg-top7)
+— both derived from the same base
+[`jensjepsen/danish-lm-400m-base-ropext2048-v1`](https://huggingface.co/jensjepsen/danish-lm-400m-base-ropext2048-v1),
+so weight interpolation is well-defined.
+
+## Base
+
+- **Base:** `jensjepsen/danish-lm-400m-base-ropext2048-v1` (2048-token context)
+- **Tokenizer:** `jensjepsen/danish-tokenizer`
+
+## Recipe
+
+Element-wise 50/50 mean of the two parents' `model.safetensors`, computed in
+float32 and cast back to the parents' original dtype at save time. No fine-tuning
+on top — just weight averaging. Script: `scripts/avg_ckpts.py` in the espllm repo.
+
+Tried 0.7·v29 + 0.3·v30 on cit-gen: **30.28%** vs balanced 30.83%. Balanced
+50/50 was the winner; weighting toward the individually-stronger parent
+did NOT help.
+
+## Downstream — soup vs parents
+
+**Wins on all 5 freeform-generation evals:**
+
+| eval | v29-avg7 | v30-ep4-avg7 | **soup** | Δ vs best parent |
+|---|---|---|---|---|
+| cit-gen (freeform Q, substring gold) | 30.00 | 27.64 | **30.83** | +0.83 |
+| GSM8K[da] gen (CoT → number)         | 17.31 | 17.54 | **18.30** | +0.76 |
+| SciQ openq (freeform Q, substring)   | 11.70 | 10.50 | **12.50** | +0.80 |
+| **IFEval-DA prompt-strict**          | 23.1  | 23.9  | **25.8**  | **+1.9** |
+| **IFEval-DA inst-strict**            | 36.6  | 37.2  | **40.5**  | **+3.3** |
+
+**Loses on 4-of-4 MC-letter emission evals** (5th is noise near random):
+
+| eval | v29-avg7 | v30-ep4-avg7 | soup | Δ vs best parent |
+|---|---|---|---|---|
+| SciQ MC-letter | 60.30 | 62.00 | 59.70 | −2.30 |
+| Cit-MC | 49.3 | 48.80 | 48.6 | −0.7 |
+| PIQA (2-choice) | — | 56.0 | 54.0 | −2.0 |
+| ARC (4-5 choice) | — | 27.76 | 26.82 | −0.9 |
+| GPQA (near-random baseline) | — | 21.72 | 24.75 | +3.0 (noise) |
+
+## Why the asymmetry
+
+Free-form generation lets the parents' complementary knowledge compose over
+multi-token outputs — e.g. avg correctly answered *"Thomas Vinterberg 2021
+Oscar?"* with **"Druk"** when v29 said *"Den gode vilje"* and v30 said
+*"Pusher II"*, and correctly said *"De jævne folk (bønder og tjenestefolk)"*
+about 1800-tallet Venstre-vælgere when neither parent had the answer.
+
+MC-letter emission has only ~4 valid single-token outputs. When the two
+parents' argmax disagrees on which letter, the averaged logits usually break
+in one specific direction — often losing v30's individual wins. Occasionally
+the averaged distribution's argmax lands on a **third** letter neither parent
+picked (surprising but rare).
+
+## When to use this soup
+
+- Best default for **assistant chat, IFEval, freeform QA, GSM-style
+  reasoning**.
+- If your benchmark is **MC-letter picking** (SciQ MC, PIQA, ARC, cit-MC),
+  use the individual parent instead:
+  - PIQA / ARC / SciQ MC → `v30-avg-top7`
+  - Cit-MC → `v29-avg-top7`
+
+## Not measured
+
+- **Textman ChrF++** — full 1000-item val on 1080 Ti takes ~2.6h; skipped
+  locally. Pattern predicts soup wins by 1-2pp.
+- **Weighted sweeps beyond 0.7/0.3** — worth exploring, but 50/50 was the
+  best of what we tried.
+
+## Related
+
+- [v29-avg-top7](https://huggingface.co/jensjepsen/danish-lm-400m-sft-v29-avg-top7)
+  — task-expansion mix, 16 sources
+- [v30-avg-top7 (ep 4)](https://huggingface.co/jensjepsen/danish-lm-400m-sft-v30-avg-top7)
+  — v29 mix + 3 STEM datasets, 4-epoch
+- [base ropext2048-v1](https://huggingface.co/jensjepsen/danish-lm-400m-base-ropext2048-v1)
+  — long-context base both parents were fine-tuned from

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 24,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.4",
+  "use_cache": true,
+  "vocab_size": 16007
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "pad_token_id": 0,
+  "transformers_version": "4.55.4"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bcf863067115e4651b4f04a049554fb0d73600c2902f49262412073353d88d82
+size 1658847376

special_tokens_map.json

@@ -0,0 +1,75 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|/tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|tool_result|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|/tool_result|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,109 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16000": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16001": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16002": {
+      "content": "<|end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16003": {
+      "content": "<|tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16004": {
+      "content": "<|/tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16005": {
+      "content": "<|tool_result|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16006": {
+      "content": "<|/tool_result|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|end|>",
+    "<|tool_call|>",
+    "<|/tool_call|>",
+    "<|tool_result|>",
+    "<|/tool_result|>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}