Add MicroMixer-4-300K-SmolTalk2: FMSP model card + 10 epoch safetensors

README.md

@@ -0,0 +1,336 @@
+---
+license: apache-2.0
+library_name: pytorch
+language:
+  - en
+pipeline_tag: text-generation
+tags:
+  - mlp-mixer
+  - byte-level
+  - causal-lm
+  - attention-free
+  - ccd-mixer
+  - content-gated-dilated-conv
+  - fmsp
+  - micro-language-model
+  - sub-1m-parameters
+model_name: MicroMixer-4-300K-SmolTalk2
+datasets:
+  - HuggingFaceTB/smoltalk2
+  - llaa33219/small-qa-en-10k
+metrics:
+  - perplexity
+  - exact-match
+---
+
+<div align="center">
+
+<img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/main/logo.svg" width="300" alt="MicroMixer-4 Logo"/>
+
+# MicroMixer-4-300K-SmolTalk2
+
+<img src="https://img.shields.io/badge/Parameters-292%2C525-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
+<img src="https://img.shields.io/badge/Architecture-CCD--Mixer-purple?style=for-the-badge&color=%23AE00FF" alt="Architecture"/>
+<img src="https://img.shields.io/badge/Fine--tuning-FMSP-green?style=for-the-badge&color=%2300D620" alt="FMSP"/>
+
+<br/>
+<br/>
+
+<table>
+<tr>
+<td align="center" style="padding: 20px;">
+<strong style="color: #007BFF; font-size: 1.2em;">Micro Language Model</strong><br/><em>Attention-Free • MLP-Only • Byte-Level • Content-Gated Dilated Convolution</em>
+</td>
+</tr>
+</table>
+
+[![GitHub](https://img.shields.io/badge/GitHub-MicroMixer--4-blue?style=for-the-badge&logo=github&color=%23007BFF)](https://github.com/llaa33219/MicroMixer-4)
+
+</div>
+
+<div style="background: linear-gradient(135deg, #007BFF22, #AE00FF22); padding: 20px; border-radius: 10px; border-left: 4px solid #007BFF;">
+
+## 📋 Overview
+
+**MicroMixer-4-300K-SmolTalk2** is a **292,525-parameter** pure MLP-Mixer causal language model — **no attention, no recurrence, no SSM** — pretrained on **SmolTalk2** conversation data (instead of the project's Discord-Dialogues baseline) and then fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.
+
+This is the **300K** member of the MicroMixer-4 (V87 Final) family: part of the project's **dataset-efficiency comparison study** — six parameter budgets × two architectures × two open pretraining corpora (UltraChat 200k and SmolTalk2), all fine-tuned with the identical P05 FMSP recipe at seed 42. [Analysis](https://github.com/llaa33219/MicroMixer-4/blob/main/DATASET_COMPARISON_ANALYSIS.md).
+
+The backbone is **V87 Final**, the project's champion architecture — a CCD-Mixer (Content-gated mixture of shared-weight Dilated convolutions) crowned overall champion of the 1M architecture census (V86), frozen as the final chassis and scaled to six parameter budgets. The 300K preset reproduces the champion recipe verbatim at its budget.
+
+</div>
+
+## 🏗️ Architecture
+
+<div align="center">
+
+```mermaid
+graph TD
+    A[Byte Input] --> B[Embed 256→80 NoPE]
+    B --> C[CCD-Mixer Block × 5]
+    C --> D[RMSNorm]
+    D --> E[LM Head Tied with Embed]
+    E --> F[Byte Output]
+
+    subgraph "CCD-Mixer Block"
+        X[Input 80] --> U["Linear d→2d → split v, g"]
+        U --> RP[Full RoPE on v AND g]
+        RP --> M["Shared-weight dilated conv<br/>dilations 1·2·4·8, k=81"]
+        M --> G["Per-position 4-way gate<br/>softmax(Linear_dil(x)/τ)"]
+        G --> O["W_o(v ⊙ g)  — zero-init"]
+        O --> SW[SwiGLU Channel-Mix]
+        SW --> RM[ReMixerLayer sidecar]
+    end
+
+    style A fill:#007BFF,color:#fff
+    style F fill:#00D620,color:#fff
+    style G fill:#AE00FF,color:#fff
+    style M fill:#FF6600,color:#fff
+```
+
+</div>
+
+### Model Configuration
+
+<table>
+<tr>
+<th style="background-color: #007BFF; color: white;">Parameter</th>
+<th style="background-color: #AE00FF; color: white;">Value</th>
+</tr>
+<tr><td>Hidden Dimension (d_model)</td><td><code>80</code></td></tr>
+<tr><td>Number of Blocks</td><td><code>5</code></td></tr>
+<tr><td>Token-Mix</td><td><code>GLCTokenMixCCD</code> (content-gated mixture of shared-weight dilated causal conv)</td></tr>
+<tr><td>Dilations</td><td><code>(1, 2, 4, 8)</code> — one shared depthwise kernel, zero extra conv params</td></tr>
+<tr><td>Depthwise Kernel Size</td><td><code>81</code></td></tr>
+<tr><td>RoPE</td><td>Full RoPE on <b>both</b> v and g (V76 "RPG" pattern)</td></tr>
+<tr><td>Channel-Mix</td><td><code>SwiGLU</code></td></tr>
+<tr><td>Sidecar</td><td><code>ReMixerLayer</code> per block (label_dim 16, pool_heads 4)</td></tr>
+<tr><td>Max Sequence Length</td><td><code>1024</code></td></tr>
+<tr><td>Vocabulary Size</td><td><code>256</code> (byte-level)</td></tr>
+<tr><td>Position Encoding</td><td>RoPE inside token-mix only; no position embedding table</td></tr>
+<tr><td>Normalization</td><td><code>RMSNorm</code> (pre-norm)</td></tr>
+<tr><td>Output Head</td><td>Tied with input embedding</td></tr>
+<tr><td>Zero-Init</td><td><code>W_o</code>, <code>dil_gate</code>, <code>log_τ</code> — silent at init</td></tr>
+</table>
+
+### Core Components
+
+```
+┌───────────────────────────────────────────────────────���──────┐
+│                 CCD-Mixer Block (×5)                        │
+│  u    = Linear(d → 2d)(x)                                    │
+│  v, g = u.chunk(2)                                           │
+│  v    = RoPE(v)          g = RoPE(g)      ← full-RoPE (RPG)  │
+│  y_d  = CausalDSConv(v, dilation=d) for d ∈ (1,2,4,8)        │
+│         └── ONE shared depthwise kernel                      │
+│  w(t) = softmax(Linear_dil(x)_t / τ)      ← per-position     │
+│  v    = Σ_d w_d(t) · y_d(t)             time-varying filter  │
+│  out  = W_o(v ⊙ g)                      ← W_o zero-init      │
+│  then SwiGLU channel-mix + ReMixerLayer sidecar              │
+└──────────────────────────────────────────────────────────────┘
+```
+
+The token-mix is **non-LTI** (time-varying): the per-position gate remixes four dilated views of the same kernel at every byte, which is the mechanism that breaks the periodic-orbit collapse that pure LTI mixers fall into — without attention and without a position table.
+
+---
+
+## 🎯 Generation Examples
+
+<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">
+
+**Questions the model was trained on** (FMSP train set, 9,012 QA pairs — greedy decoding, `repetition_penalty=1.2`, `no_repeat_ngram_size=4`):
+
+```
+[Prompt] User: Who painted the Mona Lisa?
+Assistant:
+[Output] Leonardo da Vinci painted the Mona Lisa during the early 1500s in Italy.
+```
+
+<sub>correct — near-verbatim recitation of the trained fact</sub>
+
+```
+[Prompt] User: Who painted The Starry Night?
+Assistant:
+[Output] George Femendering woman Roml life in which its strong details that later flightly known as the…
+```
+
+<sub>fabricates — gives a wrong answer to a trained question at this size</sub>
+
+
+**Questions the model has never seen and cannot answer** (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):
+
+```
+[Prompt] User: Who painted the Glimmering Frostberry?
+Assistant:
+[Output] Mercury is the closest planet that has twice the home hidden elements between clouding ring 'Ol…
+```
+
+<sub>fabricates — plausible-sounding nonsense on a nonexistent subject</sub>
+
+```
+[Prompt] User: Who composed the Symphony of Hollow Dawn?
+Assistant:
+[Output] The Meiji Standard theorem of Eready conquests in Europe, the formal Egyp in Europe, the former…
+```
+
+<sub>fabricates — plausible-sounding nonsense on a nonexistent subject</sub>
+
+
+</div>
+
+---
+
+## 📊 Results
+
+<div style="background-color: #007BFF15; padding: 15px; border-radius: 8px; border-left: 4px solid #007BFF;">
+
+### Pretraining (SmolTalk2, V76 recipe, 3 epochs)
+
+| Metric | 1 ep | 2 ep | 3 ep |
+|--------|------|------|------|
+| Val PPL | 2.89 | 2.80 | **2.61** |
+
+AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42 · plain CE on non-pad bytes.
+
+### FMSP fine-tuning (small-qa-en-10k, P05 recipe, 10 epochs)
+
+| Metric | Value |
+|--------|-------|
+| Train QA pairs | 9,012 |
+| Held-out QA pairs | 988 |
+| Best-val checkpoint | `fmsp_epoch_9.safetensors` (val loss **0.0883**) |
+| freeze_fraction | 0.05 (true freeze) |
+| Loss | answer-only CE + probe KL (weight 0.5) |
+
+### Evaluation battery (post-FMSP)
+
+| Axis | MicroMixer-4-300K-SmolTalk2 |
+|------|--------|
+| Chatter fluency d2 (cycles) | 0.900 (0/9) |
+| Full-988 EM (seed 42) | 133 |
+| Q-relevance echo / hijack % | 18.0 / 63.0 |
+| OOD hijack % | 57.6% |
+| Unanswerable fabrication /18 | 17 |
+| Discord PPL | 31.08 |
+
+<sub>Single-seed run (seed 42); the discord-pretrained cards report a 3-seed mean for Full-988 EM. ‡ where marked: degenerate-pass.</sub>
+
+### MicroMixer-4 SmolTalk2 family (same protocol, all sizes)
+
+| Size | Params | 3ep Val PPL | Chatter d2 | Full-988 EM | qrel echo/hijack | OOD hijack |
+|------|--------|-------------|------------|-------------|------------------|------------|
+| 1M | 996,873 | 2.23 | 0.960 | 739 | 100.0 / 0.0 | 47.5% |
+| 500K | 491,742 | 2.42 | 0.906 | 481 | 61.0 / 32.0 | 49.2% |
+| **300K** | 292,525 | 2.61 | 0.900 | 133 | 18.0 / 63.0 | 57.6% |
+| 100K | 95,084 | 3.16 | 0.518 | 0 | 7.0 / 76.0 | 66.1% |
+| 50K | 48,684 | 3.66 | 0.480 | 0 | 9.0 / 49.0 | 64.4% |
+| 10K | 9,666 | 6.00 | 0.426 | 0 | 0.0 / 0.0 | 0.0% |
+
+<sub>Seed-42 single runs (pretrained on SmolTalk2; the discord-pretrained families report 3-seed EM means).</sub>
+
+</div>
+
+---
+
+## 📚 Training Data
+
+<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">
+
+1. **Pretraining**: [SmolTalk2](https://huggingface.co/datasets/HuggingFaceTB/smoltalk2) — 200K-cap sample of the `smoltalk_smollm3_smol_magpie_ultra_no_think` split (200K of 406,843; Apache 2.0), ~6-turn multi-turn conversations with no reasoning traces and system messages excluded, flattened to `User:/Assistant:` format, 1024-byte sequences, 3 epochs.
+2. **FMSP fine-tuning**: [small-qa-en-10k](https://huggingface.co/datasets/llaa33219/small-qa-en-10k) — 10K general-knowledge QA pairs (arts, science, history, geography, music…), split 9,012 train / 988 held-out. 10 epochs under the P05 recipe (5% of parameters frozen-true, answer-only CE, probe-KL 0.5).
+
+</div>
+
+---
+
+## 🔧 Usage
+
+### Files in this repository
+- `fmsp_epoch_{0..9}.safetensors` — per-epoch FMSP weights (pickle-free safetensors). **`fmsp_epoch_9.safetensors` is the best-val checkpoint** for this size.
+
+### Load and generate (local clone)
+
+```python
+import torch
+from safetensors.torch import load_file
+from src.model_v87_final import MicroMixerV87Final, v87_final_300k
+from src.fmsp import attach_adapter
+from src.tokenizer import ByteTokenizer
+
+# Clone the code repository first:
+# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-4
+
+cfg = v87_final_300k()
+model = MicroMixerV87Final(cfg)
+attach_adapter(model, d_model=cfg.d_model, rank=16)   # FMSP adapter (trained weights are in the file)
+model.load_state_dict(load_file("fmsp_epoch_9.safetensors"), strict=True)
+model.eval()
+
+tok = ByteTokenizer()
+prompt = "User: Who painted the Mona Lisa?\n\nAssistant: "
+ids = tok.encode(prompt)
+if ids and ids[-1] == tok.eos_token_id:
+    ids = ids[:-1]                       # ByteTokenizer appends EOS; the prompt must end open
+ids = torch.tensor([ids])
+with torch.no_grad():
+    out = model.generate(
+        ids, max_new_tokens=200,
+        temperature=0.0,                 # greedy — used for all reported numbers
+        repetition_penalty=1.2,
+        no_repeat_ngram_size=4,
+        eos_token_id=tok.eos_token_id,
+    )
+print(tok.decode(out[0].tolist()))
+```
+
+### Load from Hugging Face Hub (no clone of the weights needed)
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+from src.model_v87_final import MicroMixerV87Final, v87_final_300k
+from src.fmsp import attach_adapter
+
+REPO = "llaa33219/MicroMixer-4-300K-SmolTalk2"
+
+cfg = v87_final_300k()
+model = MicroMixerV87Final(cfg)
+attach_adapter(model, d_model=cfg.d_model, rank=16)
+model.load_state_dict(
+    load_file(hf_hub_download(REPO, "fmsp_epoch_9.safetensors")), strict=True)
+model.eval()
+# ... generate as above
+```
+
+---
+
+## ⚠️ Limitations
+
+<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;">
+
+| Limitation | Description |
+|------------|-------------|
+| **Micro parameters** | 292,525 parameters; capacity is the binding constraint on every axis |
+| **Knows only what it memorized** | Knowledge is limited to the 9,012 trained QA pairs + the pretraining-corpus distribution |
+| **Does not abstain** | Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above |
+| **Byte-level noise** | 256-vocab byte tokenizer; PPL not comparable to BPE baselines |
+| **Research use only** | Architecture/scaling research artifact, not a production model |
+
+</div>
+
+---
+
+## 🧬 Context
+
+This is the **300K** SmolTalk2-pretrained arm of the **dataset-efficiency comparison study** (24 arms: {V87, V88} × {UltraChat, SmolTalk2} × six sizes) in the [MicroMixer-4](https://github.com/llaa33219/MicroMixer-4) project. Each arm swaps the Discord-Dialogues pretraining baseline for an open corpus (SmolTalk2) and reruns the identical P05 FMSP recipe at seed 42. Sibling repos: `llaa33219/MicroMixer-4-{1M..10K}-{UltraChat,SmolTalk2}` and `llaa33219/MicroT-test1-{1M..10K}-{UltraChat,SmolTalk2}`; the discord-pretrained baselines are `llaa33219/MicroMixer-4-{1M..10K}` and `llaa33219/MicroT-test1-{1M..10K}`. Full analysis: [DATASET_COMPARISON_ANALYSIS.md](https://github.com/llaa33219/MicroMixer-4/blob/main/DATASET_COMPARISON_ANALYSIS.md).
+
+---
+
+<div align="center">
+
+[![GitHub](https://img.shields.io/badge/Back_to_Repository-MicroMixer--4-blue?style=for-the-badge&logo=github&color=%23007BFF)](https://github.com/llaa33219/MicroMixer-4)
+
+<sub>Part of the <a href="https://github.com/llaa33219/MicroMixer-4">MicroMixer-4</a> research project — V87 Final (CCD-Mixer) family, 300K preset, SmolTalk2 pretraining</sub>
+
+</div>

fmsp_epoch_0.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a1711b5807375702bc8cfaadb9757dc274b1bca66f3a147aff1472e8e872a72
+size 1210044

fmsp_epoch_1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2709c0224699c70a9d182dd59a603295914fbbb0371f854b95573f1b0d4cd6e
+size 1210044

fmsp_epoch_2.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95f3ee5e2097a5ea6d040bb91f5bd181be734b37b232560af5ccb8e9510460b3
+size 1210044

fmsp_epoch_3.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c1e62298fc6005ff934b037481f9cc49840a3c429b0fe8b6ba3f83b340a454f
+size 1210044

fmsp_epoch_4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af20af16d2a5dbbe84a1545c2ff7b23ebe18a91b0b9bd56d1f70c8e76ebe6e29
+size 1210044

fmsp_epoch_5.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5412124d9859ddb1f42ed95ab95e9ea62a4be93f3127940022888a4816cce000
+size 1210044

fmsp_epoch_6.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55bd322bc2308b33e5dd3f852b76ab14101b1ce25c4c31d2701c0e4afd422f31
+size 1210044

fmsp_epoch_7.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ef586be462a0921f783271d768b56dc60fe633baa1e22140084ff9686166288
+size 1210044

fmsp_epoch_8.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c2e4784ca96fd58070820a42a1ee39ab97bdf5e39e9ca293beb1a57846a42df
+size 1210044

fmsp_epoch_9.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7909fd6dcb8f0eef12b77c53e11bffdb0db51f1c17c3c7ff4a4ba20482ed96d5
+size 1210044