Add MicroT-test1-10K-SmolTalk2: FMSP model card + 10 epoch safetensors

README.md

@@ -0,0 +1,327 @@
+---
+license: apache-2.0
+library_name: pytorch
+language:
+  - en
+pipeline_tag: text-generation
+tags:
+  - transformer
+  - byte-level
+  - causal-lm
+  - attention
+  - rope
+  - decoder-only
+  - fmsp
+  - micro-language-model
+  - sub-1m-parameters
+model_name: MicroT-test1-10K-SmolTalk2
+datasets:
+  - HuggingFaceTB/smoltalk2
+  - llaa33219/small-qa-en-10k
+metrics:
+  - perplexity
+  - exact-match
+---
+
+<div align="center">
+
+<img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/main/logo.svg" width="300" alt="MicroMixer-4 Logo"/>
+
+# MicroT-test1-10K-SmolTalk2
+
+<img src="https://img.shields.io/badge/Parameters-9%2C808-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/>
+<img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/>
+<img src="https://img.shields.io/badge/Fine--tuning-FMSP-green?style=for-the-badge&color=%2300D620" alt="FMSP"/>
+
+<br/>
+<br/>
+
+<table>
+<tr>
+<td align="center" style="padding: 20px;">
+<strong style="color: #FF6600; font-size: 1.2em;">Micro Transformer — Reference Baseline</strong><br/><em>Vanilla Attention • RoPE • Byte-Level • Decoder-Only</em>
+</td>
+</tr>
+</table>
+
+[![GitHub](https://img.shields.io/badge/GitHub-MicroMixer--4-blue?style=for-the-badge&logo=github&color=%23007BFF)](https://github.com/llaa33219/MicroMixer-4)
+
+</div>
+
+<div style="background: linear-gradient(135deg, #FF660022, #007BFF22); padding: 20px; border-radius: 10px; border-left: 4px solid #FF6600;">
+
+## 📋 Overview
+
+**MicroT-test1-10K-SmolTalk2** is a **9,808-parameter** vanilla decoder-only **transformer** — multi-head causal self-attention with RoPE — pretrained on **SmolTalk2** conversation data (instead of the project's Discord-Dialogues baseline) and then fine-tuned with **FMSP** on 9,012 general-knowledge QA pairs.
+
+This is the **10K** member of the **MicroT-test1** family: the registered attention-based **reference baseline** of the MicroMixer-4 project, here rerun on SmolTalk2 as part of the **dataset-efficiency comparison study** — six parameter budgets × two architectures × two open pretraining corpora, all fine-tuned with the identical P05 FMSP recipe at seed 42. [Analysis](https://github.com/llaa33219/MicroMixer-4/blob/main/DATASET_COMPARISON_ANALYSIS.md).
+
+It is deliberately boring — the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
+
+</div>
+
+## 🏗️ Architecture
+
+<div align="center">
+
+```mermaid
+graph TD
+    A[Byte Input] --> B[Embed 256→16]
+    B --> C[Transformer Block × 2]
+    C --> D[RMSNorm]
+    D --> E[LM Head Tied with Embed]
+    E --> F[Byte Output]
+
+    subgraph "Transformer Block (pre-norm)"
+        X[Input 16] --> N1[RMSNorm]
+        N1 --> AT["MHA 1 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]
+        AT --> R1[+ residual]
+        R1 --> N2[RMSNorm]
+        N2 --> MLP["GELU MLP 16→56→16"]
+        MLP --> R2[+ residual]
+    end
+
+    style A fill:#007BFF,color:#fff
+    style F fill:#00D620,color:#fff
+    style AT fill:#FF6600,color:#fff
+```
+
+</div>
+
+### Model Configuration
+
+<table>
+<tr>
+<th style="background-color: #FF6600; color: white;">Parameter</th>
+<th style="background-color: #007BFF; color: white;">Value</th>
+</tr>
+<tr><td>Hidden Dimension (d_model)</td><td><code>16</code></td></tr>
+<tr><td>Attention Heads</td><td><code>1</code> (d_head = <code>16</code> at every size)</td></tr>
+<tr><td>Number of Blocks</td><td><code>2</code></td></tr>
+<tr><td>FFN Hidden</td><td><code>56</code></td></tr>
+<tr><td>Position Encoding</td><td>RoPE θ=10000 on q/k only (non-persistent buffers)</td></tr>
+<tr><td>Attention</td><td>Causal MHA via <code>F.scaled_dot_product_attention(is_causal=True)</code></td></tr>
+<tr><td>Activation</td><td><code>GELU</code></td></tr>
+<tr><td>Biases</td><td><b>None</b> — no bias parameters anywhere</td></tr>
+<tr><td>Normalization</td><td><code>RMSNorm</code> (pre-norm)</td></tr>
+<tr><td>Max Sequence Length</td><td><code>1024</code></td></tr>
+<tr><td>Vocabulary Size</td><td><code>256</code> (byte-level)</td></tr>
+<tr><td>Output Head</td><td>Tied with input embedding</td></tr>
+</table>
+
+### Core Components
+
+```
+┌──────────────────────────────────────────────┐
+│         Transformer Block (×2)               │
+│  h = h + MHA(RMSNorm(h))    # RoPE q/k, causal│
+│  h = h + MLP(RMSNorm(h))    # GELU d→ffn→d   │
+│  no biases, no flash, no tricks — vanilla     │
+└──────────────────────────────────────────────┘
+```
+
+The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.
+
+---
+
+## 🎯 Generation Examples
+
+<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">
+
+**Questions the model was trained on** (FMSP train set, 9,012 QA pairs — greedy decoding, `repetition_penalty=1.2`, `no_repeat_ngram_size=4`):
+
+```
+[Prompt] User: Who painted the Mona Lisa?
+Assistant:
+[Output] The in a stands and and and and and and and and and and and and and the the stands and and and …
+```
+
+<sub>degenerate — collapses into a repeating token loop; no real answer</sub>
+
+```
+[Prompt] User: Who painted The Starry Night?
+Assistant:
+[Output] The in a stands and and and and and and and and and and and and and the stands and and and and …
+```
+
+<sub>degenerate — collapses into a repeating token loop; no real answer</sub>
+
+
+**Questions the model has never seen and cannot answer** (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):
+
+```
+[Prompt] User: Who painted the Glimmering Frostberry?
+Assistant:
+[Output] The in a stands and and and and the stands and and the stands and and the stands and and the st…
+```
+
+<sub>degenerate — collapses into a repeating token loop on out-of-distribution input</sub>
+
+```
+[Prompt] User: Who composed the Symphony of Hollow Dawn?
+Assistant:
+[Output] The in a stands and in the stands and and and and and and and and and the stands and and and th…
+```
+
+<sub>degenerate — collapses into a repeating token loop on out-of-distribution input</sub>
+
+
+</div>
+
+---
+
+## 📊 Results
+
+<div style="background-color: #FF660015; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">
+
+### Pretraining (SmolTalk2, V76 recipe, 3 epochs)
+
+| Metric | 1 ep | 2 ep | 3 ep |
+|--------|------|------|------|
+| Val PPL | 6.50 | 6.28 | **6.04** |
+
+Identical recipe to the MicroMixer-4 mixer: AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42.
+
+### FMSP fine-tuning (small-qa-en-10k, P05 recipe, 10 epochs)
+
+| Metric | Value |
+|--------|-------|
+| Train QA pairs | 9,012 |
+| Held-out QA pairs | 988 |
+| Best-val checkpoint | `fmsp_epoch_9.safetensors` (val loss **1.7327**) |
+| freeze_fraction | 0.05 (true freeze) |
+| Loss | answer-only CE + probe KL (weight 0.5) |
+
+### Evaluation battery (post-FMSP)
+
+| Axis | MicroT-test1-10K-SmolTalk2 |
+|------|--------|
+| Chatter fluency d2 (cycles) | 0.447 (0/9) |
+| Full-988 EM (seed 42) | 0 |
+| Q-relevance echo / hijack % | 0.0 / 0.0 |
+| OOD hijack % | 0.0% |
+| Unanswerable fabrication /18 | 0 |
+| Discord PPL | 10.15 |
+
+<sub>Single-seed run (seed 42); the discord-pretrained cards report a 3-seed mean for Full-988 EM. ‡ where marked: degenerate-pass.</sub>
+
+### MicroT-test1 SmolTalk2 family (same protocol, all sizes)
+
+| Size | Params | 3ep Val PPL | Chatter d2 | Full-988 EM | qrel echo/hijack | OOD hijack |
+|------|--------|-------------|------------|-------------|------------------|------------|
+| 1M | 996,736 | 2.12 | 0.911 | 744 | 100.0 / 0.0 | 37.3% |
+| 500K | 498,528 | 2.32 | 0.932 | 580 | 86.0 / 7.0 | 42.4% |
+| 300K | 297,680 | 2.53 | 0.775 | 137 | 55.0 / 26.0 | 40.7% |
+| 100K | 97,872 | 3.13 | 0.673 | 0 | 5.0 / 86.0 | 55.9% |
+| 50K | 49,888 | 3.68 | 0.549 | 0 | 8.0 / 49.0 | 50.8% |
+| **10K** | 9,808 | 6.04 | 0.447 | 0 | 0.0 / 0.0 | 0.0% |
+
+<sub>Seed-42 single runs (pretrained on SmolTalk2; the discord-pretrained families report 3-seed EM means).</sub>
+
+</div>
+
+---
+
+## 📚 Training Data
+
+<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">
+
+1. **Pretraining**: [SmolTalk2](https://huggingface.co/datasets/HuggingFaceTB/smoltalk2) — 200K-cap sample of the `smoltalk_smollm3_smol_magpie_ultra_no_think` split (200K of 406,843; Apache 2.0), ~6-turn multi-turn conversations with no reasoning traces and system messages excluded, flattened to `User:/Assistant:` format, 1024-byte sequences, 3 epochs.
+2. **FMSP fine-tuning**: [small-qa-en-10k](https://huggingface.co/datasets/llaa33219/small-qa-en-10k) — 10K general-knowledge QA pairs (arts, science, history, geography, music…), split 9,012 train / 988 held-out. 10 epochs under the P05 recipe (5% of parameters frozen-true, answer-only CE, probe-KL 0.5).
+
+</div>
+
+---
+
+## 🔧 Usage
+
+### Files in this repository
+- `fmsp_epoch_{0..9}.safetensors` — per-epoch FMSP weights (pickle-free safetensors). **`fmsp_epoch_9.safetensors` is the best-val checkpoint** for this size.
+
+### Load and generate (local clone)
+
+```python
+import torch
+from safetensors.torch import load_file
+from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_10k
+from src.fmsp import attach_adapter
+from src.tokenizer import ByteTokenizer
+
+# Clone the code repository first:
+# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-4
+
+cfg = v88_transformer_10k()
+model = MicroMixerV88Transformer(cfg)
+attach_adapter(model, d_model=cfg.d_model, rank=16)   # FMSP adapter (trained weights are in the file)
+model.load_state_dict(load_file("fmsp_epoch_9.safetensors"), strict=True)
+model.eval()
+
+tok = ByteTokenizer()
+prompt = "User: Who painted the Mona Lisa?\n\nAssistant: "
+ids = tok.encode(prompt)
+if ids and ids[-1] == tok.eos_token_id:
+    ids = ids[:-1]                       # ByteTokenizer appends EOS; the prompt must end open
+ids = torch.tensor([ids])
+with torch.no_grad():
+    out = model.generate(
+        ids, max_new_tokens=200,
+        temperature=0.0,                 # greedy — used for all reported numbers
+        repetition_penalty=1.2,
+        no_repeat_ngram_size=4,
+        eos_token_id=tok.eos_token_id,
+    )
+print(tok.decode(out[0].tolist()))
+```
+
+### Load from Hugging Face Hub (no clone of the weights needed)
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_10k
+from src.fmsp import attach_adapter
+
+REPO = "llaa33219/MicroT-test1-10K-SmolTalk2"
+
+cfg = v88_transformer_10k()
+model = MicroMixerV88Transformer(cfg)
+attach_adapter(model, d_model=cfg.d_model, rank=16)
+model.load_state_dict(
+    load_file(hf_hub_download(REPO, "fmsp_epoch_9.safetensors")), strict=True)
+model.eval()
+# ... generate as above
+```
+
+---
+
+## ⚠️ Limitations
+
+<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;">
+
+| Limitation | Description |
+|------------|-------------|
+| **Micro parameters** | 9,808 parameters; capacity is the binding constraint on every axis |
+| **Reference baseline, not a product** | Exists to score the Mixer against attention at matched budget |
+| **Knows only what it memorized** | Knowledge is limited to the 9,012 trained QA pairs + the pretraining-corpus distribution |
+| **Does not abstain** | Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above |
+| **Byte-level noise** | 256-vocab byte tokenizer; PPL not comparable to BPE baselines |
+| **Research use only** | Architecture/scaling research artifact, not a production model |
+
+</div>
+
+---
+
+## 🧬 Context
+
+This is the **10K** SmolTalk2-pretrained arm of the **dataset-efficiency comparison study** (24 arms: {V87, V88} × {UltraChat, SmolTalk2} × six sizes) in the [MicroMixer-4](https://github.com/llaa33219/MicroMixer-4) project. Each arm swaps the Discord-Dialogues pretraining baseline for an open corpus (SmolTalk2) and reruns the identical P05 FMSP recipe at seed 42. Sibling repos: `llaa33219/MicroMixer-4-{1M..10K}-{UltraChat,SmolTalk2}` and `llaa33219/MicroT-test1-{1M..10K}-{UltraChat,SmolTalk2}`; the discord-pretrained baselines are `llaa33219/MicroMixer-4-{1M..10K}` and `llaa33219/MicroT-test1-{1M..10K}`. Full analysis: [DATASET_COMPARISON_ANALYSIS.md](https://github.com/llaa33219/MicroMixer-4/blob/main/DATASET_COMPARISON_ANALYSIS.md).
+
+---
+
+<div align="center">
+
+[![GitHub](https://img.shields.io/badge/Back_to_Repository-MicroMixer--4-blue?style=for-the-badge&logo=github&color=%23007BFF)](https://github.com/llaa33219/MicroMixer-4)
+
+<sub>Part of the <a href="https://github.com/llaa33219/MicroMixer-4">MicroMixer-4</a> research project — V88 transformer reference (MicroT-test1), 10K preset, SmolTalk2 pretraining</sub>
+
+</div>

fmsp_epoch_0.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e9199f2a06de58ec5456af0239900e7046d26e8ad94b0fb44d6cb9cac5d07a7
+size 43008

fmsp_epoch_1.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0001d50041ad270b3f7611956a5d710753401112c1a186327cbb4345dd0a1918
+size 43008

fmsp_epoch_2.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11c1f60f46597588ce7db21f715ee606595dd9dd90c8ce499199f12234effceb
+size 43008

fmsp_epoch_3.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:646eeda3482ebe8cc4cbf86d5b8f00bb410d4c46d43376de90363479440a50e6
+size 43008

fmsp_epoch_4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d3397b9be799b1b733b563cd1ad169e99a02853e19bea4fb1448d379facb8f5
+size 43008

fmsp_epoch_5.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4bf52c1fe895ae14efdfbfaa01764ec7026460a3f258daf40040edfacadacd9
+size 43008

fmsp_epoch_6.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09367733c52a544f9fc662172690316ee45e7d983ddb8f7debe6da05596e1457
+size 43008

fmsp_epoch_7.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b28a12a38089b877ee537f0a3d7db2ff5fb71d8f39842cc607cc88d28019a618
+size 43008

fmsp_epoch_8.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ab5e1b11830ff2dfaba54dd31bd81fbc1fb79701aa5410c10e91ad0f8fa1de1
+size 43008

fmsp_epoch_9.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:029d50023b583cfde38930c5004f3629a8d827fa53037426ad403b4ed48e9120
+size 43008