Upload Summer-0.5B-Chat

README.md

@@ -26,7 +26,7 @@ starts from `N(0, 0.02)`.
 
 ## Stage
 
-**Chat (v9)** — single-stage post-training from `Summer-0.5B-S0` (not from
+**Chat (v9e2)** — single-stage post-training from `Summer-0.5B-S0` (not from
 `Summer-0.5B-S1`; the parallel-data anneal is a separate branch for the
 translation downstream, unrelated to becoming a general chat model).
 
@@ -61,9 +61,13 @@ of a dedicated full-loss warm-up stage.
   so no conversation is ever cut mid-turn. Loss is computed on assistant
   turns only.
 - **Learning rate**: single linear decay from 0.8x the pretraining peak LR
-  to 0, matching nanochat's `init_lr_frac=0.8` — identical hyperparameters to
-  v8, since only the data changed (single-variable comparison).
-- 4,768 steps.
+  to 0, matching nanochat's `init_lr_frac=0.8`, spread across the **full**
+  training run below (not decayed-then-restarted).
+- **9,536 steps = 2 epochs over the same `chat_v9` data** (v9 itself was
+  1 epoch, 4,768 steps). At 524M params / 625M post-training tokens
+  (~1.2 tokens/param), there was headroom to see the data twice without
+  the val loss plateauing — it kept dropping past where the 1-epoch run
+  ended (1.2459 → 1.1932 final).
 
 **Every input must start with `<bos>`**, same as `Summer-0.5B-S0`.
 
@@ -75,20 +79,20 @@ competitive result.
 
 ### Letter multiple-choice (nanochat's primary format)
 
-| | nanochat d20 | v8 (previous release) | v9 (this release) |
+| | nanochat d20 | v9 (1 epoch) | v9e2 (this release, 2 epochs) |
 |---|---|---|---|
-| ARC-Easy | 0.4033 | 0.4545 | 0.4440 |
-| ARC-Challenge | — | 0.3532 | 0.3584 |
-| MMLU | 0.3232 | 0.3328 | 0.3296 |
-| C-Eval | — (d20 has no Chinese) | 0.3682 | 0.3646 |
+| ARC-Easy | 0.4033 | 0.4440 | **0.4882** |
+| ARC-Challenge | — | 0.3584 | **0.3848** |
+| MMLU | 0.3232 | 0.3296 | **0.3455** |
+| C-Eval | — (d20 has no Chinese) | 0.3646 | **0.4043** |
 | Format-following (free_argmax_is_letter) | ~1.0 | 0.99-1.00 | 0.99-1.00 |
 
-**Flat vs v8** — every difference is within the metric's own sampling noise
-(e.g. MMLU's 14,042-item standard error is ~0.004, the observed 0.0032 gap is
-smaller than that). Expected: v9's three data additions target conversational
-behavior (identity, casual follow-ups, translation) that this benchmark
-doesn't measure, not the multiple-choice skill itself. Still beats d20 on
-every comparable metric, same as v8.
+**A second epoch over the identical data raised every letter-MC number by
+0.016-0.044** — a bigger jump than the v8→v9 data changes produced, and well
+outside sampling noise (e.g. MMLU's 14,042-item standard error is ~0.004;
+the observed gap is 4x that). Makes sense: repeated exposure reinforces the
+"pick a letter correctly" multiple-choice skill more directly than adding new
+data varieties does. Still beats d20 on every comparable metric.
 
 ### Stop rate / long-form generation (`repetition_penalty=1.15`, not greedy)
 
@@ -100,12 +104,12 @@ found empirically to be the deployment sweet spot (see this project's
 
 | | en natural-stop | en long-form success | zh natural-stop | zh long-form success |
 |---|---|---|---|---|
-| v8 | 99% | 64% | 79% | 24% |
-| v9 | 100% | 68% | 84% | 25% |
+| v9 (1 epoch) | 100% | 68% | 84% | 25% |
+| v9e2 (2 epochs) | 100% | 74% | 82% | 24% |
 
-Small, consistently-positive movement on all four numbers (n=100 per
-language per release, standard error ~4-5 points each) — not a decisive win
-on its own, but a useful cross-check: nothing regressed.
+English long-form success improved (68%→74%); Chinese is flat within noise
+(n=100 per language, standard error ~4-5 points). Not a clean sweep, but no
+regression either.
 "Long-form success" = >=150 tokens, repetition ratio <0.2, stopped naturally.
 **Chinese open-ended creative writing (poems, stories) is still the weakest
 spot** — it is more prone to repetition collapse than factual Q&A, a known,
@@ -113,23 +117,26 @@ unresolved limitation (root cause not fully identified; greedy decoding is
 the primary suspect, not the training recipe, per this project's own
 ablations — see `docs/POSTTRAIN.md`).
 
-### The three targeted fixes, checked by hand against v8's actual failures
+### The three targeted fixes from v9, re-checked after the second epoch
 
-Benchmarks above don't measure any of what v9 was built to fix, so these were
-checked directly, replaying the same prompts that failed on v8:
+v9 fixed identity and translation but not one specific failure mode
+(responding to a remark rather than a question); the second epoch re-sees
+the *same* data, so this was re-tested rather than assumed fixed:
 
-| Scenario | v8 | v9 |
+| Scenario | v9 (1 epoch) | v9e2 (2 epochs) |
 |---|---|---|
-| "Who are you?" | inconsistent across turns | consistently answers "Summer-0.5B" |
-| "Translate to English: 我是中国人" | ignored the instruction, produced an unrelated fabricated paragraph | **"I am Chinese."** — correct, on-topic |
-| Asked for investment advice, then "go on" | non-sequitur ("I'm an AI assistant, I can't express myself like a human") | stayed on-topic, offered more investment-related points (not a literal continuation of the same list, but topically correct) |
-| Given a bubble-sort explanation, then told "this looks like a pretty good model" (a remark, not a question) | drifted into an unrelated rambling "improved bubble sort" tangent | **still fails** — invented an unrelated "time-series analysis model" tangent, a different failure of the same underlying kind |
-
-**Three out of four fixed or improved; the fourth (responding appropriately
-to a passing remark rather than a question) is not solved.** 8,000 rows of
-WildChat data was evidently not enough coverage for this specific input
-shape. Reported here rather than glossed over — the goal is an accurate
-account of what changed, not a clean narrative.
+| "Who are you?" | consistently answers "Summer-0.5B" | still consistent (now also states the parameter count unprompted) |
+| "Translate to English: 我是中国人" | **"I am Chinese."** — correct | still correct |
+| Given a bubble-sort explanation, then told "this looks like a pretty good model" (a remark, not a question) | invented an unrelated "time-series analysis model" tangent | **still fails** — this time it stayed on the bubble-sort topic but misread the remark as a request to critique/optimize the algorithm, rather than acknowledging the compliment |
+
+**Repeating the same 8,000 rows of WildChat data a second time did not fix
+the remaining gap — as expected.** This specific pattern (a bare compliment,
+no request attached) is naturally rare even in real conversation logs
+(WildChat mining found roughly 1% of candidate turns match it), so repetition
+reinforces already-thin signal rather than creating new signal. A synthetic,
+hand-written data source targeting this exact shape (analogous to how the
+`Identity` task itself is hand-written, not mined) is the planned next step,
+not included in this release.
 
 ### What this model is / is not good at (from manual testing, not benchmarked)
 
@@ -141,9 +148,9 @@ country counts, or basic arithmetic ("how many days in a week"). This tracks
 with what the letter-MC benchmark measures (does it follow instructions, pick
 a reasonable-sounding answer) rather than what it does not measure (is the
 answer actually correct) — the two are different questions, and this model
-was optimized against the former. Self-identification is now consistent
-(see above), but responding sensibly to a remark that isn't phrased as a
-question is still unreliable.
+was optimized against the former. Self-identification and translation
+requests are handled reliably; responding sensibly to a remark that isn't
+phrased as a question is still unreliable.
 
 ## Tokenizer
 

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6eefba1d7d5fe49208d39d5849d437ca73c529a11e88ca56a597e8b3fa3a7af4
+oid sha256:e694d507dca343bb68c56c3afcd480cec9a292854bdf2897d312247ace7bf433
 size 1048707384