# COBOL SML — experiment findings (2026-09-03)

Honest write-up of a from-scratch tiny-model spike for GnuCOBOL, in the SML
(Smallest Language Model / TinkyBrain) family. Reviewed adversarially; this is the
corrected record, not the hype version.

## Question

Can a ~14M from-scratch model (MLX `MicroBrain`, the TinkyBrain architecture)
write compiling GnuCOBOL, if the *system around it* (grounding / symbolic repair /
compiler verification) does the heavy lifting? I.e. is the model a usable
**component** even if it's weak alone?

## Setup

- Data: 1,236 train / 65 val prompt→COBOL pairs (deterministic drills from the 7B
  project). Free-format target; `cobc -free -c` as ground truth.
- Tokenizer: COBOL-aware word/symbol, case-preserving, `<nl>`-structured. Round-trip
  fidelity on the corpus: **400/400 lossless** (tokenize→detokenize→compile).
- Model: `d_model 512 · 8h · 6L · d_ff 1024 · seq 512 · vocab 1431` ≈ 14.3M params.

## Results (all on the same 65-item val set)

| Variant / call | metric | value |
|---|---|---|
| v1 (30 ep), 1 greedy shot | single-output compile | **0%** |
| v1, greedy + symbolic repair | single-output compile | **6.2%** |
| v1, 8 branches × {raw,repair}, first-that-builds | **pass@16 (compiler-selected)** | **33.8% (22/65)** |
| v2 (80 ep, more training) | pass@16-ish | worse (overfit) |
| v3 (46.6M, 3× capacity) | — | not run (killed: scaling dilutes) |

## What is and isn't true

**True:**
- The model learns COBOL *structure* well (division order, `PERFORM VARYING`,
  `OCCURS`, `IF/END-IF`, `GOBACK`) but drops 1–2 mechanical tokens per program
  (unbalanced `PIC 9(4)`, missing `PROGRAM-ID` name). Compilation is all-or-nothing,
  so single-shot is 0%.
- A branch + symbolic-repair + compiler-verify **call** raises the same weights from
  0% to 33.8% pass@16 on this set. The lift is the *search scaffold*, not the model.
- More training (v2) and more capacity (v3) do **not** help — the signal is
  saturated at v1's size; scaling overfits/dilutes. (Consistent with the data being
  fully extracted.)

**Not true / withdrawn:**
- ❌ "33.8% compile-rate" — it is **pass@16** (up to 16 compiler-checked candidates),
  not pass@1. Honest single-output numbers are 0% / 6.2%.
- ❌ "Beats a 7B" — the 7B's 28.8% was **single-shot**; under this same
  16-candidate+repair pipeline a 7B would almost certainly exceed 33.8%. Withdrawn.
- ❌ "Generalizes" — the val set is ~94% near-duplicate of train. Success split
  (audited): **21/61 near-duplicate vs 1/4 novel**. The novel set (n=4) is noise.
  The number reflects memorization + scaffold, not generalization.
- Compilation ≠ correctness: exact-match to reference programs was **0%**.

## Adversarial review

Panel via `kist --dig` (Claude, Codex, GLM; DeepSeek/MiMo unavailable). Unanimous:
the headline was pass@k mislabeled as compile-rate, an unfair cross-size comparison,
and uninterpretable under contamination. The card and this note reflect their
corrections. This is the review gate working as intended — it caught the overclaim
before publication.

## Honest conclusion

A 14M from-scratch model on ~1.2k contaminated drills is **not a working COBOL
author** and this experiment does **not** establish that a tiny model rivals a large
one. It *does* show, on an in-distribution set, that call-strategy (branch + repair +
verify) can lift a near-0% component by an order of magnitude — an interesting
*method* result, nothing more.

## What would make the numbers mean something

1. **Template-disjoint, deduplicated held-out test** (hold out entire op/structure
   families from training) — the only way to get a real generalization number.
2. **Matched-protocol comparison** — run every model (incl. the 7B) under identical
   branch + repair + verify, and report pass@1 and pass@k side by side.
3. **Functional eval** — execute against expected output (pass@1 correctness), not
   just compilation.
4. Report N, seeds, and confidence intervals (differences here are ~3 programs).

## Artifacts

`bakes/cobol-sml-v1/` (weights) · `sml_call.py` (the studied call) · `cobol_repair.py`
· `cobol_sml_mcp.py` (MCP component) · `eval_gate2.py` / `eval_repair.py` /
`eval_system.py` / `audit_clean_split.py` (the evals that produced the honest
numbers above) · `fidelity_check.py`.