SkylarCobol-390M v0.1 — from-scratch COBOL specialist (COBOLEval CSR 82.2% / pass@1 5.5%, beats 7B baselines)

README.md

@@ -0,0 +1,129 @@
+---
+license: apache-2.0
+language:
+  - en
+tags:
+  - cobol
+  - code
+  - code-generation
+  - legacy-modernization
+  - sovereign-ai
+  - from-scratch
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# SkylarCobol-390M
+
+**A 390M-parameter, from-scratch language model specialized in COBOL — small enough to run
+locally and on-prem, trained entirely in-house, that outperforms 7B general code models on
+COBOL code generation.**
+
+> ⚠️ **Research preview (v0.1).** This is an honest proof-of-concept, **not** a production tool.
+> It demonstrates that a small, sovereign, specialized model can beat much larger generalist
+> code LLMs on a niche legacy language. Read the **Limitations** before using it. Numbers are
+> reported with full methodology and a decontamination audit — no hidden asterisks.
+
+## TL;DR
+
+- **~390M parameters** (386M exact), decoder-only, Qwen3-style architecture, trained **100% from
+  scratch** (no warm-start from anyone else's weights).
+- Domain: **COBOL** — the language that still runs banking, insurance and public-administration
+  back-ends, where general code models perform poorly and skilled developers are retiring.
+- **Sovereign**: runs on a single local GPU, no data leaves your infrastructure, no dependency on
+  any external LLM provider.
+- On **COBOLEval** (146 HumanEval-derived COBOL problems, compiled and executed with GnuCOBOL),
+  it **beats Qwen2.5-Coder-7B, CodeLlama-7B and StarCoder2-7B on pass@1** — at **18× fewer
+  parameters**.
+
+## Results — head-to-head on COBOLEval
+
+All models evaluated on the **same harness**, greedy decoding, seed 0, GnuCOBOL compile+execute,
+official `{NAME}.TXT` scoring.
+
+| Model | Params | CSR (compile rate) | pass@1 |
+|---|---:|---:|---:|
+| **SkylarCobol-390M (this model)** | **390M** | **82.2%** | **5.5%** |
+| Qwen2.5-Coder-7B-Instruct | 7B | 6.2% | 2.1% |
+| CodeLlama-7B-Instruct | 7B | 6.8% | 0.7% |
+| StarCoder2-7B | 7B | 48.6% | 0.0% |
+| *Claude Opus 4.8 (reference ceiling)* | — | *96.6%* | *81.5%* |
+
+<sub>pass@1 = 8/146 problems solved. 95% binomial CI ≈ [2.4%, 10.5%] — low absolute, as expected for
+a 390M model on an execution benchmark; the point is the *relative* result vs the 7B baselines on the
+identical harness.</sub>
+
+**Why this comparison is trustworthy:** our measured baseline scores match the published
+literature (StarCoder2 ~0%, CodeLlama ~0%, Qwen2.5-Coder ~0.68% on COBOLEval). The harness does
+**not** favor us — it reproduces the competitors' official numbers, and on that same harness the
+specialized 390M wins.
+
+**What the numbers mean.** The general 7B models often write *pseudo-COBOL* — they call `mod()`,
+`substr()`, `abs()` (functions from Python/Java that don't exist in COBOL), so their code looks
+right but doesn't compile. SkylarCobol writes *real* COBOL: it compiles 82% of the time and solves
+the most problems. This is the value of specialization.
+
+## Decontamination
+
+We audited overlap between training data and the COBOLEval test set. 4 problems
+(`below_zero`, `count_upper`, `find_max`, `is_sorted`) appeared in both; we **removed them from the
+training set and retrained** so the released model has never seen any test problem. The numbers
+above (5.5% / 82.2%) are reported on this clean model — and as a sanity check, **none of the 4
+formerly-overlapping problems are among the solved ones**, confirming the score carries no leakage.
+
+## Intended use & limitations
+
+**Intended use:** a *supervised assistant* for COBOL developers — scaffolding, boilerplate,
+DIVISION/LINKAGE structure, syntax, completion — where a human reviews the output.
+
+**Limitations (read this):**
+- **pass@1 is 5.5% in absolute terms — low.** COBOLEval is hard (real execution), but this model
+  is **not** capable of autonomously writing correct COBOL programs end-to-end. It is **capacity-
+  limited**: it compiles well (82%) but its algorithmic reasoning is weak.
+- **Do not** use it unsupervised or for safety-critical code.
+- English/COBOL only; not a general chat model.
+
+## How it was built
+
+1. **Pretrain (from scratch):** 8.19B tokens, mix of permissive code (oversampled), Italian/EU
+   legal text, and COBOL, Chinchilla-optimal, code-aware BPE tokenizer (vocab 32768).
+2. **SFT:** instruction tuning on COBOL completion, with **verified distillation** — a strong
+   teacher solves COBOL problems, each solution is kept **only if it compiles and passes its
+   tests**, then used as training data. Plus targeted synthetic data for structural gaps.
+3. **Decontamination + clean retrain** (see above).
+
+## Roadmap
+
+This 390M is the **first step**. A larger model (~1B) with more code pretraining and longer context
+is in progress — that is the version intended to become a genuinely usable COBOL assistant.
+
+## Usage
+
+The easiest way — the `skylar` package (CLI + loader, auto-downloads the weights):
+
+```bash
+pip install skylar
+skylar cobol --example          # complete a sample COBOL stub into a full program
+skylar chat                     # interactive REPL
+skylar generate --prompt "..."  # one-shot
+```
+
+```python
+import skylar
+m = skylar.load("Sophia-AI/SkylarCobol-390M")
+print(m.complete_cobol(my_stub))     # reassembles a full, compilable program
+```
+
+`import skylar` also registers the architecture with 🤗 Transformers, so
+`AutoModelForCausalLM.from_pretrained("Sophia-AI/SkylarCobol-390M")` works too. The weights
+(`model.safetensors`) and `tokenizer.json` are standard; the architecture is a custom decoder
+(`NanoTransformer`, `model_type: nano-transformer` — Qwen3-style RMSNorm + RoPE + GQA + QK-Norm +
+SwiGLU), described in full in the accompanying paper.
+
+**Note** — this model expects a COBOLEval-style stub (a fixed-format skeleton with the task as
+comments, ending at `WORKING-STORAGE SECTION.`); `skylar`'s `complete_cobol` / `skylar cobol` wrap
+the prompt and reassemble the program for you.
+
+## License & attribution
+
+Apache-2.0. IP: A. Ivanovitch (Sophia AI). Built in-house, no third-party pretrained weights.

config.json

@@ -0,0 +1,25 @@
+{
+  "_d_head": 128,
+  "architectures": [
+    "NanoTransformer"
+  ],
+  "bias": false,
+  "bos_token_id": 1,
+  "d_ff": 3456,
+  "d_model": 1280,
+  "dropout": 0.1,
+  "eos_token_id": 2,
+  "max_seq_len": 16384,
+  "model_type": "nano-transformer",
+  "mup_base_d_model": null,
+  "n_heads": 10,
+  "n_kv_heads": 2,
+  "n_layers": 20,
+  "pad_token_id": 0,
+  "qk_norm": true,
+  "rope_theta": 1000000.0,
+  "tie_weights": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.52.4",
+  "vocab_size": 32768
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2399c6f4098ec5900ff8a3a4281a22c6acfb0c784079a4f4e631e7fdd46585f
+size 1544287336