MwSpace Srl commited on
Commit
9ea83d6
Β·
verified Β·
1 Parent(s): ca9169b

rename to Skylar-390M-Cobol + benchmark chart

Browse files
Files changed (1) hide show
  1. README.md +10 -8
README.md CHANGED
@@ -13,7 +13,7 @@ library_name: transformers
13
  pipeline_tag: text-generation
14
  ---
15
 
16
- # SkylarCobol-390M
17
 
18
  **A 390M-parameter, from-scratch language model specialized in COBOL β€” small enough to run
19
  locally and on-prem, trained entirely in-house, that outperforms 7B general code models on
@@ -41,17 +41,19 @@ COBOL code generation.**
41
  All models evaluated on the **same harness**, greedy decoding, seed 0, GnuCOBOL compile+execute,
42
  official `{NAME}.TXT` scoring.
43
 
44
- Ordered by pass@1 (the benchmark). SkylarCobol is **#1 among all deployable models** β€” only the
45
  much larger reference teacher scores higher.
46
 
47
  | Model | Params | CSR (compile rate) | pass@1 |
48
  |---|---:|---:|---:|
49
  | *Claude Opus 4.8 (reference ceiling, not deployable)* | β€” | *96.6%* | *81.5%* |
50
- | **SkylarCobol-390M (this model)** | **390M** | **82.2%** | **5.5%** |
51
  | Qwen2.5-Coder-7B-Instruct | 7B | 6.2% | 2.1% |
52
  | CodeLlama-7B-Instruct | 7B | 6.8% | 0.7% |
53
  | StarCoder2-7B | 7B | 48.6% | 0.0% |
54
 
 
 
55
  <sub>pass@1 = 8/146 problems solved. 95% binomial CI β‰ˆ [2.4%, 10.5%] β€” low absolute, as expected for
56
  a 390M model on an execution benchmark; the point is the *relative* result vs the 7B baselines on the
57
  identical harness.</sub>
@@ -63,7 +65,7 @@ specialized 390M wins.
63
 
64
  **What the numbers mean.** The general 7B models often write *pseudo-COBOL* β€” they call `mod()`,
65
  `substr()`, `abs()` (functions from Python/Java that don't exist in COBOL), so their code looks
66
- right but doesn't compile. SkylarCobol writes *real* COBOL: it compiles 82% of the time and solves
67
  the most problems. This is the value of specialization.
68
 
69
  ## Decontamination
@@ -102,7 +104,7 @@ is in progress β€” that is the version intended to become a genuinely usable COB
102
 
103
  ## How to use it β€” a COBOL **completer**, not a chatbot
104
 
105
- **SkylarCobol-390M completes COBOL; it does not chat.** You give it a COBOL *stub* β€” a skeleton
106
  with the task written as comments, ending at `WORKING-STORAGE SECTION.` β€” and it writes the
107
  `WORKING-STORAGE` entries + `PROCEDURE DIVISION`. Do **not** prompt it with free-form natural
108
  language ("write me a program that…", and especially not in Italian): it will ramble, not code.
@@ -115,7 +117,7 @@ skylar cobol --stub-file my_task.cbl --compile # your own stub β†’ completed +
115
 
116
  ```python
117
  import skylar
118
- m = skylar.load("Sophia-AI/SkylarCobol-390M")
119
  print(m.complete_cobol(my_stub)) # reassembles a full, compilable program
120
  ```
121
 
@@ -134,7 +136,7 @@ print(m.complete_cobol(my_stub)) # reassembles a full, compilable pro
134
  * >>> max_element([5, 3, -5]) -> 5
135
  WORKING-STORAGE SECTION.
136
 
137
- *> SkylarCobol COMPLETES it:
138
  01 WS-I PIC 9(4).
139
  PROCEDURE DIVISION USING LINKED-ITEMS.
140
  MOVE L-L(1) TO RESULT
@@ -154,7 +156,7 @@ but the stub format above is where it's strongest.
154
  review the output.
155
 
156
  `import skylar` also registers the architecture with πŸ€— Transformers, so
157
- `AutoModelForCausalLM.from_pretrained("Sophia-AI/SkylarCobol-390M")` works too. The weights
158
  (`model.safetensors`) and `tokenizer.json` are standard; the architecture is a custom decoder
159
  (`NanoTransformer`, `model_type: nano-transformer` β€” Qwen3-style RMSNorm + RoPE + GQA + QK-Norm +
160
  SwiGLU), described in full in the accompanying paper.
 
13
  pipeline_tag: text-generation
14
  ---
15
 
16
+ # Skylar-390M-Cobol
17
 
18
  **A 390M-parameter, from-scratch language model specialized in COBOL β€” small enough to run
19
  locally and on-prem, trained entirely in-house, that outperforms 7B general code models on
 
41
  All models evaluated on the **same harness**, greedy decoding, seed 0, GnuCOBOL compile+execute,
42
  official `{NAME}.TXT` scoring.
43
 
44
+ Ordered by pass@1 (the benchmark). Skylar-390M-Cobol is **#1 among all deployable models** β€” only the
45
  much larger reference teacher scores higher.
46
 
47
  | Model | Params | CSR (compile rate) | pass@1 |
48
  |---|---:|---:|---:|
49
  | *Claude Opus 4.8 (reference ceiling, not deployable)* | β€” | *96.6%* | *81.5%* |
50
+ | **Skylar-390M-Cobol (this model)** | **390M** | **82.2%** | **5.5%** |
51
  | Qwen2.5-Coder-7B-Instruct | 7B | 6.2% | 2.1% |
52
  | CodeLlama-7B-Instruct | 7B | 6.8% | 0.7% |
53
  | StarCoder2-7B | 7B | 48.6% | 0.0% |
54
 
55
+ ![COBOLEval β€” Skylar-390M-Cobol vs 7B code models, same harness](coboleval_benchmark.png)
56
+
57
  <sub>pass@1 = 8/146 problems solved. 95% binomial CI β‰ˆ [2.4%, 10.5%] β€” low absolute, as expected for
58
  a 390M model on an execution benchmark; the point is the *relative* result vs the 7B baselines on the
59
  identical harness.</sub>
 
65
 
66
  **What the numbers mean.** The general 7B models often write *pseudo-COBOL* β€” they call `mod()`,
67
  `substr()`, `abs()` (functions from Python/Java that don't exist in COBOL), so their code looks
68
+ right but doesn't compile. Skylar-390M-Cobol writes *real* COBOL: it compiles 82% of the time and solves
69
  the most problems. This is the value of specialization.
70
 
71
  ## Decontamination
 
104
 
105
  ## How to use it β€” a COBOL **completer**, not a chatbot
106
 
107
+ **Skylar-390M-Cobol completes COBOL; it does not chat.** You give it a COBOL *stub* β€” a skeleton
108
  with the task written as comments, ending at `WORKING-STORAGE SECTION.` β€” and it writes the
109
  `WORKING-STORAGE` entries + `PROCEDURE DIVISION`. Do **not** prompt it with free-form natural
110
  language ("write me a program that…", and especially not in Italian): it will ramble, not code.
 
117
 
118
  ```python
119
  import skylar
120
+ m = skylar.load("Sophia-AI/Skylar-390M-Cobol")
121
  print(m.complete_cobol(my_stub)) # reassembles a full, compilable program
122
  ```
123
 
 
136
  * >>> max_element([5, 3, -5]) -> 5
137
  WORKING-STORAGE SECTION.
138
 
139
+ *> Skylar-390M-Cobol COMPLETES it:
140
  01 WS-I PIC 9(4).
141
  PROCEDURE DIVISION USING LINKED-ITEMS.
142
  MOVE L-L(1) TO RESULT
 
156
  review the output.
157
 
158
  `import skylar` also registers the architecture with πŸ€— Transformers, so
159
+ `AutoModelForCausalLM.from_pretrained("Sophia-AI/Skylar-390M-Cobol")` works too. The weights
160
  (`model.safetensors`) and `tokenizer.json` are standard; the architecture is a custom decoder
161
  (`NanoTransformer`, `model_type: nano-transformer` β€” Qwen3-style RMSNorm + RoPE + GQA + QK-Norm +
162
  SwiGLU), described in full in the accompanying paper.