MwSpace Srl commited on
Commit
ca9169b
Β·
verified Β·
1 Parent(s): af3eac8

card: COBOL completer (not chatbot) + working examples; results ordered by pass@1

Browse files
Files changed (1) hide show
  1. README.md +46 -11
README.md CHANGED
@@ -41,13 +41,16 @@ COBOL code generation.**
41
  All models evaluated on the **same harness**, greedy decoding, seed 0, GnuCOBOL compile+execute,
42
  official `{NAME}.TXT` scoring.
43
 
 
 
 
44
  | Model | Params | CSR (compile rate) | pass@1 |
45
  |---|---:|---:|---:|
 
46
  | **SkylarCobol-390M (this model)** | **390M** | **82.2%** | **5.5%** |
47
  | Qwen2.5-Coder-7B-Instruct | 7B | 6.2% | 2.1% |
48
  | CodeLlama-7B-Instruct | 7B | 6.8% | 0.7% |
49
  | StarCoder2-7B | 7B | 48.6% | 0.0% |
50
- | *Claude Opus 4.8 (reference ceiling)* | β€” | *96.6%* | *81.5%* |
51
 
52
  <sub>pass@1 = 8/146 problems solved. 95% binomial CI β‰ˆ [2.4%, 10.5%] β€” low absolute, as expected for
53
  a 390M model on an execution benchmark; the point is the *relative* result vs the 7B baselines on the
@@ -97,33 +100,65 @@ DIVISION/LINKAGE structure, syntax, completion β€” where a human reviews the out
97
  This 390M is the **first step**. A larger model (~1B) with more code pretraining and longer context
98
  is in progress β€” that is the version intended to become a genuinely usable COBOL assistant.
99
 
100
- ## Usage
101
 
102
- The easiest way β€” the `skylar` package (CLI + loader, auto-downloads the weights):
 
 
 
103
 
104
  ```bash
105
  pip install skylar
106
- skylar cobol --example # complete a sample COBOL stub into a full program
107
- skylar chat # interactive REPL
108
- skylar generate --prompt "..." # one-shot
109
  ```
110
 
111
  ```python
112
  import skylar
113
  m = skylar.load("Sophia-AI/SkylarCobol-390M")
114
- print(m.complete_cobol(my_stub)) # reassembles a full, compilable program
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
  ```
116
 
 
 
 
 
 
 
 
 
117
  `import skylar` also registers the architecture with πŸ€— Transformers, so
118
  `AutoModelForCausalLM.from_pretrained("Sophia-AI/SkylarCobol-390M")` works too. The weights
119
  (`model.safetensors`) and `tokenizer.json` are standard; the architecture is a custom decoder
120
  (`NanoTransformer`, `model_type: nano-transformer` β€” Qwen3-style RMSNorm + RoPE + GQA + QK-Norm +
121
  SwiGLU), described in full in the accompanying paper.
122
 
123
- **Note** β€” this model expects a COBOLEval-style stub (a fixed-format skeleton with the task as
124
- comments, ending at `WORKING-STORAGE SECTION.`); `skylar`'s `complete_cobol` / `skylar cobol` wrap
125
- the prompt and reassemble the program for you.
126
-
127
  ## License & attribution
128
 
129
  Apache-2.0. IP: A. Ivanovitch (Sophia AI). Built in-house, no third-party pretrained weights.
 
41
  All models evaluated on the **same harness**, greedy decoding, seed 0, GnuCOBOL compile+execute,
42
  official `{NAME}.TXT` scoring.
43
 
44
+ Ordered by pass@1 (the benchmark). SkylarCobol is **#1 among all deployable models** β€” only the
45
+ much larger reference teacher scores higher.
46
+
47
  | Model | Params | CSR (compile rate) | pass@1 |
48
  |---|---:|---:|---:|
49
+ | *Claude Opus 4.8 (reference ceiling, not deployable)* | β€” | *96.6%* | *81.5%* |
50
  | **SkylarCobol-390M (this model)** | **390M** | **82.2%** | **5.5%** |
51
  | Qwen2.5-Coder-7B-Instruct | 7B | 6.2% | 2.1% |
52
  | CodeLlama-7B-Instruct | 7B | 6.8% | 0.7% |
53
  | StarCoder2-7B | 7B | 48.6% | 0.0% |
 
54
 
55
  <sub>pass@1 = 8/146 problems solved. 95% binomial CI β‰ˆ [2.4%, 10.5%] β€” low absolute, as expected for
56
  a 390M model on an execution benchmark; the point is the *relative* result vs the 7B baselines on the
 
100
  This 390M is the **first step**. A larger model (~1B) with more code pretraining and longer context
101
  is in progress β€” that is the version intended to become a genuinely usable COBOL assistant.
102
 
103
+ ## How to use it β€” a COBOL **completer**, not a chatbot
104
 
105
+ **SkylarCobol-390M completes COBOL; it does not chat.** You give it a COBOL *stub* β€” a skeleton
106
+ with the task written as comments, ending at `WORKING-STORAGE SECTION.` β€” and it writes the
107
+ `WORKING-STORAGE` entries + `PROCEDURE DIVISION`. Do **not** prompt it with free-form natural
108
+ language ("write me a program that…", and especially not in Italian): it will ramble, not code.
109
 
110
  ```bash
111
  pip install skylar
112
+ skylar cobol --example # the built-in demo stub
113
+ skylar cobol --stub-file my_task.cbl --compile # your own stub β†’ completed + GnuCOBOL check
 
114
  ```
115
 
116
  ```python
117
  import skylar
118
  m = skylar.load("Sophia-AI/SkylarCobol-390M")
119
+ print(m.complete_cobol(my_stub)) # reassembles a full, compilable program
120
+ ```
121
+
122
+ **βœ… Tasks it handles reliably** β€” simple list / loop / accumulate / string work, e.g.
123
+ `max_element`, `sum_to_n`, `strlen`, `incr_list`, `is_palindrome`, `flip_case`,
124
+ `pairs_sum_to_zero`, `count_up_to`. Give it a stub shaped like these.
125
+
126
+ **βœ… Worked example β€” stub in β†’ compilable program out** (real output, `cobc` accepts it):
127
+
128
+ ```cobol
129
+ *> INPUT STUB (ends at WORKING-STORAGE SECTION.):
130
+ 01 LINKED-ITEMS.
131
+ 05 L-L OCCURS 3 TIMES INDEXED BY NI PIC S9(10).
132
+ 05 RESULT PIC S9(10).
133
+ * Return maximum element in the list.
134
+ * >>> max_element([5, 3, -5]) -> 5
135
+ WORKING-STORAGE SECTION.
136
+
137
+ *> SkylarCobol COMPLETES it:
138
+ 01 WS-I PIC 9(4).
139
+ PROCEDURE DIVISION USING LINKED-ITEMS.
140
+ MOVE L-L(1) TO RESULT
141
+ PERFORM VARYING WS-I FROM 2 BY 1 UNTIL WS-I > 3
142
+ IF L-L(WS-I) > RESULT MOVE L-L(WS-I) TO RESULT END-IF
143
+ END-PERFORM
144
+ GOBACK.
145
+ END PROGRAM MAX-ELEMENT.
146
  ```
147
 
148
+ **⚠️ Secondary mode:** a precise *technical* English instruction also works, e.g.
149
+ `skylar generate --prompt "Write a COBOL PROCEDURE DIVISION that sums WS-NUMBERS into WS-TOTAL and displays it."` β€”
150
+ but the stub format above is where it's strongest.
151
+
152
+ **❌ Don't:** chat with it, prompt in natural Italian, or expect correct programs for hard tasks
153
+ (pass@1 is 5.5% β€” it *compiles* ~82% of the time but its algorithmic reasoning is weak). Always
154
+ review the output.
155
+
156
  `import skylar` also registers the architecture with πŸ€— Transformers, so
157
  `AutoModelForCausalLM.from_pretrained("Sophia-AI/SkylarCobol-390M")` works too. The weights
158
  (`model.safetensors`) and `tokenizer.json` are standard; the architecture is a custom decoder
159
  (`NanoTransformer`, `model_type: nano-transformer` β€” Qwen3-style RMSNorm + RoPE + GQA + QK-Norm +
160
  SwiGLU), described in full in the accompanying paper.
161
 
 
 
 
 
162
  ## License & attribution
163
 
164
  Apache-2.0. IP: A. Ivanovitch (Sophia AI). Built in-house, no third-party pretrained weights.