caiovicentino1 commited on
Commit
ff04ca2
·
verified ·
1 Parent(s): 477c370

Add launch figures; final release tables

Browse files
.gitattributes CHANGED
@@ -34,3 +34,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ assets/chart_calibration.png filter=lfs diff=lfs merge=lfs -text
38
+ assets/chart_confidence.png filter=lfs diff=lfs merge=lfs -text
39
+ assets/chart_head_to_head.png filter=lfs diff=lfs merge=lfs -text
40
+ assets/chart_long_context.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/eikos_launch.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -23,6 +23,8 @@ tags:
23
 
24
  # Eikos-4B — open typed-decision model (System-1)
25
 
 
 
26
  Eikos-4B answers **typed decisions** about a given state in a **single forward pass**, with **calibrated
27
  probabilities** over a bounded set of options. It does not generate text at decision time.
28
 
@@ -155,9 +157,17 @@ one task family (trade-offs) and one topic (healthcare administration) were held
155
 
156
  \* generated by the same rule generator family used in training (different rules and cases); see notes.
157
 
 
 
 
 
158
  **How to read the table:**
159
  - **JevBench:** public items only, not the official leaderboard run.
160
  - **Jev:** measured through its API on our batteries, not distilled from.
 
 
 
 
161
  - **WCB (central-bank stance):** reported as balanced accuracy, because the classes are imbalanced.
162
  - **Rules suites:** "rules / new domain / rulebooks" come from the same generator family used in training, so
163
  they measure learning of that generator. Transfer to rules the model never saw is measured by "Trade (unseen
@@ -166,6 +176,13 @@ one task family (trade-offs) and one topic (healthcare administration) were held
166
  The other 108 items use a rule combination held out of training (an exception over a business-day window):
167
  27B 91.7, 4B 90.7. That combination appears in only 6 of the ~6,000 rule rows used in training.
168
 
 
 
 
 
 
 
 
169
  ## Release builds
170
 
171
  Final evaluation of every published build, run on the exact files in these repos: 7 suites, 7,371 items, vLLM 0.30
@@ -206,6 +223,8 @@ of answers unchanged.
206
  ship T = 1.
207
  - **Changing it:** `calib.json` holds the temperature, and callers can re-fit it on their own data.
208
 
 
 
209
  ## Parallelism and speed
210
 
211
  - **Same state, many questions:** the hybrid prefix cache processes the state once.
 
23
 
24
  # Eikos-4B — open typed-decision model (System-1)
25
 
26
+ ![Eikos overview: JevBench hard accuracy, error when at least 90% confident and long context, against Jev and Laya](https://huggingface.co/caiovicentino1/Eikos-4B/resolve/main/assets/eikos_launch.png)
27
+
28
  Eikos-4B answers **typed decisions** about a given state in a **single forward pass**, with **calibrated
29
  probabilities** over a bounded set of options. It does not generate text at decision time.
30
 
 
157
 
158
  \* generated by the same rule generator family used in training (different rules and cases); see notes.
159
 
160
+ ![Same items, same scoring: accuracy per benchmark for Eikos-27B, Eikos-4B, Jev and Laya](https://huggingface.co/caiovicentino1/Eikos-4B/resolve/main/assets/chart_head_to_head.png)
161
+
162
+ ![Confidence you can act on: error among the decisions each system takes on its own as its confidence threshold drops](https://huggingface.co/caiovicentino1/Eikos-4B/resolve/main/assets/chart_confidence.png)
163
+
164
  **How to read the table:**
165
  - **JevBench:** public items only, not the official leaderboard run.
166
  - **Jev:** measured through its API on our batteries, not distilled from.
167
+ - **Laya** (charts only): run with its official package; its JevBench scores are within one item of its official
168
+ leaderboard results.
169
+ - **Charts:** 7,140 items common to all four systems. The confidence chart shows, for every confidence threshold,
170
+ how many decisions a system would take on its own and how often those decisions are wrong.
171
  - **WCB (central-bank stance):** reported as balanced accuracy, because the classes are imbalanced.
172
  - **Rules suites:** "rules / new domain / rulebooks" come from the same generator family used in training, so
173
  they measure learning of that generator. Transfer to rules the model never saw is measured by "Trade (unseen
 
176
  The other 108 items use a rule combination held out of training (an exception over a business-day window):
177
  27B 91.7, 4B 90.7. That combination appears in only 6 of the ~6,000 rule rows used in training.
178
 
179
+ ### Long context
180
+
181
+ The case file is hidden inside unrelated text at the start, middle or end of the prompt (80 public JevBench
182
+ decisions per point; released checkpoints served with vLLM 0.30).
183
+
184
+ ![Decisions buried in long context: accuracy from about 1k to 64k tokens](https://huggingface.co/caiovicentino1/Eikos-4B/resolve/main/assets/chart_long_context.png)
185
+
186
  ## Release builds
187
 
188
  Final evaluation of every published build, run on the exact files in these repos: 7 suites, 7,371 items, vLLM 0.30
 
223
  ship T = 1.
224
  - **Changing it:** `calib.json` holds the temperature, and callers can re-fit it on their own data.
225
 
226
+ ![Probabilities you can trust: reliability diagram and expected calibration error on 7,140 items](https://huggingface.co/caiovicentino1/Eikos-4B/resolve/main/assets/chart_calibration.png)
227
+
228
  ## Parallelism and speed
229
 
230
  - **Same state, many questions:** the hybrid prefix cache processes the state once.
SHA256SUMS CHANGED
@@ -1,7 +1,12 @@
1
  2cf8c18cf63fd0e31addba3775a2bbec6f58aa31bb5bdfb549c640bd9b676611 LICENSE
2
  bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a LICENSE-Qwen
3
  281b5c258f113cf0fd53df3bd20868eb8b3c63aef37e6496d1c172e468b3a073 NOTICE
4
- 2d66cef673cf6939bbbac9a372b732fb8508cc8fb09e23154d570555a44e5aca README.md
 
 
 
 
 
5
  2f1645391d393a8fdf77661f765640ecb0f850fd1a16025ade22ea737a7b2686 calib.json
6
  a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715 chat_template.jinja
7
  ddc63e1c717afa86c865bb5e01313d89d72bb53b97ad4a8a03ba8510c0621670 config.json
 
1
  2cf8c18cf63fd0e31addba3775a2bbec6f58aa31bb5bdfb549c640bd9b676611 LICENSE
2
  bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a LICENSE-Qwen
3
  281b5c258f113cf0fd53df3bd20868eb8b3c63aef37e6496d1c172e468b3a073 NOTICE
4
+ 43702d71b04929ed28329eef7d3762a29fb2e9f4b4ee5485b3ca96e79e60f27f README.md
5
+ 08cbc176f11afd74bfd87ae095ad68665d659ad67d06c85782ffd4d77169f8b4 assets/chart_calibration.png
6
+ f75c9334eff9de54322e655b0e6c22b1fcb776154ac0fc417ed87700f388a164 assets/chart_confidence.png
7
+ f1cbab3e2e80d733ed844884eaddfdcde7779c67cbc9ca43fad3b50484780549 assets/chart_head_to_head.png
8
+ c6074e2c06ed7c2bde4b65d3292109b92f7e267800b22d40e755fb3fa9757234 assets/chart_long_context.png
9
+ 09bec078d3fd7e77e70498365d79d602259917f3d6342dbec961148a3c31b796 assets/eikos_launch.png
10
  2f1645391d393a8fdf77661f765640ecb0f850fd1a16025ade22ea737a7b2686 calib.json
11
  a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715 chat_template.jinja
12
  ddc63e1c717afa86c865bb5e01313d89d72bb53b97ad4a8a03ba8510c0621670 config.json
assets/chart_calibration.png ADDED

Git LFS Details

  • SHA256: 08cbc176f11afd74bfd87ae095ad68665d659ad67d06c85782ffd4d77169f8b4
  • Pointer size: 131 Bytes
  • Size of remote file: 410 kB
assets/chart_confidence.png ADDED

Git LFS Details

  • SHA256: f75c9334eff9de54322e655b0e6c22b1fcb776154ac0fc417ed87700f388a164
  • Pointer size: 131 Bytes
  • Size of remote file: 531 kB
assets/chart_head_to_head.png ADDED

Git LFS Details

  • SHA256: f1cbab3e2e80d733ed844884eaddfdcde7779c67cbc9ca43fad3b50484780549
  • Pointer size: 131 Bytes
  • Size of remote file: 334 kB
assets/chart_long_context.png ADDED

Git LFS Details

  • SHA256: c6074e2c06ed7c2bde4b65d3292109b92f7e267800b22d40e755fb3fa9757234
  • Pointer size: 131 Bytes
  • Size of remote file: 320 kB
assets/eikos_launch.png ADDED

Git LFS Details

  • SHA256: 09bec078d3fd7e77e70498365d79d602259917f3d6342dbec961148a3c31b796
  • Pointer size: 132 Bytes
  • Size of remote file: 1.22 MB