RoeiG
/

laya-hebrew

@@ -105,21 +105,23 @@ Question types:
 Every evaluation set below was held out of training. he_bench v1 has 8 Hebrew tasks, each asked in 3 phrasings; the
 set is frozen by sha256. Brier and ECE are better when lower. "laya-multilingual" is Laya's published multilingual
-checkpoint. "Previous" is this project's previous checkpoint, trained without the reading and teacher data.
-| Metric | laya-multilingual | Previous | **Laya-Hebrew** |
 |---|---|---|---|
-| MASSIVE he, 20 intents (500) | 0.352 | 0.816 | **0.806** |
 | MASSIVE he, 4 intents | 0.710 | 0.928 | **0.938** |
-| MASSIVE en, 20 intents | 0.652 | 0.816 | **0.816** |
-| he_bench accuracy / Brier / ECE (4,290) | 0.439 / 0.701 / 0.266 | 0.527 / 0.538 / 0.119 | **0.595 / 0.499 / 0.125** |
 | Belebele-he reading comprehension (900; chance 0.25) | | 0.468 | **0.767** |
-| SIB-200-he topic | | 0.808 | **0.801** |
 | Yes/no as a question: P(yes \| true) − P(yes \| false) | | 0.047 | **0.379** |
-| Yes/no as a claim: same gap (390) | | 0.692 | **0.674** |
 | Hebrew BoolQ, held out (875) | | 0.611 | **0.838** |
 | Rule-direction probe (208) | | 0.798 | **0.832** |
-| Held-out soft-label set, Brier (900) | | 0.230 | **0.231** |
 he_bench accuracy by task:
@@ -131,8 +133,11 @@ he_bench accuracy by task:
 | sentiment | 0.633 | 0.33 |
 | winograd | 0.607 | 0.50 |
 | hellaswag | 0.447 | 0.25 |
-| tone arousal | 0.389 | 5 levels |
-| tone valence | 0.328 | 5 levels |
 **Run-to-run noise.** A second training seed, with the same data and settings, differed by these amounts:
 - he_bench overall: 0.2 points
@@ -216,4 +221,4 @@ and runtime are by Laya's authors (Apache-2.0).
 - Dicta, for NeoDictaBERT-bilingual and DictaLM 3.0
 - Laya's authors, for the architecture, the RLCD training method and the runtime
-- The creators of every dataset listed above

 Every evaluation set below was held out of training. he_bench v1 has 8 Hebrew tasks, each asked in 3 phrasings; the
 set is frozen by sha256. Brier and ECE are better when lower. "laya-multilingual" is Laya's published multilingual
+checkpoint. "Previous" is this project's previous checkpoint, trained without the reading and teacher data. **Bold**
+marks the best value in each row: the highest, or the lowest for Brier and ECE. Differences smaller than the run-to-run
+noise below are not meaningful.
+| Metric | laya-multilingual | Previous | Laya-Hebrew |
 |---|---|---|---|
+| MASSIVE he, 20 intents (500) | 0.352 | **0.816** | 0.806 |
 | MASSIVE he, 4 intents | 0.710 | 0.928 | **0.938** |
+| MASSIVE en, 20 intents | 0.652 | **0.816** | **0.816** |
+| he_bench accuracy / Brier / ECE (4,290) | 0.439 / 0.701 / 0.266 | 0.527 / 0.538 / **0.119** | **0.595** / **0.499** / 0.125 |
 | Belebele-he reading comprehension (900; chance 0.25) | | 0.468 | **0.767** |
+| SIB-200-he topic | | **0.808** | 0.801 |
 | Yes/no as a question: P(yes \| true) − P(yes \| false) | | 0.047 | **0.379** |
+| Yes/no as a claim: same gap (390) | | **0.692** | 0.674 |
 | Hebrew BoolQ, held out (875) | | 0.611 | **0.838** |
 | Rule-direction probe (208) | | 0.798 | **0.832** |
+| Held-out soft-label set, Brier (900) | | **0.230** | 0.231 |
 he_bench accuracy by task:
 | sentiment | 0.633 | 0.33 |
 | winograd | 0.607 | 0.50 |
 | hellaswag | 0.447 | 0.25 |
+| tone arousal | 0.389 | 0.21 |
+| tone valence | 0.328 | 0.23 |
+Chance is the accuracy of a uniformly random pick. The tone tasks are 5-level scales, and their chance is slightly above
+0.20 because some items tie between two levels.
 **Run-to-run noise.** A second training seed, with the same data and settings, differed by these amounts:
 - he_bench overall: 0.2 points
 - Dicta, for NeoDictaBERT-bilingual and DictaLM 3.0
 - Laya's authors, for the architecture, the RLCD training method and the runtime
+- The creators of every dataset listed above