Focus Lex evaluation on fine-tuned typed-decision models

BENCHMARK_COMPARISON.md

@@ -1,56 +0,0 @@
-# Decision family — same-benchmark comparison
-
-Fixed public Sol and Nox weights were evaluated once on the original Kai and Lex benchmark samples. Kai, Lex and Laya results reuse their saved predictions; their original aggregations were reproduced before adding the decoder columns. All columns use their released temperature. No model or temperature was fitted on these evaluations.
-
-## General decision capability
-
-| Text language · decision | Kai | Laya English | Laya Multilingual | Sol | Nox |
-|---|---:|---:|---:|---:|---:|
-| English · Choice | 41.01 | 40.80 | 30.97 | 65.96 | **70.82** |
-| English · Noul (BoolQ)¹ | 71.80 | 72.92 | 62.17 | 74.93 | **85.24** |
-| English · Score | **91.80** | 84.71 | 83.06 | 86.63 | 85.00 |
-| Non-English · Choice | 48.44 | 39.96 | 43.82 | 61.41 | **68.47** |
-| Non-English · Noul | 74.80 | 51.04 | 53.58 | 75.34 | **81.78** |
-| Non-English · Score | **87.47** | 82.08 | 83.41 | 84.29 | 80.10 |
-
-Choice is macro accuracy, Noul is balanced accuracy, and Score is 1 − normalized RPS. These are distinct metrics, not a single blended score. The original five-family common window contains 41,720 decisions: Belebele 31,354, XCOPA 5,500, OASST 3,803, Afri 747 and TyDi 316. The English Noul row separately uses 2,303 BoolQ questions. All 44,023 inputs fit both decoder models without truncation. English and non-English aggregation, class balancing and source weights are unchanged.
-
-The original common window is Kai/Laya Multilingual ≤1,024 complete tokens and Laya English ≤512. Sol/Nox use their released 16,384-token capacity on these same complete inputs; larger capacity does not change the sample denominator. This table compares general published weights. These panels were already observed during development and are not presented as a new independent holdout.
-
-[Original Kai evaluation](https://gist.github.com/Xunzhuo/bfc87193ac13633019fd73cfaccc615f#file-01-kai-evaluation-md).
-
-## Typed decisions
-
-| Test accuracy | Lex | Laya Typed Decisions | Sol | Nox |
-|---|---:|---:|---:|---:|
-| **Overall** | **78.15%** | 76.60% | 47.55% | 51.50% |
-| Choice · 600 decisions | **74.00%** | 73.33% | 44.50% | 48.33% |
-| Noul · 600 decisions | 84.67% | **85.67%** | 58.50% | 60.50% |
-| Score · 800 decisions | **76.38%** | 72.25% | 41.63% | 47.13% |
-
-The original TEST contains 2,000 decisions across 400 state components and four English workflows. **Lex and Laya Typed Decisions are fine-tuned specialists; Sol and Nox are general published models.** This describes their fixed releases on the same tasks, rather than equal-budget fine-tuning. The TEST remains excluded from subsequent Lex recipe, checkpoint and temperature selection.
-
-| Model | Soft NLL ↓ | Mean Brier ↓ | Score RPS ↓ | Score MAE ↓ |
-|---|---:|---:|---:|---:|
-| Lex | 0.961902 | 0.030858 | 0.022994 | 0.251408 |
-| Laya Typed Decisions | 0.884390 | 0.018452 | 0.015602 | 0.242408 |
-| Sol | 1.357052 | 0.094882 | 0.081148 | 0.601998 |
-| Nox | 2.375962 | 0.136159 | 0.117172 | 0.676995 |
-
-Soft metrics measure agreement with the original normalized soft labels. Score MAE uses the original numerical rubric values. These do not establish calibration for an arbitrary application.
-
-| Paired accuracy difference | Points | 95% interval |
-|---|---:|---:|
-| Nox minus Lex | -26.65 | [-29.35, -24.05] |
-| Nox minus Laya Typed Decisions | -25.10 | [-27.80, -22.45] |
-| Sol minus Lex | -30.60 | [-33.20, -28.05] |
-| Sol minus Laya Typed Decisions | -29.05 | [-31.70, -26.45] |
-
-Intervals resample whole state components using the original 2,000-replicate stratified bootstrap. No new confidence intervals are claimed for the six general panels. [Original Lex evaluation](https://gist.github.com/Xunzhuo/76b59cb158ce5069a53b946c6f0ee656#file-01-lex-evaluation-md).
-
-## Fixed releases and execution
-
-- [Nox · c9bad4f4](https://huggingface.co/llm-semantic-router/Decision-1.0-Nox/tree/c9bad4f4b8bafa56522f1ce1a90dcbe1e44df0a0)
-- [Sol · d29e9a71](https://huggingface.co/llm-semantic-router/Decision-1.0-Sol/tree/d29e9a71e3e7feeac667140a7180e9cfdcabc0b2)
-
-Each decoder completed 46,023 decisions in 5,753 AMD B8 forwards, with zero unsupported inputs and no parameter updates. Sol/Nox retain their public BF16 backbone and FP32 head; original encoder predictions use their existing FP32 runtime. This is an accuracy comparison, not a serving-latency measurement. Raw and shipped-temperature scores are retained in [structured results](BENCHMARK_COMPARISON.json).

EVALUATION.md

@@ -1,7 +1,5 @@
 # Official typed-decisions TEST
 
-[Expanded comparison with fixed published Sol and Nox](BENCHMARK_COMPARISON.md). Original encoder evaluation follows unchanged.
-
 The fixed final Lex checkpoint and the official `laya-typed-decisions` specialist were each evaluated once on the original 2,000 TEST decisions, from 400 state components. Both models support all 2,000 complete inputs within 1,024 tokens, their native windows and their untruncated official-default support. No TRAIN state component or full input overlaps this TEST support. The test contains four English workflows and 20 question schemas. This is not evidence of multilingual or unseen-schema generalization.
 
 | Metric | Lex | Official specialist (shipped temperature) |

PACKAGE_MANIFEST.json

@@ -13,8 +13,8 @@
       "sha256": "54a5895871af33845ee939f2e8f37b7f821cee8e89ff25ec1fd78e9cf83e9c63"
     },
     "EVALUATION.md": {
-      "bytes": 2942,
-      "sha256": "11e98e56fca020fb513d1b5d062536c91ff0eb7b9c53503655085ffcf006f925"
+      "bytes": 2813,
+      "sha256": "00862d0a49c69c406b7dbfa890f95f09a34680568309d1cc0571733abe7b933c"
     },
     "FINETUNING.md": {
       "bytes": 3629,
@@ -61,8 +61,8 @@
       "sha256": "2e5936d3ea196e00968cc37b7662a6afbe8cfd6fd11d04b9fecba554b1e1e37b"
     },
     "README.md": {
-      "bytes": 4049,
-      "sha256": "8db73986b14f1fb52172a1e1d2fe0dceb07bd6438466014e256a5f8847322fd0"
+      "bytes": 3816,
+      "sha256": "43406a86a3c38ccda748d6c4f96e7be18af9dca8aaf3229c283154b102320b70"
     },
     "TECHNICAL_VALIDATION.json": {
       "bytes": 1547,
@@ -363,14 +363,6 @@
     "LICENSES/Upstream-MIT.txt": {
       "bytes": 1191,
       "sha256": "3ec44d2f046b27986e28e3b4705110a33b139e6392ef3868ff594a0453311038"
-    },
-    "BENCHMARK_COMPARISON.md": {
-      "bytes": 4485,
-      "sha256": "ba0c0eb7f7c65b584ceee6ba2d08068a6e188e904c32856df136202dce98763b"
-    },
-    "BENCHMARK_COMPARISON.json": {
-      "bytes": 609635,
-      "sha256": "45990555c872f93ef866f59653019b6876d7ca44b9d430f44d0c084d46a5ceb1"
     }
   },
   "language_scope": "English typed-decisions specialist",
@@ -385,12 +377,11 @@
   "subject_manifest_sha256": "f288d873999832a3f37c6a7c4268c2ab309691e621794dbf7acab891acbbb7e6",
   "technical_validation": "TECHNICAL_VALIDATION.json",
   "training_provenance": "TRAINING_PROVENANCE.json",
-  "file_count_excluding_this_manifest": 93,
-  "bytes_excluding_this_manifest": 2327955224,
+  "file_count_excluding_this_manifest": 91,
+  "bytes_excluding_this_manifest": 2327340742,
   "publication": {
     "download_access": "public_ungated",
     "new_contribution_license": "Apache-2.0",
     "retained_third_party_terms": true
-  },
-  "public_non_native_bytes_excluding_this_manifest": 5696013
+  }
 }

README.md

@@ -29,16 +29,16 @@ Apache 2.0 · No access request · [Try Decision Studio](https://llm-semantic-ro
 
 **78.15% accuracy — 1.55 points above Laya Typed Decisions.**
 
-Lex answers **31 more decisions correctly** than the official fine-tuned Laya checkpoint on the original 2,000-decision TEST. All columns use the same complete inputs.
+Both models were evaluated on the original 2,000-decision typed-decisions test split, with complete inputs and the official fine-tuned Laya checkpoint. Lex answers **31 more decisions correctly**.
 
-| Test accuracy | Lex | Laya Typed Decisions | Sol | Nox |
-|---|---:|---:|---:|---:|
-| **Overall** | **78.15%** | 76.60% | 47.55% | 51.50% |
-| Choice · 600 decisions | **74.00%** | 73.33% | 44.50% | 48.33% |
-| Noul · 600 decisions | 84.67% | **85.67%** | 58.50% | 60.50% |
-| Score · 800 decisions | **76.38%** | 72.25% | 41.63% | 47.13% |
+| Test accuracy | Lex | Laya Typed Decisions |
+|---|---:|---:|
+| **Overall** | **78.15%** | 76.60% |
+| Choice · 600 decisions | **74.00%** | 73.33% |
+| Noul · 600 decisions | 84.67% | **85.67%** |
+| Score · 800 decisions | **76.38%** | 72.25% |
 
-Lex and Laya Typed Decisions are fine-tuned specialists; [Sol](https://huggingface.co/llm-semantic-router/Decision-1.0-Sol) (1.88B) and [Nox](https://huggingface.co/llm-semantic-router/Decision-1.0-Nox) (4.21B) are general-model references. The Lex–Laya gain has a paired 95% interval of −0.20 to +3.15 points; Laya leads Lex on Noul and probability quality. [Full comparison](BENCHMARK_COMPARISON.md).
+Observed accuracy gain; the paired 95% interval is −0.20 to +3.15 points. Laya leads on Noul and probability-quality metrics. [Full evaluation](https://gist.github.com/Xunzhuo/76b59cb158ce5069a53b946c6f0ee656#file-01-lex-evaluation-md).
 
 ## Built for operational decisions