Upload 45 files

LICENSE

@@ -0,0 +1,176 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS

THIRD_PARTY.md

@@ -0,0 +1,13 @@
+# Third-party sources
+
+Ariadne's prompts and preparation recipes were adapted from:
+
+- NandhaKishorM/laya, commit `4066d5d5fbf08b66c6757ddeedbd797bd7655bc0`, Apache-2.0. Source: https://github.com/NandhaKishorM/laya
+- AbdelStark/jev-benchmarks, commit `0d610cc53e79bcbec691312b0c4adb4a0e371642`, Apache-2.0. The BTZSC grouping and balanced sample recipe follows this project. Source: https://github.com/AbdelStark/jev-benchmarks
+
+Copyright and license notices are retained in `licenses/`. The code is adapted for this repository's canonical case format, adapters and scorer. No upstream claims of affiliation or endorsement apply.
+
+Datasets and model weights keep their own licenses. Their source URLs, immutable revisions and model-card license metadata appear in `benchmarks/sources.lock.json` and model configs. Data is downloaded on request into ignored local directories; the harness does not relicense it. See each source card for terms, provenance and training exposure.
+
+TypeSafe's public documentation defines the native System One request/response contract. The official evaluation website is cited for research; its examples are not redistributed.
+

USAGE.md

@@ -0,0 +1,48 @@
+# Load the Laya linear adapter
+
+This package contains the validation-selected **seed 1, epoch 14** checkpoint from the deterministic frozen-base experiment. It trained 1,049,600 affine parameters at LR 3e-4 while all base Laya weights remained frozen. Its 4.2 MB weight file requires the pinned base model, which the loader downloads separately.
+
+The ongoing lower-learning-rate experiment is separate. These are the already verified weights, not a claim that the training stability investigation is complete.
+
+## Upload
+
+Extract the ZIP and upload the files and folders **at the root of your existing Hugging Face model repository**. There is deliberately no README.md in this bundle, so it can coexist with your existing model card. The ZIP is a transport archive; upload its extracted contents for normal use.
+
+This is a custom input adapter. Loading requires the bundled loader; bare `laya.load(repo_id)` and `AutoModel.from_pretrained(repo_id)` do not install the extra affine layer.
+
+## Load a downloaded copy
+
+Install an appropriate PyTorch build and the versions in `requirements.txt`. The measured GPU environment is recorded in `environment.json`. From the downloaded repository directory:
+
+```python
+from load_adapter import load
+
+model = load(device="cuda:0")
+answer = model.predict("The delivery arrived damaged.", {
+    "route": {
+        "type": "choice",
+        "instructions": "Choose the customer-support queue.",
+        "criteria": {"delivery": "Delivery and damaged items", "billing": "Payments and invoices"},
+    }
+})
+print(answer)
+model.close()
+```
+
+After you upload, first download your repository with `huggingface_hub.snapshot_download("YOUR_ACCOUNT/YOUR_REPOSITORY")`, then run the example from that downloaded directory. Use `device="cpu"` for CPU inference. `base_path` can point to the pinned base snapshot already on disk; `local_files_only=True` uses cached base files.
+
+The loader verifies the adapter checksum, base configuration/tokenizer hashes and pretrained tensor hash. Strict determinism is enabled by default and requires the CUBLAS workspace environment to be set before CUDA is initialized. A fresh process handles this automatically. Numerical agreement across other hardware or software is not guaranteed.
+
+## Measured performance and limits
+
+Selected checkpoint accuracy: typed decisions **76.95%**, AG News **93.17%**, BoolQ **79.83%**, Emotion **57.33%**, prompt injections **71.55%**, SST-5 **42.00%**, MASSIVE EN **74.33%**, XNLI EN **87.67%**. Native base typed accuracy was 36.25% under the matched protocol.
+
+Across all five training seeds, typed accuracy was **70.23% ± 6.68 pp** (sample SD). Three seeds had severe retention losses. The selected model also lost 3.17 pp on BoolQ and 4.33 pp on MASSIVE versus base. Full results and paired uncertainty are in `metrics.json` and `evidence/candidate-uncertainty.json`. Other seeds' scores are included for transparency; this bundle contains only seed 1's weights.
+
+The benchmark uses 2,000 typed decisions plus fixed subsets of seven other tasks, totaling 5,116 decisions. MASSIVE uses 20 candidate intents. The same test subsets were inspected in earlier experiments. These results describe an experimental task adapter, not established broad task improvement.
+
+The base checkpoint's choice:11+ temperature is clamped from 0.1005828 to 0.5 by Laya 0.3.20. This expected load warning concerns confidence calibration; raw-logit training and class ordering are unaffected. Calibration was not refitted after adaptation.
+
+All 5,116 answer objects matched the original selected checkpoint when the portable export was tested from outside the workspace. `export-verification.json` records that check. `upload-manifest.json` verifies that this bundle keeps the same weights and inference code, with the other seeds removed from its loading configuration.
+
+License: Apache-2.0. Base model: `convaiinnovations/laya`, pinned to revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Attribution is in `THIRD_PARTY.md` and `licenses/`.

adapter.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a
+size 4198616

adapter_config.json

@@ -0,0 +1,25 @@
+{
+  "format_version": 1,
+  "interface": "exact_linear",
+  "position": "after native embeddings, before ModernBERT block 0",
+  "base_model": "convaiinnovations/laya",
+  "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+  "candidate_seed": 1,
+  "max_len": 1024,
+  "head_max_len": 256,
+  "trainable_parameters_during_training": 1049600,
+  "frozen_state_sha256": "7560d83b1e1c17cce7eb67b25674fc8e7f0958bac4b47f559c725f8695c4d59b",
+  "base_config_sha256": {
+    "rl_agent_config.json": "ae287b56bbcf5f8c4f4541ae9dfd00c914c4c48b940b8398c3058af37ba92bbd",
+    "encoder/config.json": "bf3ab80598fdccf414855a2ce80f22859e4492d06ca8a62ddd1cfb63972f8979",
+    "tokenizer/tokenizer.json": "6c8aaa9a542084f2457eab775d4eeb51f92a70c0fd9de28d5edb0ddec3c08d30",
+    "tokenizer/tokenizer_config.json": "50044de60daaa73df97d262e15a40d4faf0160e7d742df64b377877a1320dd12"
+  },
+  "seeds": {
+    "1": {
+      "weights_file": "adapter.safetensors",
+      "weights_sha256": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
+      "selected_epoch": 14
+    }
+  }
+}

ariadne_bench/__init__.py

@@ -0,0 +1,3 @@
+"""Typed-decision benchmark harness."""
+
+__version__ = "0.1.0"

ariadne_bench/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

ariadne_bench/adapters.py

@@ -0,0 +1,154 @@
+"""Adapters receive only state and questions; labels never enter model input."""
+
+import importlib
+import os
+import urllib.error
+import urllib.request
+
+from .schema import dumps, labels
+
+
+class Uniform:
+    def predict(self, state, questions):
+        answers = {}
+        for qid, q in questions.items():
+            keys = labels(q)
+            a = {"type": q["type"], "probabilities": {k: 1 / len(keys) for k in keys}}
+            if q["type"] == "noul":
+                a["noul"] = 0.5
+            elif q["type"] == "choice":
+                a["choice"] = keys[0]
+            else:
+                a["score"] = (len(keys) - 1) / 2
+            answers[qid] = a
+        return {"model": "uniform", "answers": answers}
+
+
+class NativeHTTP:
+    def __init__(self, config):
+        self.endpoint = config["endpoint"]
+        self.model = config["model"]
+        self.timeout = config.get("timeout_seconds", 60)
+        self.key = os.environ.get(config.get("api_key_env", ""))
+        if config.get("api_key_env") and not self.key:
+            raise ValueError(f"set {config['api_key_env']} before running this adapter")
+        if not self.endpoint.startswith(("http://", "https://")):
+            raise ValueError("endpoint must be an HTTP(S) URL")
+
+        # Refuse redirects so a bearer credential cannot move to a different host.
+        class NoRedirect(urllib.request.HTTPRedirectHandler):
+            def redirect_request(self, req, fp, code, msg, headers, newurl):
+                return None
+
+        self.opener = urllib.request.build_opener(NoRedirect())
+
+    def predict(self, state, questions):
+        import json
+
+        headers = {"Content-Type": "application/json"}
+        if self.key:
+            headers["Authorization"] = "Bearer " + self.key
+        payload = {"model": self.model, "state": state, "questions": questions}
+        request = urllib.request.Request(
+            self.endpoint, data=dumps(payload).encode(), headers=headers
+        )
+        try:
+            with self.opener.open(request, timeout=self.timeout) as response:
+                return json.load(response)
+        except urllib.error.HTTPError as exc:
+            # Response bodies may contain credentials or echoed inputs. Store status only.
+            raise RuntimeError(f"HTTP {exc.code}") from None
+        except urllib.error.URLError:
+            raise RuntimeError("HTTP connection failed") from None
+
+
+class Laya:
+    def __init__(self, config):
+        os.environ.setdefault("USE_TF", "0")
+        try:
+            import laya
+        except ImportError as exc:
+            raise RuntimeError("install the laya extra to use the local adapter") from exc
+        from pathlib import Path
+
+        model_path = Path(config["model"])
+        revision = None
+        if not model_path.is_dir():
+            from huggingface_hub import HfApi, snapshot_download
+
+            revision = config.get("revision")
+            if not (
+                isinstance(revision, str)
+                and len(revision) == 40
+                and all(c in "0123456789abcdef" for c in revision)
+            ):
+                revision = HfApi().model_info(config["model"], revision=revision).sha
+            prefix = config.get("subfolder", "").strip("/")
+            prefix = prefix + "/" if prefix else ""
+            model_path = Path(
+                snapshot_download(
+                    config["model"],
+                    revision=revision,
+                    allow_patterns=[
+                        prefix + name
+                        for name in (
+                            "rl_agent_config.json",
+                            "model.safetensors",
+                            "tokenizer/*",
+                            "encoder/*",
+                        )
+                    ],
+                )
+            )
+        if config.get("subfolder"):
+            model_path = model_path / config["subfolder"]
+        self.agent = laya.load(str(model_path), device=config.get("device"))
+        self.max_len = config.get("max_len")
+        if config.get("freeze_parameters", False):
+            self.agent.model.requires_grad_(False)
+        import torch
+
+        self.metadata = {
+            "resolved_revision": revision,
+            "checkpoint_path": str(model_path.resolve()),
+            "device": str(self.agent.device),
+            "laya_version": laya.__version__,
+            "model_config": self.agent.cfg,
+            "dtype": str(self.agent.dtype),
+            "torch_version": torch.__version__,
+            "cpu_threads": torch.get_num_threads(),
+            "gpu_name": torch.cuda.get_device_name(self.agent.device)
+            if self.agent.device.type == "cuda"
+            else None,
+            "frozen_parameters": bool(config.get("freeze_parameters", False)),
+            "served_temperatures": [float(t) for t in self.agent.temperature],
+            "served_temperature_by_options": dict(self.agent.temperature_by_options),
+        }
+
+    def synchronize(self):
+        import torch
+
+        if self.agent.device.type == "cuda":
+            torch.cuda.synchronize(self.agent.device)
+        elif self.agent.device.type == "mps":
+            torch.mps.synchronize()
+        elif self.agent.device.type == "xpu":
+            torch.xpu.synchronize(self.agent.device)
+
+    def predict(self, state, questions):
+        kw = {} if self.max_len is None else {"max_len": self.max_len}
+        return self.agent.predict(state, questions, **kw)
+
+
+def create_adapter(config):
+    kind = config["adapter"]
+    if kind == "uniform":
+        return Uniform()
+    if kind == "http":
+        return NativeHTTP(config)
+    if kind == "laya":
+        return Laya(config)
+    if kind == "python":
+        module, name = config["factory"].split(":", 1)
+        return getattr(importlib.import_module(module), name)(config.get("options", {}))
+    raise ValueError(f"unknown adapter: {kind}")

ariadne_bench/cli.py

@@ -0,0 +1,132 @@
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from .datasets import APPS, CORE, prepare
+from .runner import compare, run, score
+from .schema import read_jsonl, save_bundle
+
+
+def positive(value):
+    number = int(value)
+    if number <= 0:
+        raise argparse.ArgumentTypeError("must be positive")
+    return number
+
+
+def nonnegative(value):
+    number = int(value)
+    if number < 0:
+        raise argparse.ArgumentTypeError("must be nonnegative")
+    return number
+
+
+def tolerance_value(value):
+    number = float(value)
+    if not 0 <= number <= 0.1:
+        raise argparse.ArgumentTypeError("tolerance must be in [0, 0.1]")
+    return number
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        description="Freeze data, evaluate typed decisions, compare models."
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+    sub.add_parser("list", help="list available suites")
+    p = sub.add_parser("prepare", help="download pinned sources and freeze prompts")
+    p.add_argument(
+        "--profile",
+        choices=["smoke", "latency", "laya-core", "laya-apps", "laya-multilingual", "jev-btzsc"],
+        default="laya-core",
+    )
+    p.add_argument(
+        "--suites", help="comma-separated suites; multilingual names include language, e.g. xnli.en"
+    )
+    p.add_argument("--out", required=True)
+    p.add_argument("--lock", default="benchmarks/sources.lock.json")
+    p.add_argument("--cache", default=".cache/datasets")
+    p.add_argument("--limit", type=positive, help="cases per suite; changes the protocol")
+    p.add_argument("--seed", type=int, default=13)
+    p.add_argument("--permutations", type=nonnegative, default=0)
+    p = sub.add_parser("import", help="validate and freeze your own canonical JSONL")
+    p.add_argument("--cases", required=True)
+    p.add_argument("--out", required=True)
+    p = sub.add_parser("run", help="run a model; nonzero exit if any decisions fail")
+    p.add_argument("--data", required=True)
+    p.add_argument("--config", required=True)
+    p.add_argument("--out", required=True)
+    p.add_argument("--warmup", type=nonnegative, default=0)
+    p.add_argument("--resume", action="store_true")
+    p.add_argument("--ece-bins", type=positive, default=15)
+    p.add_argument("--probability-tolerance", type=tolerance_value, default=0.02)
+    p = sub.add_parser("score", help="score saved predictions without model calls")
+    p.add_argument("--data", required=True)
+    p.add_argument("--predictions", required=True)
+    p.add_argument("--out", required=True)
+    p.add_argument("--ece-bins", type=positive, default=15)
+    p.add_argument("--probability-tolerance", type=tolerance_value, default=0.02)
+    p = sub.add_parser("compare", help="compare reports from identical frozen cases")
+    p.add_argument("reports", nargs="+")
+    p.add_argument("--out")
+    args = parser.parse_args(argv)
+    try:
+        if args.command == "list":
+            print(
+                "\n".join(
+                    dict.fromkeys(
+                        CORE
+                        + APPS
+                        + [
+                            "massive-intent.<language>",
+                            "massive-scenario.<language>",
+                            "xnli.<language>",
+                            "btzsc.agnews",
+                            "btzsc.emotiondair",
+                            "btzsc.banking77",
+                        ]
+                    )
+                )
+            )
+        elif args.command == "prepare":
+            result = prepare(
+                args.profile,
+                args.suites,
+                args.out,
+                args.lock,
+                args.cache,
+                args.limit,
+                args.seed,
+                args.permutations,
+            )
+            print(json.dumps({k: result[k] for k in ("cases", "decisions", "sha256")}, indent=2))
+        elif args.command == "import":
+            result = save_bundle(args.out, read_jsonl(args.cases), {"profile": "custom"})
+            print(f"Frozen {result['cases']} cases")
+        elif args.command == "run":
+            result = run(
+                args.data,
+                args.config,
+                args.out,
+                args.ece_bins,
+                args.probability_tolerance,
+                args.warmup,
+                args.resume,
+            )
+            print(f"Report: {Path(args.out) / 'report.md'}")
+            return 2 if result["overall"]["failed"] else 0
+        elif args.command == "score":
+            result = score(
+                args.data, args.predictions, args.out, args.ece_bins, args.probability_tolerance
+            )
+            return 2 if result["overall"]["failed"] else 0
+        else:
+            table = compare(args.reports)
+            if args.out:
+                Path(args.out).write_text(table + "\n")
+            print(table)
+    except (ValueError, KeyError, OSError, ImportError) as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 1
+    return 0

ariadne_bench/datasets.py

@@ -0,0 +1,532 @@
+"""Dataset preparation is separate from inference and freezes all prompt bytes."""
+
+import copy
+import json
+import random
+from collections import defaultdict
+from pathlib import Path
+
+from .schema import labels, save_bundle
+
+LAYA_REVISION = "4066d5d5fbf08b66c6757ddeedbd797bd7655bc0"
+BTZSC_REVISION = "fef2a2ac62b69c58670047dddf045c53d7c3cb5e"
+CORE = ["typed-decisions", "ag-news", "emotion", "banking77", "boolq", "sst5", "prompt-injections"]
+APPS = [
+    "ag-news",
+    "emotion",
+    "banking77",
+    "enron-spam",
+    "phishing",
+    "toxic-chat",
+    "jailbreak",
+    "support-triage",
+    "rag-relevance",
+    "model-routing",
+]
+MASSIVE_LANGS = "en,de,fr,es,pt,ru,tr,ar,hi,ta,zh-CN,ja,ko,sw".split(",")
+XNLI_LANGS = "en,de,fr,es,ru,tr,ar,hi,ur,vi,th,el,bg,zh,sw".split(",")
+SPECS = {
+    "typed-decisions": ("LocalLLaMA/typed-decisions", "all", "test", None),
+    "ag-news": ("fancyzhx/ag_news", None, "test", 600),
+    "emotion": ("dair-ai/emotion", "split", "test", 600),
+    "banking77": ("mteb/banking77", None, "test", 500),
+    "boolq": ("google/boolq", None, "validation", 600),
+    "sst5": ("SetFit/sst5", None, "test", 600),
+    "prompt-injections": ("deepset/prompt-injections", None, "test", None),
+    "enron-spam": ("SetFit/enron_spam", None, "test", 400),
+    "phishing": ("zefang-liu/phishing-email-dataset", None, "train", 400),
+    "toxic-chat": ("lmsys/toxic-chat", "toxicchat0124", "test", 400),
+    "jailbreak": ("lmsys/toxic-chat", "toxicchat0124", "test", 400),
+    "support-triage": ("Tobi-Bueck/customer-support-tickets", None, "train", 400),
+    "rag-relevance": ("microsoft/ms_marco", "v1.1", "validation", 400),
+    "model-routing": ("openai/gsm8k", "main", "test", 400),
+}
+AG_CRITERIA = {
+    "world": "world news and international politics",
+    "sports": "sports",
+    "business": "business and economy",
+    "sci_tech": "science and technology",
+}
+NLI_CRITERIA = {
+    "entailment": "the premise implies the hypothesis is true",
+    "neutral": "the premise neither implies nor contradicts the hypothesis",
+    "contradiction": "the premise implies the hypothesis is false",
+}
+QUEUES = {
+    "Technical Support": "technical problems, bugs, outages, integrations",
+    "Product Support": "help using a product or feature",
+    "Customer Service": "general account or service questions",
+    "IT Support": "internal IT, devices, access, networks",
+    "Billing and Payments": "invoices, charges, refunds, payment methods",
+    "Returns and Exchanges": "returning or exchanging an item",
+    "Service Outages and Maintenance": "downtime, outages, scheduled maintenance",
+    "Sales and Pre-Sales": "pricing, quotes, buying",
+    "Human Resources": "employment, payroll, leave, hiring",
+    "General Inquiry": "anything else",
+}
+
+
+def parsed(value):
+    if isinstance(value, str):
+        try:
+            return json.loads(value)
+        except json.JSONDecodeError:
+            pass
+    return value
+
+
+def single(suite, index, state, qid, question, target, **metadata):
+    keys = labels(question)
+    gold = {"label": keys[int(target)]}
+    if question["type"] == "score":
+        gold["score"] = float(target)
+    return {
+        "id": f"{suite}:{index}",
+        "suite": suite,
+        "state": state,
+        "questions": {qid: question},
+        "gold": {qid: gold},
+        **metadata,
+    }
+
+
+def question(kind, instructions, criteria=None):
+    q = {"type": kind, "instructions": instructions}
+    if criteria is not None:
+        q["criteria"] = criteria
+    return q
+
+
+class Sources:
+    def __init__(self, lock, cache):
+        self.lock = lock
+        self.cache = cache
+        self.used = {}
+
+    def load(self, repo, config=None, split="test"):
+        from datasets import load_dataset
+
+        revision = self.lock[repo]["revision"]
+        if len(revision) != 40 or any(c not in "0123456789abcdef" for c in revision):
+            raise ValueError(f"source lock must pin an immutable commit: {repo}")
+        key = f"{repo}:{config or 'default'}:{split}"
+        self.used[key] = {"repository": repo, "config": config, "split": split, **self.lock[repo]}
+        return load_dataset(repo, name=config, split=split, revision=revision, cache_dir=self.cache)
+
+
+def typed(rows):
+    result = []
+    for r in rows:
+        qs, gold = parsed(r["questions"]), parsed(r["gold"])
+        gs = {}
+        for qid, q in qs.items():
+            g = gold[qid]
+            label = str(g["label"]).lower() if q["type"] == "noul" else str(g["label"])
+            gs[qid] = {"label": label, "probabilities": g["probabilities"]}
+            if q["type"] == "score":
+                gs[qid]["score"] = float(g["score"])
+        result.append(
+            {
+                "id": "typed-decisions:" + r["id"],
+                "suite": "typed-decisions",
+                "workflow": r["workflow"],
+                "state": parsed(r["state"]),
+                "questions": qs,
+                "gold": gs,
+            }
+        )
+    return result
+
+
+def build_core(name, rows):
+    if name == "typed-decisions":
+        return typed(rows)
+    bank_labels = sorted(set(rows["label_text"])) if name == "banking77" else None
+    output = []
+    for i, r in enumerate(rows):
+        if name == "ag-news":
+            state, qid, q, target = (
+                {"article": r["text"]},
+                "topic",
+                question("choice", "What is the topic of `article`?", AG_CRITERIA),
+                r["label"],
+            )
+        elif name == "emotion":
+            state, qid, q, target = (
+                {"text": r["text"]},
+                "emotion",
+                question(
+                    "choice",
+                    "Which emotion is most strongly expressed in `text`?",
+                    dict.fromkeys(["sadness", "joy", "love", "anger", "fear", "surprise"]),
+                ),
+                r["label"],
+            )
+        elif name == "banking77":
+            state, qid, q, target = (
+                {"message": r["text"]},
+                "intent",
+                question(
+                    "choice",
+                    "Which banking intent does `message` express?",
+                    {s.replace("_", " "): None for s in bank_labels},
+                ),
+                bank_labels.index(r["label_text"]),
+            )
+        elif name == "boolq":
+            state, qid, q, target = (
+                {"passage": r["passage"], "question": r["question"]},
+                "answer",
+                question("noul", "Based on `passage`, is the answer to `question` yes?"),
+                int(r["answer"]),
+            )
+        elif name == "sst5":
+            state, qid, q, target = (
+                {"text": r["text"]},
+                "sentiment",
+                question(
+                    "score",
+                    "How positive is the sentiment of `text`?",
+                    ["very negative", "negative", "neutral", "positive", "very positive"],
+                ),
+                r["label"],
+            )
+        else:
+            state, qid, q, target = (
+                {"text": r["text"]},
+                "injection",
+                question(
+                    "noul",
+                    "Does `text` try to inject or override instructions given to an AI system?",
+                ),
+                r["label"],
+            )
+        output.append(single(name, i, state, qid, q, target))
+    return output
+
+
+def build_multilingual(name, rows, n, seed=13):
+    family, language = name.split(".", 1)
+    rng = random.Random(seed)
+    pool = sorted(set(rows["label_text"])) if family.startswith("massive") else []
+    output = []
+    for i, r in enumerate(rows):
+        if i >= n:
+            break
+        if family == "xnli":
+            output.append(
+                single(
+                    name,
+                    i,
+                    {"premise": r["premise"], "hypothesis": r["hypothesis"]},
+                    "relation",
+                    question(
+                        "choice",
+                        "What is the relationship between `premise` and `hypothesis`?",
+                        NLI_CRITERIA,
+                    ),
+                    r["label"],
+                    language=language,
+                )
+            )
+        else:
+            gold = r["label_text"]
+            distractors = [x for x in pool if x != gold]
+            options = [gold] + rng.sample(distractors, min(19, len(distractors)))
+            rng.shuffle(options)
+            instructions = (
+                "What is the user asking for in `utterance`?"
+                if family == "massive-intent"
+                else "Which domain does `utterance` belong to?"
+            )
+            q = question(
+                "choice", instructions, {k: k.replace("_", " ").replace(".", ": ") for k in options}
+            )
+            output.append(
+                single(
+                    name,
+                    i,
+                    {"utterance": r["text"]},
+                    "label",
+                    q,
+                    options.index(gold),
+                    language=language,
+                )
+            )
+    return output
+
+
+def balanced_indices(targets, limit, seed):
+    groups = defaultdict(list)
+    for i, target in enumerate(targets):
+        groups[target].append(i)
+    rng = random.Random(seed)
+    for indices in groups.values():
+        rng.shuffle(indices)
+    chosen = []
+    while len(chosen) < min(limit, len(targets)):
+        for cls in sorted(groups):
+            if groups[cls] and len(chosen) < limit:
+                chosen.append(groups[cls].pop())
+    return sorted(chosen)
+
+
+def build_btzsc(name, rows, n=100):
+    task = name.split(".", 1)[1]
+    texts = list(rows["text"])
+    k = next(i for i in range(1, len(texts)) if texts[i] != texts[0])
+    options = [str(rows[i]["hypothesis"]) for i in range(k)]
+    if len(rows) % k:
+        raise ValueError("incomplete BTZSC candidate group")
+    valid, targets = [], []
+    for start in range(0, len(rows), k):
+        group = rows[start : start + k]
+        if len(set(group["text"])) != 1 or list(group["hypothesis"]) != options:
+            raise ValueError("BTZSC candidate order changed")
+        y = [int(v) for v in group["labels"]]
+        if sum(y) == 1:
+            valid.append(start)
+            targets.append(y.index(1))
+    offset = ["agnews", "emotiondair", "banking77"].index(task)
+    q = question(
+        "choice",
+        "Which single label best describes the input text?",
+        {f"label_{i:03d}": option for i, option in enumerate(options)},
+    )
+    return [
+        single(name, valid[pos] // k, {"text": texts[valid[pos]]}, "label", q, targets[pos])
+        for pos in balanced_indices(targets, n, 20260917 + offset)
+    ]
+
+
+def build_app(name, rows, n, seed, sources):
+    rng = random.Random(seed)
+    indexed = list(enumerate(rows))
+    if name == "phishing":
+        indexed = [
+            (i, r)
+            for i, r in indexed[:6000]
+            if (r.get("Email Text") or "").strip()
+            and r.get("Email Type") in ("Safe Email", "Phishing Email")
+        ]
+        rng.shuffle(indexed)
+    if name in ("toxic-chat", "jailbreak"):
+        field = "toxicity" if name == "toxic-chat" else "jailbreaking"
+        indexed = [(i, r) for i, r in indexed if (r.get("user_input") or "").strip()]
+        positive = [(i, r) for i, r in indexed if int(r[field]) == 1][: n // 2]
+        indexed = positive + [(i, r) for i, r in indexed if int(r[field]) == 0][: n - len(positive)]
+        rng.shuffle(indexed)
+    if name == "model-routing":
+        domains = {
+            "code": "software engineering, programming, refactoring, architecture, debugging",
+            "math_or_logic": "mathematics, logic puzzles, proofs, complex calculation",
+            "writing": "creative writing, essays, emails, blog posts, copywriting",
+            "factual_lookup": "facts, definitions, trivia, history",
+            "data_analysis": "statistics, SQL, data manipulation, metrics",
+            "chitchat": "casual conversation, greetings, small talk",
+        }
+        code = sources.load("google-research-datasets/mbpp", "full")
+        news = sources.load("fancyzhx/ag_news")
+        pool = [(r["question"], "math_or_logic") for r in list(rows)[: n // 3]]
+        pool += [(r["text"], "code") for r in list(code)[: n // 3]]
+        pool += [(r["text"][:400], "factual_lookup") for r in list(news)[: n // 3]]
+        rng.shuffle(pool)
+        q = question("choice", "What domain does `request` belong to?", domains)
+        return [
+            single(name, i, {"request": text}, "domain", q, list(domains).index(label))
+            for i, (text, label) in enumerate(pool)
+        ]
+    output = []
+    for i, r in indexed:
+        if len(output) >= n:
+            break
+        if name == "enron-spam":
+            state = {"subject": r.get("subject") or "", "body": (r.get("message") or "")[:3000]}
+            qid, q, target = (
+                "is_spam",
+                question("noul", "Is this email unsolicited spam or bulk marketing?"),
+                r["label"],
+            )
+        elif name == "phishing":
+            state = {"email": r["Email Text"][:3000]}
+            qid, q, target = (
+                "is_phishing",
+                question(
+                    "noul",
+                    "Is this email a phishing or scam attempt to steal money, credentials, or personal data?",
+                    {
+                        "true": "phishing, scam, or fraud",
+                        "false": "a legitimate email (even if promotional)",
+                    },
+                ),
+                int(r["Email Type"] == "Phishing Email"),
+            )
+        elif name in ("toxic-chat", "jailbreak"):
+            toxic = name == "toxic-chat"
+            state = {"post" if toxic else "prompt": r["user_input"][:3000]}
+            qid = "toxic" if toxic else "jailbreak"
+            q = question(
+                "noul",
+                "Is `post` toxic: rude, disrespectful or likely to make someone leave the discussion?"
+                if toxic
+                else "Does `prompt` try to make an AI assistant ignore its rules, policies or system instructions?",
+            )
+            target = int(r["toxicity" if toxic else "jailbreaking"])
+        elif name == "support-triage":
+            if r.get("language") != "en" or r.get("queue") not in QUEUES or not r.get("body"):
+                continue
+            state = {"subject": r["subject"] or "", "body": r["body"].replace("\\n", "\n")[:3000]}
+            qid, q, target = (
+                "queue",
+                question("choice", "Which support queue should handle this ticket?", QUEUES),
+                list(QUEUES).index(r["queue"]),
+            )
+        else:
+            passages = r["passages"]
+            pos = [t for t, s in zip(passages["passage_text"], passages["is_selected"]) if s == 1]
+            neg = [t for t, s in zip(passages["passage_text"], passages["is_selected"]) if s == 0]
+            if not pos or not neg:
+                continue
+            target = int(len(output) % 2 == 0)
+            state = {"query": r["query"], "passage": rng.choice(pos if target else neg)}
+            qid, q = "relevant", question("noul", "Does `passage` help answer `query`?")
+        output.append(single(name, i, state, qid, q, target))
+    return output
+
+
+def smoke():
+    return [
+        {
+            "id": "smoke:0",
+            "suite": "smoke",
+            "state": "The package is late. Please refund me.",
+            "questions": {
+                "refund": question("noul", "Is a refund requested?"),
+                "department": question(
+                    "choice",
+                    "Which team should handle this?",
+                    {"billing": "refunds", "technical": "bugs"},
+                ),
+                "urgency": question(
+                    "score", "How urgent is the request?", ["low", "medium", "high"]
+                ),
+            },
+            "gold": {
+                "refund": {"label": "true"},
+                "department": {"label": "billing"},
+                "urgency": {"label": "1", "score": 1.0},
+            },
+        }
+    ]
+
+
+def latency():
+    result = []
+    for n in (1, 5, 10, 50):
+        for i in range(30):
+            base = smoke()[0]
+            result.append(
+                {
+                    "id": f"latency:{n}:{i}",
+                    "suite": f"latency-{n}",
+                    "state": base["state"],
+                    "questions": {f"q{j}": base["questions"]["department"] for j in range(n)},
+                    "gold": {f"q{j}": {"label": "billing"} for j in range(n)},
+                }
+            )
+    return result
+
+
+def permute(cases, count, seed):
+    result = list(cases)
+    rng = random.Random(seed)
+    for case in cases:
+        if not any(q["type"] == "choice" for q in case["questions"].values()):
+            continue
+        for variant in range(1, count + 1):
+            c = copy.deepcopy(case)
+            c.update(
+                id=f"{case['id']}:permutation:{variant}", base_id=case["id"], permutation=variant
+            )
+            for q in c["questions"].values():
+                if q["type"] == "choice":
+                    keys = list(q["criteria"])
+                    rng.shuffle(keys)
+                    q["criteria"] = {k: q["criteria"][k] for k in keys}
+            result.append(c)
+    return result
+
+
+def prepare(profile, suites, output, lock_path, cache, limit=None, seed=13, permutations=0):
+    sources = (
+        Sources(json.loads(Path(lock_path).read_text()), cache)
+        if profile not in ("smoke", "latency")
+        else None
+    )
+    if profile == "smoke":
+        cases = smoke()
+    elif profile == "latency":
+        cases = latency()
+    else:
+        if suites:
+            names = suites.split(",")
+        elif profile == "laya-core":
+            names = CORE
+        elif profile == "laya-apps":
+            names = APPS
+        elif profile == "jev-btzsc":
+            names = ["btzsc.agnews", "btzsc.emotiondair", "btzsc.banking77"]
+        elif profile == "laya-multilingual":
+            names = [
+                f"{family}.{lang}"
+                for family in ("massive-intent", "massive-scenario")
+                for lang in MASSIVE_LANGS
+            ]
+            names += [f"xnli.{lang}" for lang in XNLI_LANGS]
+        else:
+            raise ValueError(f"unknown profile: {profile}")
+        if len(set(names)) != len(names):
+            raise ValueError("duplicate suites")
+        cases = []
+        for name in names:
+            print(f"Preparing {name} ...", flush=True)
+            if name.startswith("btzsc."):
+                rows = sources.load("btzsc/btzsc", name.split(".", 1)[1])
+                built = build_btzsc(name, rows, limit or 100)
+            elif name.startswith(("massive-intent.", "massive-scenario.", "xnli.")):
+                family, lang = name.split(".", 1)
+                repo = {
+                    "massive-intent": "mteb/amazon_massive_intent",
+                    "massive-scenario": "mteb/amazon_massive_scenario",
+                    "xnli": "facebook/xnli",
+                }[family]
+                rows = sources.load(repo, lang)
+                built = build_multilingual(name, rows, limit or 300, seed)
+            else:
+                if name not in SPECS:
+                    raise ValueError(f"unknown suite: {name}")
+                repo, config, split, default_n = SPECS[name]
+                rows = sources.load(repo, config, split)
+                n = limit or (400 if profile == "laya-apps" else default_n) or len(rows)
+                if name in CORE:
+                    # Build label inventories on the full split before taking the prefix.
+                    built = build_core(name, rows)[:n]
+                else:
+                    built = build_app(name, rows, n, seed, sources)
+            if not built:
+                raise ValueError(f"suite produced no cases: {name}")
+            cases.extend(built)
+    cases = permute(cases, permutations, seed)
+    return save_bundle(
+        output,
+        cases,
+        {
+            "profile": profile,
+            "seed": seed,
+            "limit": limit,
+            "permutations": permutations,
+            "sources": sources.used if sources else {},
+            "prompt_source": f"https://github.com/NandhaKishorM/laya/tree/{LAYA_REVISION}/research",
+            "protocol_note": "See docs/BENCHMARKS.md for differences from historical runs; freeze and reuse this bundle.",
+        },
+    )

ariadne_bench/experiments/__init__.py

@@ -0,0 +1 @@
+"""Experimental training entry points, separate from benchmark evaluation."""

ariadne_bench/experiments/align.py

@@ -0,0 +1,336 @@
+"""Train the hybrid connection using a separate case-level validation split."""
+
+import argparse
+import copy
+import itertools
+import json
+import math
+import random
+import shutil
+import time
+from pathlib import Path
+
+import torch
+
+from ariadne_bench.hybrid import collate
+from ariadne_bench.adapters import create_adapter
+from ariadne_bench.schema import distribution, load_bundle, write_json
+
+
+def examples(adapter, cases):
+    result = []
+    for case in cases:
+        items = adapter.prepare(case["state"], case["questions"])
+        for (qid, _), item in zip(case["questions"].items(), items):
+            gold = case["gold"][qid]
+            keys = item["keys"]
+            item["target"] = (
+                distribution(gold["probabilities"], keys)[0]
+                if "probabilities" in gold
+                else [float(k == gold["label"]) for k in keys]
+            )
+            result.append(item)
+    return result
+
+
+def forward_loss(adapter, items):
+    batch = collate(items, adapter.pad_id, adapter.device)
+    target = torch.zeros(batch["marker_mask"].shape, device=adapter.device)
+    for row, item in enumerate(items):
+        target[row, : len(item["target"])] = torch.tensor(item["target"], device=adapter.device)
+    with adapter.autocast():
+        logits, _ = adapter.model(**batch)
+        loss = -(target * logits.log_softmax(-1)).sum(-1).mean()
+    return loss, logits, target
+
+
+@torch.no_grad()
+def evaluate(adapter, items, batch_size):
+    adapter.model.eval()
+    total_loss, correct, brier, count = 0.0, 0, 0.0, 0
+    for start in range(0, len(items), batch_size):
+        chunk = items[start : start + batch_size]
+        loss, logits, target = forward_loss(adapter, chunk)
+        total_loss += loss.item() * len(chunk)
+        correct += (logits.argmax(-1) == target.argmax(-1)).sum().item()
+        brier += ((logits.softmax(-1) - target) ** 2).sum().item()
+        count += len(chunk)
+    return {
+        "soft_cross_entropy": total_loss / count,
+        "accuracy": correct / count,
+        "brier_soft": brier / count,
+        "decisions": count,
+    }
+
+
+class ValidationPatience:
+    """Count consecutive epochs that fail to beat the best validation loss."""
+
+    def __init__(self, patience=None, min_epochs=10):
+        if patience is not None and patience < 1:
+            raise ValueError("patience must be positive")
+        if min_epochs < 1:
+            raise ValueError("minimum epochs must be positive")
+        self.patience = patience
+        self.min_epochs = min_epochs
+        self.epochs = 0
+        self.best = float("inf")
+        self.stale = 0
+
+    def observe(self, loss):
+        if not math.isfinite(loss):
+            raise ValueError("validation loss must be finite")
+        self.epochs += 1
+        improved = loss < self.best
+        if improved:
+            self.best = loss
+            self.stale = 0
+        else:
+            self.stale += 1
+        return improved, (
+            self.patience is not None
+            and self.stale >= self.patience
+            and self.epochs >= self.min_epochs
+        )
+
+
+def main():
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--config", required=True)
+    p.add_argument("--train", required=True)
+    p.add_argument("--validation", required=True)
+    p.add_argument("--out", required=True)
+    p.add_argument(
+        "--epochs", type=int, default=3, help="epoch limit; 0 means unlimited with early stopping"
+    )
+    p.add_argument(
+        "--early-stopping-patience",
+        type=int,
+        help="stop after this many consecutive epochs without a new lowest validation loss",
+    )
+    p.add_argument(
+        "--min-epochs",
+        type=int,
+        default=10,
+        help="minimum completed epochs before early stopping can fire",
+    )
+    p.add_argument("--batch-size", type=int, default=8)
+    p.add_argument("--accumulation", type=int, default=4)
+    p.add_argument("--lr", type=float, default=3e-4)
+    p.add_argument("--device", help="override config device, e.g. cuda:1")
+    p.add_argument("--head-lr", type=float, help="full fine-tuning head learning rate")
+    p.add_argument(
+        "--keep-best-only",
+        action="store_true",
+        help="remove superseded checkpoints created by this run",
+    )
+    args = p.parse_args()
+    if (
+        min(args.batch_size, args.accumulation) < 1
+        or args.epochs < 0
+        or args.min_epochs < 1
+        or (args.early_stopping_patience is not None and 0 < args.epochs < args.min_epochs)
+        or (args.epochs == 0 and args.early_stopping_patience is None)
+        or (args.early_stopping_patience is not None and args.early_stopping_patience < 1)
+        or not math.isfinite(args.lr)
+        or args.lr <= 0
+    ):
+        p.error(
+            "use positive batch/accumulation/lr/patience and a positive epoch limit, or epochs=0 with patience"
+        )
+    train, train_manifest = load_bundle(args.train)
+    validation, validation_manifest = load_bundle(args.validation)
+    if train_manifest.get("role") != "train" or validation_manifest.get("role") != "validation":
+        raise ValueError(
+            "explicit train and validation bundle roles are required; do not use benchmark test data"
+        )
+    if {c["id"] for c in train} & {c["id"] for c in validation}:
+        raise ValueError("train and validation case IDs overlap")
+    config = json.loads(Path(args.config).read_text())
+    if args.device:
+        config["options"]["device"] = args.device
+    scope = config["options"].get("training_scope", "bridge")
+    if scope not in ("bridge", "full"):
+        raise ValueError("supported training scopes are bridge and full")
+    if args.head_lr is not None and (not math.isfinite(args.head_lr) or args.head_lr <= 0):
+        p.error("head learning rate must be positive")
+    out = Path(args.out)
+    out.mkdir(parents=True, exist_ok=False)
+    adapter = create_adapter(config)
+    train_items, validation_items = examples(adapter, train), examples(adapter, validation)
+    parameters = [p for p in adapter.model.parameters() if p.requires_grad]
+    frozen_versions = {
+        name: value._version
+        for name, value in adapter.model.named_parameters()
+        if not value.requires_grad
+    }
+    if scope == "full":
+        optimizer_parameters = [
+            {
+                "params": [
+                    p
+                    for n, p in adapter.model.named_parameters()
+                    if p.requires_grad and n.startswith("encoder.")
+                ],
+                "lr": args.lr,
+            },
+            {
+                "params": [
+                    p
+                    for n, p in adapter.model.named_parameters()
+                    if p.requires_grad and not n.startswith("encoder.")
+                ],
+                "lr": args.head_lr or args.lr,
+            },
+        ]
+    else:
+        optimizer_parameters = parameters
+    optimizer = torch.optim.AdamW(optimizer_parameters, lr=args.lr, weight_decay=0.01)
+    rng = random.Random(config["options"].get("seed", 42))
+    log = {
+        "configuration": vars(args),
+        "model": adapter.metadata,
+        "train_manifest": train_manifest,
+        "validation_manifest": validation_manifest,
+        "trainable_names": [n for n, p in adapter.model.named_parameters() if p.requires_grad],
+        "initial_validation": evaluate(adapter, validation_items, args.batch_size),
+        "epochs": [],
+    }
+    write_json(out / "training.json", log)
+    print(
+        json.dumps(
+            {
+                "initial_validation": log["initial_validation"],
+                "trainable_parameters": sum(p.numel() for p in parameters),
+            }
+        ),
+        flush=True,
+    )
+    start_time = time.perf_counter()
+    best_checkpoint = None
+    stopping = ValidationPatience(args.early_stopping_patience, args.min_epochs)
+    epochs = range(1, args.epochs + 1) if args.epochs else itertools.count(1)
+    for epoch in epochs:
+        rng.shuffle(train_items)
+        adapter.training_mode()
+        total_loss, completed = 0.0, 0
+        effective_batch = args.batch_size * args.accumulation
+        for start in range(0, len(train_items), effective_batch):
+            group = train_items[start : start + effective_batch]
+            optimizer.zero_grad(set_to_none=True)
+            for micro in range(0, len(group), args.batch_size):
+                chunk = group[micro : micro + args.batch_size]
+                loss, _, _ = forward_loss(adapter, chunk)
+                if not torch.isfinite(loss):
+                    raise RuntimeError("nonfinite alignment loss")
+                (loss * (len(chunk) / len(group))).backward()
+                total_loss += loss.item() * len(chunk)
+                completed += len(chunk)
+            norm = torch.nn.utils.clip_grad_norm_(parameters, 1.0, error_if_nonfinite=True)
+            optimizer.step()
+            if start == 0 or (start // effective_batch + 1) % 20 == 0:
+                adapter.synchronize()
+                elapsed = time.perf_counter() - start_time
+                processed = (epoch - 1) * len(train_items) + completed
+                remaining = (
+                    ((args.epochs * len(train_items) - processed) * elapsed / processed)
+                    if args.epochs
+                    else None
+                )
+                print(
+                    json.dumps(
+                        {
+                            "epoch": epoch,
+                            "decisions": completed,
+                            "train_ce": total_loss / completed,
+                            "gradient_norm": norm.item(),
+                            "elapsed_s": elapsed,
+                            "estimated_remaining_s": remaining,
+                            "peak_gpu_allocated_gb": torch.cuda.max_memory_allocated(adapter.device)
+                            / 1e9
+                            if adapter.device.type == "cuda"
+                            else None,
+                        }
+                    ),
+                    flush=True,
+                )
+        stats = {
+            "epoch": epoch,
+            "train_soft_cross_entropy": total_loss / completed,
+            "validation": evaluate(adapter, validation_items, args.batch_size),
+        }
+        log["epochs"].append(stats)
+        improved, should_stop = stopping.observe(stats["validation"]["soft_cross_entropy"])
+        if improved:
+            previous_checkpoint = best_checkpoint
+            best_checkpoint = out / f"epoch-{epoch}"
+            adapter.save_checkpoint(
+                best_checkpoint,
+                {
+                    "scope": "full decision path; unused auxiliary action head frozen"
+                    if scope == "full"
+                    else "interface only; pretrained source parameters frozen",
+                    "epoch": epoch,
+                    "train_sha256": train_manifest["sha256"],
+                    "validation_sha256": validation_manifest["sha256"],
+                    "selection": "lowest validation soft cross entropy",
+                    "validation": stats["validation"],
+                },
+            )
+            if args.keep_best_only and previous_checkpoint is not None:
+                # Only delete an earlier checkpoint created within this new run.
+                if previous_checkpoint.parent != out or not previous_checkpoint.name.startswith(
+                    "epoch-"
+                ):
+                    raise ValueError("refusing to remove a checkpoint outside this run")
+                shutil.rmtree(previous_checkpoint)
+        if args.early_stopping_patience is not None:
+            stats["early_stopping"] = {
+                "patience": stopping.patience,
+                "min_epochs": stopping.min_epochs,
+                "epochs_without_improvement": stopping.stale,
+                "best_validation_loss": stopping.best,
+                "improved": improved,
+            }
+        write_json(out / "training.json", log)
+        print(json.dumps(stats), flush=True)
+        if should_stop:
+            break
+    log["stopping"] = {
+        "reason": "early_stopping" if should_stop else "epoch_limit",
+        "epochs_completed": len(log["epochs"]),
+        "patience": stopping.patience,
+        "min_epochs": stopping.min_epochs,
+        "epochs_without_improvement": stopping.stale,
+        "metric": "validation.soft_cross_entropy",
+        "min_delta": 0.0,
+    }
+    assert all(
+        value._version == frozen_versions[name] and value.grad is None
+        for name, value in adapter.model.named_parameters()
+        if name in frozen_versions
+    )
+    log.update(
+        {
+            "elapsed_s": time.perf_counter() - start_time,
+            "frozen_parameters_unchanged": True,
+            "best_checkpoint": str(best_checkpoint.resolve()),
+        }
+    )
+    write_json(out / "training.json", log)
+    trained_config = copy.deepcopy(config)
+    trained_config["name"] = config["name"].replace("(UNTRAINED)", "(alignment only)")
+    if scope == "full":
+        trained_config["name"] = config["name"].replace("(UNTRAINED)", "(full fine-tuned)")
+        trained_config["options"]["inference_only"] = True
+    trained_config["mode"] = "specialist"
+    trained_config["options"]["checkpoint"] = str(best_checkpoint.resolve())
+    write_json(out / "benchmark-config.json", trained_config)
+    print(
+        json.dumps({"best_checkpoint": str(best_checkpoint), "elapsed_s": log["elapsed_s"]}),
+        flush=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

ariadne_bench/experiments/prepare_alignment.py

@@ -0,0 +1,43 @@
+"""Freeze disjoint train/validation/test bundles for the interface experiment."""
+
+import argparse
+import random
+from pathlib import Path
+
+from ariadne_bench.datasets import typed
+from ariadne_bench.schema import save_bundle
+
+
+def main():
+    from datasets import load_dataset
+
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--out", required=True)
+    parser.add_argument("--cache", default=".cache/datasets")
+    args = parser.parse_args()
+    out = Path(args.out)
+    if out.exists():
+        raise ValueError("choose a new output directory")
+    revision = "f7a2487edd7a043a5441a5e9ccc7fe5ddbd9ebe8"
+    ds = load_dataset("LocalLLaMA/typed-decisions", "all", revision=revision, cache_dir=args.cache)
+    train, test = typed(ds["train"]), typed(ds["test"])
+    if {c["id"] for c in train} & {c["id"] for c in test}:
+        raise ValueError("source training and test IDs overlap")
+    random.Random(42).shuffle(train)
+    for role, rows in [("train", train[120:]), ("validation", train[:120]), ("test", test)]:
+        save_bundle(
+            out / role,
+            rows,
+            {
+                "profile": "hybrid-" + role,
+                "role": role,
+                "source": "LocalLLaMA/typed-decisions",
+                "revision": revision,
+                "source_split": "test" if role == "test" else "train",
+                "split_seed": 42,
+            },
+        )
+
+
+if __name__ == "__main__":
+    main()

ariadne_bench/experiments/spectrum.py

@@ -0,0 +1,84 @@
+"""Analyze both the complete learned linear map and its departure from identity."""
+
+import argparse
+import json
+from pathlib import Path
+
+import torch
+from safetensors.torch import load_file, save_file
+
+from ariadne_bench.schema import write_json
+
+
+def describe(matrix):
+    u, s, vh = torch.linalg.svd(matrix.double(), full_matrices=False)
+    energy = s.square()
+    cumulative = energy.cumsum(0) / energy.sum()
+    probability = s / s.sum()
+    entropy = -(probability * probability.clamp_min(1e-300).log()).sum()
+    eigen = torch.linalg.eigvals(matrix.double())
+    result = {
+        "frobenius_norm": matrix.double().norm().item(),
+        "singular_values": s.tolist(),
+        "cumulative_squared_singular_value_fraction": cumulative.tolist(),
+        "singular_value_entropy_effective_rank": entropy.exp().item(),
+        "stable_rank": (energy.sum() / energy[0]).item(),
+        "numerical_rank": int(torch.linalg.matrix_rank(matrix.double())),
+        "condition_number": (s[0] / s[-1]).item(),
+        "ranks_for_energy": {
+            str(p): int(torch.searchsorted(cumulative, torch.tensor(p, dtype=cumulative.dtype))) + 1
+            for p in [0.5, 0.9, 0.95, 0.99]
+        },
+        "eigenvalues": [[z.real.item(), z.imag.item()] for z in eigen],
+    }
+    return result, {"u": u.float(), "s": s.float(), "vh": vh.float()}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--checkpoint", default="runs/hybrid-laya-control/epoch-3")
+    parser.add_argument("--out", default="runs/rank-ablation/spectrum")
+    args = parser.parse_args()
+    source = Path(args.checkpoint)
+    out = Path(args.out)
+    out.mkdir(parents=True, exist_ok=False)
+    state = load_file(source / "adapter.safetensors")
+    weight = state["encoder.bridge.1.weight"]
+    identity = torch.eye(weight.shape[0], dtype=weight.dtype)
+    result = {
+        "source_manifest": json.loads((source / "manifest.json").read_text()),
+        "width": weight.shape[0],
+        "interpretation": "Singular energy describes the matrix, not data-weighted activation variance; W and W-I answer different questions.",
+    }
+    factors = {}
+    for name, matrix in [("weight", weight), ("delta", weight - identity)]:
+        result[name], values = describe(matrix)
+        factors.update({name + "." + k: v for k, v in values.items()})
+    result["relative_delta_frobenius"] = (weight - identity).norm().item() / identity.norm().item()
+    result["layernorm_scale_change_l2"] = (state["encoder.bridge.0.weight"] - 1).norm().item()
+    result["layernorm_bias_l2"] = state["encoder.bridge.0.bias"].norm().item()
+    result["linear_bias_l2"] = state["encoder.bridge.1.bias"].norm().item()
+    save_file({k:v.contiguous() for k,v in factors.items()}, out / "svd.safetensors")
+    write_json(out / "analysis.json", result)
+    print(
+        json.dumps(
+            {
+                name: {
+                    k: v
+                    for k, v in result[name].items()
+                    if k
+                    not in (
+                        "singular_values",
+                        "eigenvalues",
+                        "cumulative_squared_singular_value_fraction",
+                    )
+                }
+                for name in ("weight", "delta")
+            },
+            indent=2,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

ariadne_bench/frozen_input_interface.py

@@ -0,0 +1,64 @@
+"""Train only the exact-identity input affine map on frozen native Laya."""
+
+import copy
+import hashlib
+
+from .full_input_interface import FullInputInterface
+
+
+class FrozenInputInterface(FullInputInterface):
+    def __init__(self, options):
+        if not options.get("add_interface", True):
+            raise ValueError("the frozen-interface adapter requires its linear interface")
+        initial = copy.deepcopy(options)
+        initial.pop("checkpoint", None)
+        initial.pop("inference_only", None)
+        super().__init__(initial)
+        self.options = copy.deepcopy(options)
+        self.model.requires_grad_(False)
+        self.model.encoder.embeddings.interface.requires_grad_(True)
+        self.training_scope = "bridge"
+        self.identity["training_scope"] = "bridge"
+        self.metadata.update(
+            **self.identity,
+            trainable_parameters=sum(p.numel() for p in self.model.parameters() if p.requires_grad),
+            frozen_pretrained_model=True,
+        )
+        self.initial_frozen_sha256 = self.frozen_state_sha256()
+        self.metadata["frozen_state_sha256"] = self.initial_frozen_sha256
+        if options.get("checkpoint"):
+            self.load_checkpoint(options["checkpoint"])
+        if options.get("inference_only", False):
+            self.model.requires_grad_(False)
+        self.model.eval()
+
+    def training_mode(self):
+        # Autograd still traverses the frozen stack to compute interface gradients.
+        self.model.eval()
+
+    def checkpoint_tensor_names(self):
+        return {
+            name
+            for name in self.model.state_dict()
+            if name.startswith("encoder.embeddings.interface.")
+        }
+
+    def frozen_state_sha256(self):
+        digest = hashlib.sha256()
+        for name, value in sorted(self.model.state_dict().items()):
+            if name.startswith("encoder.embeddings.interface."):
+                continue
+            digest.update(name.encode())
+            digest.update(str((value.dtype, tuple(value.shape))).encode())
+            digest.update(value.detach().cpu().contiguous().numpy().tobytes())
+        return digest.hexdigest()
+
+    def save_checkpoint(self, directory, training):
+        actual = self.frozen_state_sha256()
+        if actual != self.initial_frozen_sha256:
+            raise RuntimeError("frozen pretrained tensors changed")
+        super().save_checkpoint(directory, {**training, "frozen_state_sha256": actual})
+
+
+def create(options):
+    return FrozenInputInterface(options)

ariadne_bench/full_finetune.py

@@ -0,0 +1,104 @@
+"""Full task-model fine-tuning of the native Laya specialist."""
+
+import copy
+from pathlib import Path
+
+import torch
+
+from .hybrid import Hybrid
+from .hybrid_control import LayaControl
+
+
+class FullFineTune(Hybrid):
+    prepare = LayaControl.prepare
+
+    def __init__(self, options):
+        import laya
+        import transformers
+
+        self.options = copy.deepcopy(options)
+        if options.get("deterministic", False):
+            from .reproducibility import configure_determinism
+
+            configure_determinism(options.get("seed", 42))
+        torch.manual_seed(options.get("seed", 42))
+        self.device = torch.device(options.get("device", "cpu"))
+        if self.device.type == "cuda":
+            torch.cuda.set_device(self.device)
+        path = Path(options["laya_model"]).resolve(strict=True)
+        self.agent = laya.load(str(path), device=str(self.device))
+        if self.agent.device != self.device:
+            raise RuntimeError("native Laya loaded on an unexpected device")
+        self.model, self.tokenizer = self.agent.model, self.agent.tok
+        self.dtype = self.agent.dtype
+        self.pad_id = self.tokenizer.pad_token_id
+        self.max_len = int(options.get("max_len", self.agent.cfg.get("max_len", 512)))
+        self.head_max_len = self.agent.cfg.get("head_max_len", 192)
+        self.batch_size = int(options.get("batch_size", 16))
+        self.training_scope = "full"
+        self.native_prediction = True
+        self.model.requires_grad_(True)
+        # The supervised decision loss has no action/reward target. All parameters
+        # on the decision path train; the unused auxiliary action head stays frozen.
+        self.model.act_head.requires_grad_(False)
+        if options.get("gradient_checkpointing", False):
+            self.model.encoder.gradient_checkpointing_enable(
+                gradient_checkpointing_kwargs={"use_reentrant": False}
+            )
+            self.model.head_checkpointing = True
+        self.identity = {
+            "format_version": 1,
+            "laya_model": str(path),
+            "training_scope": "full",
+            "prompt_format": "original-laya-v1",
+            "max_len": self.max_len,
+            "head_max_len": self.head_max_len,
+            "native_prediction": True,
+        }
+        self.metadata = {
+            **self.identity,
+            "device": str(self.device),
+            "gpu_name": torch.cuda.get_device_name(self.device)
+            if self.device.type == "cuda"
+            else None,
+            "dtype": str(self.dtype),
+            "torch_version": torch.__version__,
+            "transformers_version": transformers.__version__,
+            "laya_version": laya.__version__,
+            "parameters": sum(p.numel() for p in self.model.parameters()),
+            "trainable_parameters": sum(
+                p.numel() for p in self.model.parameters() if p.requires_grad
+            ),
+            "frozen_auxiliary_action_head": True,
+            "gradient_checkpointing": bool(options.get("gradient_checkpointing", False)),
+            "temperature": list(self.agent.temperature),
+            "temperature_by_options": dict(self.agent.temperature_by_options),
+        }
+        if options.get("deterministic", False):
+            from .reproducibility import settings
+
+            self.metadata["reproducibility"] = settings()
+        if options.get("checkpoint"):
+            self.load_checkpoint(options["checkpoint"])
+        if options.get("inference_only", False):
+            self.model.requires_grad_(False)
+        self.model.eval()
+
+    def close(self):
+        if hasattr(self, "agent"):
+            del self.agent
+        super().close()
+
+    def training_mode(self):
+        self.model.train()
+
+    def checkpoint_tensor_names(self):
+        return set(self.model.state_dict())
+
+    def predict(self, state, questions):
+        self.model.eval()
+        return self.agent.predict(state, questions, max_len=self.max_len)
+
+
+def create(options):
+    return FullFineTune(options)

ariadne_bench/full_input_interface.py

@@ -0,0 +1,77 @@
+"""Jointly fine-tune native Laya and an identity-initialized input affine map."""
+
+import copy
+
+import torch
+from torch import nn
+
+from .full_finetune import FullFineTune
+from .interfaces import IdentityAffine
+
+
+class TrainableInputInterface(nn.Module):
+    """Keep gradients through the original embedding, normalization, and map."""
+
+    def __init__(self, original, width):
+        super().__init__()
+        self.original = original
+        self.interface = IdentityAffine(width)
+
+    @property
+    def tok_embeddings(self):
+        return self.original.tok_embeddings
+
+    def forward(self, input_ids=None, inputs_embeds=None):
+        hidden = self.original(input_ids=input_ids, inputs_embeds=inputs_embeds)
+        return self.interface(hidden)
+
+
+class FullInputInterface(FullFineTune):
+    def __init__(self, options):
+        initial_options = copy.deepcopy(options)
+        initial_options.pop("checkpoint", None)
+        initial_options.pop("inference_only", None)
+        super().__init__(initial_options)
+        self.options = copy.deepcopy(options)
+        self.head_max_len = int(options.get("head_max_len", self.head_max_len))
+        if self.head_max_len < 1:
+            raise ValueError("head_max_len must be positive")
+        torch.set_float32_matmul_precision("highest")
+        add_interface = options.get("add_interface", True)
+        if add_interface:
+            encoder = self.model.encoder
+            encoder.embeddings = TrainableInputInterface(
+                encoder.embeddings, encoder.config.hidden_size
+            ).to(self.device)
+        self.identity.update(
+            head_max_len=self.head_max_len,
+            interface={"kind": "exact_linear", "position": "after native embeddings"}
+            if add_interface
+            else None,
+        )
+        self.metadata.update(
+            **self.identity,
+            parameters=sum(p.numel() for p in self.model.parameters()),
+            trainable_parameters=sum(p.numel() for p in self.model.parameters() if p.requires_grad),
+            float32_matmul_precision=torch.get_float32_matmul_precision(),
+            interface_parameters=sum(
+                p.numel() for p in self.model.encoder.embeddings.interface.parameters()
+            )
+            if add_interface
+            else 0,
+        )
+        if options.get("checkpoint"):
+            self.load_checkpoint(options["checkpoint"])
+        if options.get("inference_only", False):
+            self.model.requires_grad_(False)
+        self.model.eval()
+
+    def predict(self, state, questions):
+        self.model.eval()
+        return self.agent.predict(
+            state, questions, max_len=self.max_len, head_max_len=self.head_max_len
+        )
+
+
+def create(options):
+    return FullInputInterface(options)

ariadne_bench/hybrid.py

@@ -0,0 +1,342 @@
+"""Experimental small causal LM -> ModernBERT -> Laya decision model.
+
+Optional torch/transformers/laya dependencies are imported only by this module.
+No generation, vocabulary output head, or persistent KV cache is used.
+"""
+
+import copy
+import hashlib
+import json
+from contextlib import nullcontext
+from pathlib import Path
+
+import torch
+from torch import nn
+
+from .schema import labels
+
+
+class ContextualEmbeddings(nn.Module):
+    """Replace ModernBERT's entire token lookup / normalization / dropout module."""
+
+    def forward(self, input_ids=None, inputs_embeds=None):
+        if input_ids is not None or inputs_embeds is None:
+            raise ValueError("the hybrid encoder accepts contextual embeddings only")
+        return inputs_embeds
+
+
+class SemanticEncoder(nn.Module):
+    def __init__(self, llm, bert, layer_indices):
+        super().__init__()
+        indices = list(layer_indices)
+        if not indices or indices != sorted(set(indices)):
+            raise ValueError("bert_layers must be a nonempty, increasing list of distinct indices")
+        if indices[0] < 0 or indices[-1] >= len(bert.layers):
+            raise ValueError("bert_layers contains an out-of-range index")
+        self.llm, self.bert = llm, bert
+        self.layer_indices = indices
+        self.config = bert.config
+        self.bert.layers = nn.ModuleList([bert.layers[i] for i in indices])
+        self.bert.config.num_hidden_layers = len(indices)
+        # Preserve each retained block's original attention type and rotary settings.
+        self.bert.config.layer_types = [layer.attention_type for layer in self.bert.layers]
+        self.bert.embeddings = ContextualEmbeddings()
+        width = llm.config.hidden_size
+        self.bridge = nn.Sequential(nn.LayerNorm(width), nn.Linear(width, self.config.hidden_size))
+        if width == self.config.hidden_size:
+            nn.init.eye_(self.bridge[1].weight)
+        nn.init.zeros_(self.bridge[1].bias)
+        self.llm.requires_grad_(False).eval()
+
+    def train(self, mode=True):
+        super().train(mode)
+        self.llm.eval()
+        return self
+
+    def forward(self, input_ids, attention_mask, **kwargs):
+        with torch.no_grad():
+            h = self.llm(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                use_cache=False,
+                output_hidden_states=False,
+                return_dict=True,
+            ).last_hidden_state
+        h = self.bridge(h)
+        return self.bert(inputs_embeds=h, attention_mask=attention_mask, return_dict=True)
+
+
+def build_item(tokenizer, state_ids, question, max_len=1024, min_state_tokens=32):
+    """Use the LM tokenizer throughout; gather the final token of each whole option.
+
+    State comes first so causal option features can attend to it. Only the state
+    may be truncated. Reject an overlong rubric instead of silently deleting options.
+    """
+    from laya.common import QTYPES, render_options
+
+    keys = labels(question)
+    q = {"t": question["type"], "ins": question["instructions"], "crit": question.get("criteria")}
+    if "labels" in question:
+        q["labels"] = question["labels"]
+
+    def encode(text):
+        return tokenizer.encode(text, add_special_tokens=False)
+
+    prefix = encode("State:\n")
+    heading = encode(f"\nQuestion ({q['t']}): {q['ins']}\nOptions:")
+    options = [encode("\nOption: " + text) for text in render_options(q)]
+    if len(options) != len(keys) or any(not option for option in options):
+        raise ValueError("every option must have a token span")
+    room = max_len - len(prefix) - len(heading) - sum(map(len, options))
+    if room < min(len(state_ids), min_state_tokens):
+        raise ValueError("question and full options exceed the token budget; increase max_len")
+    ids = prefix + state_ids[:room] + heading
+    markers = []
+    for option in options:
+        ids.extend(option)
+        markers.append(len(ids) - 1)
+    return {
+        "ids": ids,
+        "markers": markers,
+        "qtype": QTYPES[question["type"]],
+        "keys": keys,
+        "state_tokens_original": len(state_ids),
+        "state_tokens_used": min(len(state_ids), room),
+    }
+
+
+def collate(items, pad_token_id, device="cpu"):
+    if not items:
+        raise ValueError("cannot collate an empty decision batch")
+    batch, length, options = (
+        len(items),
+        max(len(i["ids"]) for i in items),
+        max(len(i["markers"]) for i in items),
+    )
+    ids = torch.full((batch, length), pad_token_id, dtype=torch.long, device=device)
+    attention = torch.zeros_like(ids)
+    positions = torch.zeros((batch, options), dtype=torch.long, device=device)
+    marker_mask = torch.zeros_like(positions, dtype=torch.bool)
+    for row, item in enumerate(items):
+        n, k = len(item["ids"]), len(item["markers"])
+        ids[row, :n] = torch.tensor(item["ids"], device=device)
+        attention[row, :n] = 1
+        positions[row, :k] = torch.tensor(item["markers"], device=device)
+        marker_mask[row, :k] = True
+    return {
+        "input_ids": ids,
+        "attention_mask": attention,
+        "marker_pos": positions,
+        "marker_mask": marker_mask,
+        "qtype": torch.tensor([i["qtype"] for i in items], device=device),
+    }
+
+
+class Hybrid:
+    """Harness adapter; local source checkpoints remain unchanged."""
+
+    def __init__(self, options):
+        import laya
+        import transformers
+        from transformers import AutoModel, AutoTokenizer
+
+        self.options = copy.deepcopy(options)
+        torch.manual_seed(options.get("seed", 42))
+        self.device = torch.device(options.get("device", "cpu"))
+        if self.device.type not in ("cpu", "cuda"):
+            raise ValueError("the prototype supports CPU and CUDA")
+        self.dtype = torch.bfloat16 if self.device.type == "cuda" else torch.float32
+        llm_path = Path(options["llm_model"]).resolve(strict=True)
+        laya_path = Path(options["laya_model"]).resolve(strict=True)
+        self.tokenizer = AutoTokenizer.from_pretrained(llm_path, local_files_only=True)
+        self.pad_id = self.tokenizer.pad_token_id
+        if self.pad_id is None:
+            self.pad_id = self.tokenizer.eos_token_id
+        if self.pad_id is None:
+            raise ValueError("the small LM needs a pad or EOS token")
+        llm = AutoModel.from_pretrained(
+            llm_path, local_files_only=True, dtype=self.dtype, attn_implementation="sdpa"
+        )
+        if sum(p.numel() for p in llm.parameters()) >= 1_000_000_000:
+            raise ValueError("this prototype requires a small LM with fewer than 1B parameters")
+        agent = laya.load(str(laya_path), device="cpu")
+        self.model = agent.model
+        indices = options.get("bert_layers", list(range(28)))
+        self.model.encoder = SemanticEncoder(llm, self.model.encoder, indices)
+        self.training_scope = options.get("training_scope", "bridge")
+        self.model.requires_grad_(False)
+        if self.training_scope == "bridge":
+            self.model.encoder.bridge.requires_grad_(True)
+        elif self.training_scope == "downstream":
+            for name, parameter in self.model.named_parameters():
+                parameter.requires_grad_(not name.startswith(("encoder.llm.", "act_head.")))
+        else:
+            raise ValueError("training_scope must be bridge or downstream")
+        self.max_len = int(options.get("max_len", 1024))
+        self.min_state_tokens = int(options.get("min_state_tokens", 32))
+        self.batch_size = int(options.get("batch_size", 16))
+        if self.batch_size < 1 or self.max_len < 1 or self.min_state_tokens < 1:
+            raise ValueError("batch_size, max_len, and min_state_tokens must be positive")
+        if self.max_len > min(
+            llm.config.max_position_embeddings, self.model.encoder.config.max_position_embeddings
+        ):
+            raise ValueError("max_len exceeds a backbone's positional limit")
+        self.identity = {
+            "format_version": 1,
+            "llm_model": str(llm_path),
+            "laya_model": str(laya_path),
+            "bert_layers": list(indices),
+            "max_len": self.max_len,
+            "min_state_tokens": self.min_state_tokens,
+            "prompt_format": "state-question-whole-options-v1",
+            "training_scope": self.training_scope,
+        }
+        self.metadata = {
+            **self.identity,
+            "training_status": "untrained hybrid connection",
+            "device": str(self.device),
+            "gpu_name": torch.cuda.get_device_name(self.device)
+            if self.device.type == "cuda"
+            else None,
+            "dtype": str(self.dtype),
+            "torch_version": torch.__version__,
+            "transformers_version": transformers.__version__,
+            "batch_size": self.batch_size,
+            "llm_layers": llm.config.num_hidden_layers,
+            "llm_width": llm.config.hidden_size,
+            "bert_width": self.model.encoder.config.hidden_size,
+            "head_layers": len(self.model.head.layers) if self.model.head else 0,
+            "parameters": sum(p.numel() for p in self.model.parameters()),
+            "llm_parameters": sum(p.numel() for p in llm.parameters()),
+            "trainable_parameters": sum(
+                p.numel() for p in self.model.parameters() if p.requires_grad
+            ),
+            "temperature": 1.0,
+            "use_cache": False,
+        }
+        if options.get("checkpoint"):
+            self.load_checkpoint(options["checkpoint"])
+        self.model.to(self.device).eval()
+
+    def training_mode(self):
+        # Frozen modules remain deterministic, but autograd still traverses BERT
+        # and the scorer to train the connection below them.
+        self.model.train(self.training_scope == "downstream")
+
+    def checkpoint_tensor_names(self):
+        if self.training_scope == "bridge":
+            return {k for k in self.model.state_dict() if k.startswith("encoder.bridge.")}
+        return {k for k in self.model.state_dict() if not k.startswith("encoder.llm.")}
+
+    def autocast(self):
+        return (
+            torch.autocast("cuda", dtype=self.dtype)
+            if self.device.type == "cuda"
+            else nullcontext()
+        )
+
+    def prepare(self, state, questions):
+        from laya.common import serialize_state
+
+        if not questions:
+            raise ValueError("at least one question is required")
+        state_ids = self.tokenizer.encode(serialize_state(state), add_special_tokens=False)
+        return [
+            build_item(self.tokenizer, state_ids, q, self.max_len, self.min_state_tokens)
+            for q in questions.values()
+        ]
+
+    @torch.inference_mode()
+    def predict(self, state, questions):
+        self.model.eval()
+        items = self.prepare(state, questions)
+        probabilities = []
+        for start in range(0, len(items), self.batch_size):
+            chunk = items[start : start + self.batch_size]
+            with self.autocast():
+                logits, _ = self.model(**collate(chunk, self.pad_id, self.device))
+            p = logits.softmax(-1).cpu().tolist()
+            probabilities.extend([row[: len(item["keys"])] for row, item in zip(p, chunk)])
+        answers = {}
+        for (qid, q), item, p in zip(questions.items(), items, probabilities):
+            keys = item["keys"]
+            answer = {"type": q["type"], "probabilities": dict(zip(keys, p)), "confidence": max(p)}
+            if q["type"] == "choice":
+                answer["choice"] = keys[max(range(len(p)), key=p.__getitem__)]
+            elif q["type"] == "noul":
+                answer["noul"] = p[1]
+            else:
+                answer["score"] = sum(i * value for i, value in enumerate(p))
+            answers[qid] = answer
+        return {
+            "model": "experimental-lm-laya-hybrid",
+            "answers": answers,
+            "usage": {
+                "llm_forward_calls": (len(items) + self.batch_size - 1) // self.batch_size,
+                "generated_tokens": 0,
+                "input_tokens": sum(len(i["ids"]) for i in items),
+                "truncated_state_questions": sum(
+                    i["state_tokens_used"] < i["state_tokens_original"] for i in items
+                ),
+            },
+        }
+
+    def synchronize(self):
+        if self.device.type == "cuda":
+            torch.cuda.synchronize(self.device)
+
+    def close(self):
+        if hasattr(self, "model"):
+            del self.model
+        if self.device.type == "cuda":
+            torch.cuda.empty_cache()
+
+    def save_checkpoint(self, directory, training):
+        from safetensors.torch import save_file
+        from .schema import write_json
+
+        directory = Path(directory)
+        directory.mkdir(parents=True, exist_ok=False)
+        # Frozen source weights are referenced; only the selected training scope is saved.
+        names = self.checkpoint_tensor_names()
+        state = {
+            k: v.detach().cpu().contiguous()
+            for k, v in self.model.state_dict().items()
+            if k in names
+        }
+        weights = directory / "adapter.safetensors"
+        save_file(state, weights)
+        write_json(
+            directory / "manifest.json",
+            {
+                "identity": self.identity,
+                "training": training,
+                "weights_sha256": hashlib.sha256(weights.read_bytes()).hexdigest(),
+            },
+        )
+
+    def load_checkpoint(self, directory):
+        from safetensors.torch import load_file
+
+        directory = Path(directory)
+        manifest = json.loads((directory / "manifest.json").read_text())
+        if manifest["identity"] != self.identity:
+            raise ValueError("hybrid checkpoint architecture or tokenizer provenance differs")
+        weights = directory / "adapter.safetensors"
+        actual_hash = hashlib.sha256(weights.read_bytes()).hexdigest()
+        if actual_hash != manifest["weights_sha256"]:
+            raise ValueError("hybrid checkpoint hash mismatch")
+        state = load_file(weights)
+        expected = self.checkpoint_tensor_names()
+        if set(state) != expected:
+            raise ValueError(
+                "hybrid checkpoint must contain exactly the tensors for its training scope"
+            )
+        self.model.load_state_dict(state, strict=False)
+        self.metadata.update(
+            {"training_status": manifest["training"], "checkpoint_sha256": actual_hash}
+        )
+
+
+def create(options):
+    return Hybrid(options)

ariadne_bench/hybrid_control.py

@@ -0,0 +1,169 @@
+"""Matched trainable-interface control using Laya's original token embeddings."""
+
+import copy
+from pathlib import Path
+
+import torch
+from torch import nn
+from transformers.modeling_outputs import BaseModelOutput
+
+from .hybrid import Hybrid, SemanticEncoder
+from .interfaces import make_interface
+from .schema import labels
+
+
+class FrozenTokenEncoder(nn.Module):
+    def __init__(self, embeddings, config):
+        super().__init__()
+        self.embeddings = embeddings
+        self.config = copy.deepcopy(config)
+        self.config.num_hidden_layers = 0
+
+    def forward(self, input_ids, **kwargs):
+        return BaseModelOutput(last_hidden_state=self.embeddings(input_ids=input_ids))
+
+
+class LayaControl(Hybrid):
+    def __init__(self, options):
+        import laya
+        import transformers
+
+        self.options = copy.deepcopy(options)
+        torch.manual_seed(options.get("seed", 42))
+        self.device = torch.device(options.get("device", "cpu"))
+        if self.device.type not in ("cpu", "cuda"):
+            raise ValueError("the prototype supports CPU and CUDA")
+        self.dtype = torch.bfloat16 if self.device.type == "cuda" else torch.float32
+        self.training_scope = "bridge"
+        laya_path = Path(options["laya_model"]).resolve(strict=True)
+        self.native_prediction = bool(options.get("native_prediction", False))
+        agent = laya.load(
+            str(laya_path), device=str(self.device) if self.native_prediction else "cpu"
+        )
+        if self.native_prediction:
+            if agent.device != self.device:
+                raise RuntimeError("native Laya loaded on an unexpected device")
+            self.agent = agent
+            self.dtype = agent.dtype
+        self.tokenizer, self.model = agent.tok, agent.model
+        self.pad_id = self.tokenizer.pad_token_id
+        bert = self.model.encoder
+        tokens = FrozenTokenEncoder(bert.embeddings, bert.config)
+        indices = options.get("bert_layers", list(range(len(bert.layers))))
+        self.model.encoder = SemanticEncoder(tokens, bert, indices)
+        if "interface" in options:
+            specification = options["interface"]
+            if specification.get("kind", "").startswith("exact_linear"):
+                torch.set_float32_matmul_precision("highest")
+            self.model.encoder.bridge = make_interface(bert.config.hidden_size, specification)
+        self.model.requires_grad_(False)
+        self.model.encoder.bridge.requires_grad_(True)
+        self.max_len = int(options.get("max_len", 512))
+        self.head_max_len = agent.cfg.get("head_max_len", 192)
+        self.batch_size = int(options.get("batch_size", 16))
+        if self.max_len < 1 or self.batch_size < 1:
+            raise ValueError("max_len and batch_size must be positive")
+        self.identity = {
+            "format_version": 1,
+            "feature_source": "original Laya token embeddings",
+            "laya_model": str(laya_path),
+            "bert_layers": list(indices),
+            "max_len": self.max_len,
+            "head_max_len": self.head_max_len,
+            "prompt_format": "original-laya-v1",
+            "training_scope": "bridge",
+        }
+        # Preserve the identity schema of existing saved LayerNorm+Linear checkpoints.
+        if "interface" in options:
+            self.identity["interface"] = copy.deepcopy(options["interface"])
+        if self.native_prediction:
+            self.identity["native_prediction"] = True
+        self.metadata = {
+            **self.identity,
+            "training_status": "untrained control interface",
+            "device": str(self.device),
+            "gpu_name": torch.cuda.get_device_name(self.device)
+            if self.device.type == "cuda"
+            else None,
+            "dtype": str(self.dtype),
+            "torch_version": torch.__version__,
+            "transformers_version": transformers.__version__,
+            "batch_size": self.batch_size,
+            "parameters": sum(p.numel() for p in self.model.parameters()),
+            "trainable_parameters": sum(
+                p.numel() for p in self.model.parameters() if p.requires_grad
+            ),
+            "temperature": 1.0,
+        }
+        if "interface" in options:
+            self.metadata["float32_matmul_precision"] = torch.get_float32_matmul_precision()
+        if self.native_prediction:
+            self.metadata.update(
+                {
+                    "prediction_path": "native Laya SDK",
+                    "temperature": list(agent.temperature),
+                    "temperature_by_options": dict(agent.temperature_by_options),
+                    "laya_version": laya.__version__,
+                }
+            )
+        if options.get("checkpoint"):
+            self.load_checkpoint(options["checkpoint"])
+        self.model.to(self.device).eval()
+
+    def prepare(self, state, questions):
+        from laya.common import QTYPES, build_sequence, serialize_state
+
+        if not questions:
+            raise ValueError("at least one question is required")
+        state_ids = self.tokenizer.encode(
+            serialize_state(state).replace(self.tokenizer.mask_token, " "), add_special_tokens=False
+        )
+        result = []
+        for question in questions.values():
+            q = {
+                "t": question["type"],
+                "ins": question["instructions"],
+                "crit": question.get("criteria"),
+            }
+            if "labels" in question:
+                q["labels"] = question["labels"]
+            ids, markers = build_sequence(
+                self.tokenizer, state, q, self.max_len, self.head_max_len, state_ids=state_ids
+            )
+            keys = labels(question)
+            if len(keys) != len(markers):
+                raise ValueError("the Laya token budget removed an option")
+            # build_sequence appends state after its final head SEP and leaves a final SEP.
+            last_head_sep = next(
+                i for i in range(markers[-1] + 1, len(ids)) if ids[i] == self.tokenizer.sep_token_id
+            )
+            state_used = min(len(state_ids), max(0, len(ids) - last_head_sep - 2))
+            result.append(
+                {
+                    "ids": ids,
+                    "markers": markers,
+                    "qtype": QTYPES[q["t"]],
+                    "keys": keys,
+                    "state_tokens_original": len(state_ids),
+                    "state_tokens_used": state_used,
+                }
+            )
+        return result
+
+    def predict(self, state, questions):
+        if self.native_prediction:
+            self.model.eval()
+            return self.agent.predict(state, questions, max_len=self.max_len)
+        response = super().predict(state, questions)
+        response["model"] = "laya-interface-control"
+        response["usage"]["llm_forward_calls"] = 0
+        return response
+
+    def close(self):
+        if hasattr(self, "agent"):
+            del self.agent
+        super().close()
+
+
+def create(options):
+    return LayaControl(options)

ariadne_bench/interfaces.py

@@ -0,0 +1,121 @@
+"""Input transformations with explicit initialization and low-rank factors."""
+
+import math
+
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+
+class IdentityAffine(nn.Module):
+    """Affine map initialized as identity, preserving input precision under AMP.
+
+    Autocasting an identity GEMM to BF16 would round FP32 embeddings before
+    BERT sees them. Keep this small interface in FP32 and return the input dtype.
+    CUDA callers must use highest float32 matmul precision (no TF32 rounding).
+    """
+
+    def __init__(self, width, residual=False):
+        super().__init__()
+        self.residual = residual
+        self.proj = nn.Linear(width, width)
+        if residual:
+            nn.init.zeros_(self.proj.weight)
+        else:
+            nn.init.eye_(self.proj.weight)
+        nn.init.zeros_(self.proj.bias)
+
+    def forward(self, x):
+        with torch.autocast(device_type=x.device.type, enabled=False):
+            value = F.linear(x.float(), self.proj.weight.float(), self.proj.bias.float())
+            if self.residual:
+                value = x.float() + value
+        return value.to(x.dtype)
+
+
+class LowRankLinear(nn.Module):
+    def __init__(self, width, rank, residual=False):
+        super().__init__()
+        if not 1 <= rank <= width:
+            raise ValueError("rank must be between one and the hidden width")
+        self.rank, self.residual = rank, residual
+        self.a = nn.Parameter(torch.empty(rank, width))
+        self.b = nn.Parameter(torch.empty(width, rank))
+        self.bias = nn.Parameter(torch.zeros(width))
+        if residual:
+            nn.init.kaiming_uniform_(self.a, a=math.sqrt(5))
+            nn.init.zeros_(self.b)
+        else:
+            # A pure rank-r map cannot start at identity. Start as an orthogonal
+            # rank-r projection instead, and record this in experiment metadata.
+            q, _ = torch.linalg.qr(torch.randn(width, rank), mode="reduced")
+            with torch.no_grad():
+                self.a.copy_(q.T)
+                self.b.copy_(q)
+
+    def forward(self, x):
+        value = F.linear(F.linear(x, self.a), self.b, self.bias)
+        return x + value if self.residual else value
+
+    def dense_weight(self):
+        weight = self.b @ self.a
+        if self.residual:
+            weight = weight + torch.eye(weight.shape[0], device=weight.device, dtype=weight.dtype)
+        return weight
+
+    @classmethod
+    def from_svd(cls, u, s, vh, bias, rank, residual=False):
+        layer = cls(u.shape[0], rank, residual=residual)
+        with torch.no_grad():
+            root = s[:rank].sqrt()
+            layer.a.copy_(root[:, None] * vh[:rank])
+            layer.b.copy_(u[:, :rank] * root[None, :])
+            layer.bias.copy_(bias)
+        return layer
+
+
+class Diagonal(nn.Module):
+    def __init__(self, width):
+        super().__init__()
+        self.scale = nn.Parameter(torch.ones(width))
+        self.bias = nn.Parameter(torch.zeros(width))
+
+    def forward(self, x):
+        return x * self.scale + self.bias
+
+
+class ResidualInput(nn.Module):
+    def __init__(self, width):
+        super().__init__()
+        self.norm = nn.LayerNorm(width)
+        self.linear = nn.Linear(width, width)
+        nn.init.zeros_(self.linear.weight)
+        nn.init.zeros_(self.linear.bias)
+
+    def forward(self, x):
+        return x + self.linear(self.norm(x))
+
+
+def make_interface(width, specification):
+    kind = specification.get("kind", "norm_linear")
+    if kind in ("exact_linear", "exact_linear_residual"):
+        return IdentityAffine(width, residual=kind == "exact_linear_residual")
+    if kind == "identity":
+        return nn.Identity()
+    if kind == "norm":
+        return nn.LayerNorm(width)
+    if kind == "diagonal":
+        return Diagonal(width)
+    if kind == "residual":
+        return ResidualInput(width)
+    if kind in ("low_rank", "low_rank_delta"):
+        return nn.Sequential(
+            nn.LayerNorm(width),
+            LowRankLinear(width, specification["rank"], residual=kind == "low_rank_delta"),
+        )
+    if kind in ("linear", "norm_linear"):
+        linear = nn.Linear(width, width)
+        nn.init.eye_(linear.weight)
+        nn.init.zeros_(linear.bias)
+        return linear if kind == "linear" else nn.Sequential(nn.LayerNorm(width), linear)
+    raise ValueError(f"unknown interface kind: {kind}")

ariadne_bench/metrics.py

@@ -0,0 +1,229 @@
+"""Explicit scoring conventions; no fitted temperatures or test-set thresholds."""
+
+import math
+from collections import defaultdict
+from statistics import mean
+
+from .schema import distribution, labels
+
+METRIC_VERSION = "1:sum-brier:top-label-ece:right-closed:eps-1e-12"
+
+
+def decision(case, qid, answer):
+    q, gold = case["questions"][qid], case["gold"][qid]
+    keys = labels(q)
+    p = [answer["probabilities"][k] for k in keys]
+    target = keys.index(gold["label"])
+    row = {
+        "case_id": case["id"],
+        "suite": case["suite"],
+        "question": qid,
+        "workflow": case.get("workflow", ""),
+        "language": case.get("language", ""),
+        "type": q["type"],
+        "gold": gold["label"],
+        "prediction": answer["label"],
+        "correct": float(answer["label"] == gold["label"]),
+        "confidence": answer["confidence"],
+        "brier_hard": sum((v - float(i == target)) ** 2 for i, v in enumerate(p)),
+        "nll_hard": -math.log(max(p[target], 1e-12)),
+        "zero_probability_gold": float(p[target] == 0),
+        "raw_probability_sum": answer["raw_probability_sum"],
+    }
+    if "probabilities" in gold:
+        g, _ = distribution(gold["probabilities"], keys)
+        row.update(
+            soft_accuracy=sum(a * b for a, b in zip(p, g)),
+            brier_soft=sum((a - b) ** 2 for a, b in zip(p, g)),
+            kl_gold_to_prediction=sum(
+                b * math.log(b / max(a, 1e-12)) for a, b in zip(p, g) if b > 0
+            ),
+            total_variation=0.5 * sum(abs(a - b) for a, b in zip(p, g)),
+        )
+    if q["type"] == "score":
+        score = sum(i * v for i, v in enumerate(p))
+        error = abs(score - gold.get("score", float(gold["label"])))
+        row.update(score_mae=error, within_one_level=float(error <= 1))
+    return row
+
+
+def ece(rows, bins):
+    # [0, 1/B], (1/B, 2/B], ..., ((B-1)/B, 1]; includes confidence 0 and 1.
+    buckets = defaultdict(list)
+    for row in rows:
+        index = max(0, min(bins - 1, math.ceil(row["confidence"] * bins) - 1))
+        buckets[index].append(row)
+    return sum(
+        len(rs) * abs(mean(r["confidence"] for r in rs) - mean(r["correct"] for r in rs))
+        for rs in buckets.values()
+    ) / len(rows)
+
+
+def summarize(rows, attempted, bins=15):
+    result = {
+        "attempted": attempted,
+        "valid": len(rows),
+        "failed": attempted - len(rows),
+        "coverage": len(rows) / attempted if attempted else None,
+        "accuracy_all": sum(r["correct"] for r in rows) / attempted if attempted else None,
+    }
+    if not rows:
+        result.update(accuracy_valid=None, ece_top_label=None, macro_f1=None)
+        return result
+    result.update(
+        accuracy_valid=mean(r["correct"] for r in rows),
+        ece_top_label=ece(rows, bins),
+        mean_confidence=mean(r["confidence"] for r in rows),
+    )
+    for metric in (
+        "brier_hard",
+        "nll_hard",
+        "zero_probability_gold",
+        "soft_accuracy",
+        "brier_soft",
+        "kl_gold_to_prediction",
+        "total_variation",
+        "score_mae",
+        "within_one_level",
+    ):
+        values = [r[metric] for r in rows if metric in r]
+        if values:
+            result[metric] = mean(values)
+            result[metric + "_n"] = len(values)
+    # Semantic label namespaces prevent conflating e.g. ordinal 0 across unrelated tasks.
+    confusion = defaultdict(lambda: [0, 0, 0])
+    for r in rows:
+        ns = (r["suite"], r["workflow"], r["question"])
+        gold, pred = ns + (r["gold"],), ns + (r["prediction"],)
+        if gold == pred:
+            confusion[gold][0] += 1
+        else:
+            confusion[pred][1] += 1
+            confusion[gold][2] += 1
+    result["macro_f1"] = mean(2 * tp / (2 * tp + fp + fn) for tp, fp, fn in confusion.values())
+    # Include complete confidence ties; these are descriptive curves, not chosen thresholds.
+    ranked = sorted(rows, key=lambda r: -r["confidence"])
+    curve, errors = [], 0
+    for i, row in enumerate(ranked, 1):
+        errors += 1 - row["correct"]
+        if i == len(ranked) or ranked[i]["confidence"] != row["confidence"]:
+            curve.append(
+                {
+                    "threshold": row["confidence"],
+                    "coverage": i / attempted,
+                    "risk": errors / i,
+                    "accepted": i,
+                }
+            )
+    result["risk_coverage"] = curve
+    return result
+
+
+def percentile(values, q):
+    if not values:
+        return None
+    values = sorted(values)
+    pos = (len(values) - 1) * q
+    lo, hi = math.floor(pos), math.ceil(pos)
+    return values[lo] + (values[hi] - values[lo]) * (pos - lo)
+
+
+def timing(records):
+    values = [r["latency_ms"] for r in records if r.get("latency_ms") is not None]
+    seconds = sum(values) / 1000
+    return {
+        "requests": len(records),
+        "timed_requests": len(values),
+        "p50_ms": percentile(values, 0.5),
+        "p95_ms": percentile(values, 0.95),
+        "p99_ms": percentile(values, 0.99),
+        "total_request_seconds": seconds,
+        "attempted_decisions_per_second": sum(r["question_count"] for r in records) / seconds
+        if seconds and len(values) == len(records)
+        else None,
+    }
+
+
+def report(cases, records, bins=15, tolerance=0.02):
+    from .schema import normalize_answer
+
+    by_id = {r["case_id"]: r for r in records}
+    if len(by_id) != len(records) or set(by_id) != {c["id"] for c in cases}:
+        raise ValueError("records must contain each case exactly once")
+    rows, errors = [], []
+    counts = {field: defaultdict(int) for field in ("suite", "workflow", "language", "type")}
+    for case in cases:
+        record = by_id[case["id"]]
+        response = record.get("response") or {}
+        answers = response.get("answers", {}) if isinstance(response, dict) else {}
+        if not isinstance(answers, dict):
+            answers = {}
+        extra_keys = set(answers) - set(case["questions"])
+        for qid, q in case["questions"].items():
+            for field in counts:
+                counts[field][q["type"] if field == "type" else case.get(field, "")] += 1
+            try:
+                if record.get("error"):
+                    raise ValueError(record["error"])
+                if extra_keys:
+                    raise ValueError("response contains unexpected question keys")
+                a = normalize_answer(q, answers.get(qid), tolerance)
+                rows.append(decision(case, qid, a))
+            except (ValueError, TypeError, KeyError) as exc:
+                errors.append({"case_id": case["id"], "question": qid, "error": str(exc)})
+    total = sum(len(c["questions"]) for c in cases)
+    result = {
+        "metric_version": METRIC_VERSION,
+        "ece_bins": bins,
+        "probability_sum_tolerance": tolerance,
+        "overall": summarize(rows, total, bins),
+        "latency": timing(records),
+        "errors": errors,
+    }
+    for field, sizes in counts.items():
+        result["by_" + field] = {
+            key: summarize([r for r in rows if r[field] == key], n, bins)
+            for key, n in sizes.items()
+            if key
+        }
+    result["latency_by_question_count"] = {
+        str(n): timing([r for r in records if r["question_count"] == n])
+        for n in sorted({r["question_count"] for r in records})
+    }
+    usages = [
+        r.get("response", {}).get("usage", {})
+        for r in records
+        if isinstance(r.get("response"), dict)
+    ]
+    usages = [
+        u
+        for u in usages
+        if isinstance(u, dict)
+        and all(
+            isinstance(u.get(k, 0), int) and not isinstance(u.get(k, 0), bool) and u.get(k, 0) >= 0
+            for k in ("input_tokens", "output_tokens")
+        )
+    ]
+    result["usage"] = {
+        "requests_with_usage": sum(bool(u) for u in usages),
+        "input_tokens_known": sum(u.get("input_tokens", 0) or 0 for u in usages),
+        "output_tokens_known": sum(u.get("output_tokens", 0) or 0 for u in usages),
+    }
+    paired = defaultdict(dict)
+    case_map = {c["id"]: c for c in cases}
+    for row in rows:
+        case = case_map[row["case_id"]]
+        base = case.get("base_id", case["id"])
+        paired[(base, row["question"])][case.get("permutation", 0)] = row["prediction"]
+    flips = [
+        pred != variants[0]
+        for variants in paired.values()
+        if 0 in variants
+        for variant, pred in variants.items()
+        if variant != 0
+    ]
+    result["option_order"] = {
+        "valid_pairs": len(flips),
+        "flip_rate": mean(flips) if flips else None,
+    }
+    return result

ariadne_bench/portable.py

@@ -0,0 +1,88 @@
+"""Load a compact affine adapter with pinned, independently verified base files."""
+
+import hashlib
+import json
+from pathlib import Path
+
+
+def sha256(path):
+    return hashlib.sha256(Path(path).read_bytes()).hexdigest()
+
+
+def load_release(
+    directory, device="cpu", seed=None, base_path=None, local_files_only=False, deterministic=True
+):
+    from huggingface_hub import snapshot_download
+    from safetensors.torch import load_file
+    from .frozen_input_interface import FrozenInputInterface
+
+    root = Path(directory).resolve()
+    config = json.loads((root / "adapter_config.json").read_text())
+    if config["format_version"] != 1 or config["interface"] != "exact_linear":
+        raise ValueError("unsupported adapter format")
+    selected = config["candidate_seed"] if seed is None else int(seed)
+    entry = config["seeds"][str(selected)]
+    weights = (root / entry["weights_file"]).resolve()
+    if not weights.is_relative_to(root):
+        raise ValueError("adapter weights must be inside the release directory")
+    if sha256(weights) != entry["weights_sha256"]:
+        raise ValueError("adapter checkpoint checksum mismatch")
+    source = (
+        Path(base_path)
+        if base_path
+        else Path(
+            snapshot_download(
+                config["base_model"],
+                revision=config["base_revision"],
+                local_files_only=local_files_only,
+                allow_patterns=[
+                    "model.safetensors",
+                    "rl_agent_config.json",
+                    "encoder/*",
+                    "tokenizer/*",
+                ],
+            )
+        )
+    )
+    for name, expected in config["base_config_sha256"].items():
+        if sha256(source / name) != expected:
+            raise ValueError(f"base tokenizer/configuration mismatch: {name}")
+    adapter = FrozenInputInterface(
+        {
+            "laya_model": str(source.resolve()),
+            "device": device,
+            "max_len": config["max_len"],
+            "head_max_len": config["head_max_len"],
+            "seed": selected,
+            "training_scope": "bridge",
+            "add_interface": True,
+            "deterministic": deterministic,
+        }
+    )
+    try:
+        if adapter.initial_frozen_sha256 != config["frozen_state_sha256"]:
+            raise ValueError("base model tensors do not match the pinned training source")
+        state = load_file(str(weights))
+        if set(state) != adapter.checkpoint_tensor_names():
+            raise ValueError("checkpoint must contain exactly the affine weight and bias")
+        adapter.model.load_state_dict(state, strict=False)
+        adapter.model.requires_grad_(False)
+        adapter.model.eval()
+        adapter.metadata.update(
+            trainable_parameters=0,
+            checkpoint_sha256=entry["weights_sha256"],
+            portable_adapter={
+                "base_model": config["base_model"],
+                "base_revision": config["base_revision"],
+                "seed": selected,
+                "selected_epoch": entry["selected_epoch"],
+            },
+        )
+    except Exception:
+        adapter.close()
+        raise
+    return adapter
+
+
+def create(options):
+    return load_release(**options)

ariadne_bench/reproducibility.py

@@ -0,0 +1,48 @@
+"""Opt-in deterministic execution for a fixed GPU, runtime, and data order."""
+
+import os
+import random
+
+import numpy as np
+import torch
+
+
+def configure_determinism(seed):
+    workspace = os.environ.get("CUBLAS_WORKSPACE_CONFIG")
+    if workspace is None:
+        if torch.cuda.is_initialized():
+            raise RuntimeError("set CUBLAS_WORKSPACE_CONFIG=:4096:8 before CUDA initialization")
+        os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
+    elif workspace not in {":4096:8", ":16:8"}:
+        raise ValueError("deterministic execution requires a supported cuBLAS workspace config")
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.use_deterministic_algorithms(True, warn_only=False)
+    torch.backends.cudnn.benchmark = False
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.allow_tf32 = False
+    torch.set_float32_matmul_precision("highest")
+    # Avoid a backend whose deterministic backward support varies by cuDNN version.
+    torch.backends.cuda.enable_cudnn_sdp(False)
+    return settings()
+
+
+def settings():
+    return {
+        "deterministic_algorithms": torch.are_deterministic_algorithms_enabled(),
+        "warn_only": torch.is_deterministic_algorithms_warn_only_enabled(),
+        "cublas_workspace_config": os.environ.get("CUBLAS_WORKSPACE_CONFIG"),
+        "python_hash_seed": os.environ.get("PYTHONHASHSEED"),
+        "cudnn_benchmark": torch.backends.cudnn.benchmark,
+        "cudnn_deterministic": torch.backends.cudnn.deterministic,
+        "cudnn_allow_tf32": torch.backends.cudnn.allow_tf32,
+        "float32_matmul_precision": torch.get_float32_matmul_precision(),
+        "cudnn_sdp_enabled": torch.backends.cuda.cudnn_sdp_enabled(),
+        "flash_sdp_enabled": torch.backends.cuda.flash_sdp_enabled(),
+        "mem_efficient_sdp_enabled": torch.backends.cuda.mem_efficient_sdp_enabled(),
+        "math_sdp_enabled": torch.backends.cuda.math_sdp_enabled(),
+        "cuda_version": torch.version.cuda,
+        "cudnn_version": torch.backends.cudnn.version(),
+        "scope": "fixed hardware/runtime; cross-platform bitwise agreement is not promised",
+    }

ariadne_bench/runner.py

@@ -0,0 +1,257 @@
+import copy
+import importlib.metadata
+import json
+import platform
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+
+from . import __version__
+from .adapters import create_adapter
+from .metrics import report
+from .schema import digest, dumps, load_bundle, read_jsonl, write_json
+
+
+def environment():
+    versions = {}
+    for name in ("laya", "torch", "transformers", "datasets", "huggingface-hub"):
+        try:
+            versions[name] = importlib.metadata.version(name)
+        except importlib.metadata.PackageNotFoundError:
+            pass
+    return {
+        "python": platform.python_version(),
+        "platform": platform.platform(),
+        "processor": platform.processor(),
+        "packages": versions,
+        "harness": __version__,
+    }
+
+
+def check_config(config):
+    # Config files are copied into artifacts. Credentials must stay in the environment.
+    def visit(value):
+        if isinstance(value, dict):
+            for key, child in value.items():
+                if key.lower() in ("api_key", "token", "password", "authorization", "secret"):
+                    raise ValueError(
+                        "put credentials in environment variables, not the saved config"
+                    )
+                visit(child)
+        elif isinstance(value, list):
+            for child in value:
+                visit(child)
+
+    visit(config)
+    if config.get("mode") not in ("generalist", "specialist", "reference"):
+        raise ValueError("config mode must be generalist, specialist or reference")
+    if not config.get("name"):
+        raise ValueError("config requires a display name")
+    if "endpoint" in config:
+        from urllib.parse import urlsplit
+
+        url = urlsplit(config["endpoint"])
+        if url.username or url.password or url.query or url.fragment:
+            raise ValueError("endpoint must not contain credentials, a query or a fragment")
+
+
+def invoke(adapter, case):
+    start = time.perf_counter()
+    result = {"case_id": case["id"], "question_count": len(case["questions"])}
+    try:
+        if hasattr(adapter, "synchronize"):
+            adapter.synchronize()
+        start = time.perf_counter()
+        # Copy protects the frozen bundle from model code that mutates its arguments.
+        response = adapter.predict(copy.deepcopy(case["state"]), copy.deepcopy(case["questions"]))
+        if hasattr(adapter, "synchronize"):
+            adapter.synchronize()
+        result["latency_ms"] = (time.perf_counter() - start) * 1000
+        dumps(response)
+        result["response"] = response
+    except Exception as exc:
+        result["latency_ms"] = (time.perf_counter() - start) * 1000
+        # Arbitrary plugin errors can embed secret configuration. Keep their type only.
+        result["error"] = type(exc).__name__
+        if isinstance(exc, RuntimeError) and str(exc).startswith("HTTP "):
+            result["error"] = str(exc)
+    return result
+
+
+def render_report(result):
+    meta = result["run"]
+    lines = [
+        f"# {meta['config']['name']}",
+        "",
+        f"Mode: **{meta['config']['mode']}**. Profile: `{meta['manifest']['profile']}`.",
+        "",
+        f"Cases SHA-256: `{meta['manifest']['sha256']}`.",
+        "",
+        "| Suite | Valid / attempted | Accuracy (all) | Brier (hard) | Brier (soft) | ECE |",
+        "|---|---:|---:|---:|---:|---:|",
+    ]
+    for suite, r in result["by_suite"].items():
+
+        def fmt(v):
+            return "—" if v is None else f"{v:.4f}"
+
+        lines.append(
+            f"| {suite} | {r['valid']} / {r['attempted']} | {fmt(r['accuracy_all'])} | "
+            f"{fmt(r.get('brier_hard'))} | {fmt(r.get('brier_soft'))} | {fmt(r.get('ece_top_label'))} |"
+        )
+    t = result["latency"]
+    lines.extend(
+        [
+            "",
+            f"Request p50: {t['p50_ms']} ms; p95: {t['p95_ms']} ms.",
+            "",
+            "Accuracy (all) counts invalid or missing decisions as incorrect. Proper scores and ECE use valid decisions; check coverage.",
+            "ECE uses the predicted option's probability, not the provider's confidence field. Brier sums over classes.",
+            "Latency includes the adapter call and local device synchronization; model loading and explicit warmups are separate.",
+            "Published reference numbers and local timing runs require matching protocols before comparison.",
+            "",
+        ]
+    )
+    return "\n".join(lines)
+
+
+def run(bundle, config_path, output, bins=15, tolerance=0.02, warmup=0, resume=False):
+    cases, manifest = load_bundle(bundle)
+    config = json.loads(Path(config_path).read_text())
+    check_config(config)
+    directory = Path(output)
+    identity = digest(
+        {
+            "manifest": manifest,
+            "config": config,
+            "bins": bins,
+            "tolerance": tolerance,
+            "warmup": warmup,
+            "harness": __version__,
+        }
+    )
+    if resume:
+        saved = json.loads((directory / "run.json").read_text())
+        if saved["identity"] != identity:
+            raise ValueError("resume requires identical data, config and scoring settings")
+        records = (
+            read_jsonl(directory / "predictions.jsonl")
+            if (directory / "predictions.jsonl").exists()
+            else []
+        )
+        ids = [r["case_id"] for r in records]
+        if ids != [c["id"] for c in cases[: len(ids)]]:
+            raise ValueError("saved records are not the expected case prefix")
+        metadata = saved
+    else:
+        if directory.exists():
+            raise ValueError("run directory exists; choose a new path or --resume")
+        records = []
+        metadata = {
+            "identity": identity,
+            "created_utc": datetime.now(timezone.utc).isoformat(),
+            "config": config,
+            "manifest": manifest,
+            "environment": environment(),
+            "ece_bins": bins,
+            "probability_sum_tolerance": tolerance,
+            "warmup_per_question_count": warmup,
+            "sessions": [],
+        }
+    if len(records) < len(cases):
+        load_start = time.perf_counter()
+        adapter = create_adapter(config)
+        session = {
+            "model_load_seconds": time.perf_counter() - load_start,
+            "adapter_metadata": getattr(adapter, "metadata", {}),
+            "start_index": len(records),
+            "utc": datetime.now(timezone.utc).isoformat(),
+        }
+        metadata["sessions"].append(session)
+        directory.mkdir(parents=True, exist_ok=True)
+        write_json(directory / "run.json", metadata)
+        try:
+            shapes = {}
+            for case in cases:
+                shapes.setdefault(len(case["questions"]), case)
+            with (directory / "warmup.jsonl").open("a") as f:
+                for case in shapes.values():
+                    for _ in range(warmup):
+                        record = invoke(adapter, case)
+                        f.write(dumps(record) + "\n")
+                        f.flush()
+                        if record.get("error"):
+                            raise ValueError(f"warmup failed: {record['error']}; details saved")
+            with (directory / "predictions.jsonl").open("a") as f:
+                for case in cases[len(records) :]:
+                    record = invoke(adapter, case)
+                    f.write(dumps(record) + "\n")
+                    f.flush()
+                    records.append(record)
+                    if len(records) % 50 == 0 or len(records) == len(cases):
+                        print(f"Completed {len(records)}/{len(cases)} requests", flush=True)
+        finally:
+            if hasattr(adapter, "close"):
+                adapter.close()
+    result = report(cases, records, bins, tolerance)
+    result["run"] = metadata
+    warmups = read_jsonl(directory / "warmup.jsonl")
+    result["warmup"] = {
+        "requests": len(warmups),
+        "usage": [r.get("response", {}).get("usage") for r in warmups],
+    }
+    result["resolved_models"] = sorted(
+        {str(r["response"].get("model")) for r in records if isinstance(r.get("response"), dict)}
+    )
+    write_json(directory / "report.json", result)
+    (directory / "report.md").write_text(render_report(result))
+    return result
+
+
+def score(bundle, predictions, output, bins=15, tolerance=0.02):
+    cases, manifest = load_bundle(bundle)
+    records = read_jsonl(predictions)
+    mapping = {r["case_id"]: r for r in records}
+    expected = {c["id"] for c in cases}
+    if len(mapping) != len(records) or set(mapping) - expected:
+        raise ValueError("duplicate or unknown prediction IDs")
+    complete = []
+    for case in cases:
+        r = dict(mapping.get(case["id"], {"case_id": case["id"], "error": "missing prediction"}))
+        r["question_count"] = len(case["questions"])
+        complete.append(r)
+    result = report(cases, complete, bins, tolerance)
+    result["manifest"] = manifest
+    write_json(output, result)
+    return result
+
+
+def compare(paths):
+    reports = [json.loads(Path(p).read_text()) for p in paths]
+    protocols = {
+        (
+            r["run"]["manifest"]["sha256"],
+            r["metric_version"],
+            r["ece_bins"],
+            r["probability_sum_tolerance"],
+        )
+        for r in reports
+    }
+    if len(protocols) != 1:
+        raise ValueError("comparison requires identical frozen cases and scoring conventions")
+    lines = [
+        "| Model | Mode | Valid / attempted | Accuracy (all) | ECE | Request p50 ms |",
+        "|---|---|---:|---:|---:|---:|",
+    ]
+    for r in reports:
+        c, s = r["run"]["config"], r["overall"]
+        e = "—" if s["ece_top_label"] is None else f"{s['ece_top_label']:.4f}"
+        lines.append(
+            f"| {c['name']} | {c['mode']} | {s['valid']}/{s['attempted']} | "
+            f"{s['accuracy_all']:.4f} | {e} | {r['latency']['p50_ms']} |"
+        )
+    lines += [
+        "",
+        "Training modes, runtime versions, hardware and network location remain material differences.",
+    ]
+    return "\n".join(lines)

ariadne_bench/schema.py

@@ -0,0 +1,172 @@
+"""One ordered question schema for data, native APIs and local model plugins."""
+
+import hashlib
+import json
+import math
+from pathlib import Path
+
+
+def dumps(value):
+    # Ordering is intentional: changing option order changes the model's input.
+    return json.dumps(value, ensure_ascii=False, separators=(",", ":"), allow_nan=False)
+
+
+def digest(value):
+    return hashlib.sha256(dumps(value).encode()).hexdigest()
+
+
+def read_jsonl(path):
+    return [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]
+
+
+def write_json(path, value):
+    Path(path).write_text(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False) + "\n")
+
+
+def labels(question):
+    kind = question["type"]
+    if kind == "noul":
+        return ["false", "true"]
+    if kind == "choice":
+        criteria = question["criteria"]
+        if not isinstance(criteria, dict) or not criteria:
+            raise ValueError("choice requires a nonempty criteria map")
+        return list(criteria)
+    if kind == "score":
+        criteria = question["criteria"]
+        if not isinstance(criteria, list) or len(criteria) < 2:
+            raise ValueError("score requires at least two ordered criteria")
+        return [str(i) for i in range(len(criteria))]
+    raise ValueError(f"unknown question type: {kind}")
+
+
+def distribution(raw, keys, tolerance=0.02):
+    if not isinstance(raw, dict) or set(raw) != set(keys):
+        raise ValueError("probability keys must match every option exactly")
+    if any(isinstance(raw[k], bool) or not isinstance(raw[k], (int, float)) for k in keys):
+        raise ValueError("probabilities must be numbers")
+    values = [float(raw[k]) for k in keys]
+    if any(not math.isfinite(p) or p < 0 or p > 1 for p in values):
+        raise ValueError("probabilities must be finite and in [0, 1]")
+    total = sum(values)
+    if total <= 0 or abs(total - 1) > tolerance + 1e-12:
+        raise ValueError(f"probabilities sum to {total}, outside rounding tolerance {tolerance}")
+    return [p / total for p in values], total
+
+
+def validate_case(case):
+    if not isinstance(case.get("id"), str) or not case["id"]:
+        raise ValueError("case requires a nonempty string id")
+    if not isinstance(case.get("suite"), str) or not case["suite"]:
+        raise ValueError("case requires a suite")
+    if not isinstance(case.get("state"), (str, dict, list)):
+        raise ValueError("state must be text, an object or an array")
+    questions, gold = case.get("questions"), case.get("gold")
+    if not isinstance(questions, dict) or not questions or set(questions) != set(gold or {}):
+        raise ValueError("each question needs exactly one gold entry")
+    for qid, question in questions.items():
+        if "instructions" not in question:
+            raise ValueError(f"missing instructions: {qid}")
+        keys = labels(question)
+        if gold[qid]["label"] not in keys:
+            raise ValueError(f"gold label outside options: {qid}")
+        if "probabilities" in gold[qid]:
+            distribution(gold[qid]["probabilities"], keys)
+        if "score" in gold[qid]:
+            score = gold[qid]["score"]
+            if not math.isfinite(score) or not 0 <= score <= len(keys) - 1:
+                raise ValueError(f"invalid gold score: {qid}")
+    dumps(case)
+
+
+def normalize_answer(question, answer, tolerance=0.02):
+    if not isinstance(answer, dict) or answer.get("type") != question["type"]:
+        raise ValueError("answer type does not match question")
+    keys = labels(question)
+    if question["type"] == "noul":
+        p = answer.get("noul")
+        if isinstance(p, bool) or not isinstance(p, (int, float)):
+            raise ValueError("noul must be a probability")
+        raw = {"false": 1 - p, "true": p}
+    else:
+        raw = answer.get("probabilities")
+    probs, total = distribution(raw, keys, tolerance)
+    idx = max(range(len(keys)), key=probs.__getitem__)
+    if question["type"] == "choice":
+        chosen = answer.get("choice")
+        if chosen not in keys or probs[keys.index(chosen)] < max(probs) - 1e-12:
+            raise ValueError("choice is missing or disagrees with the distribution")
+        idx = keys.index(chosen)  # retain the provider's tie break
+    if question["type"] == "score":
+        score = answer.get("score")
+        if isinstance(score, bool) or not isinstance(score, (int, float)):
+            raise ValueError("score is missing or nonnumeric")
+        if not math.isfinite(score) or not 0 <= score <= len(keys) - 1:
+            raise ValueError("score is outside its rubric")
+        expected = sum(i * p for i, p in enumerate(probs))
+        if abs(score - expected) > max(0.05, tolerance * (len(keys) - 1)):
+            raise ValueError("score disagrees with the distribution's expectation")
+    if "confidence" in answer:
+        c = answer["confidence"]
+        if (
+            isinstance(c, bool)
+            or not isinstance(c, (int, float))
+            or not math.isfinite(c)
+            or not 0 <= c <= 1
+        ):
+            raise ValueError("invalid provider confidence")
+    return {
+        "label": keys[idx],
+        "probabilities": dict(zip(keys, probs)),
+        "confidence": probs[idx],
+        "raw_probability_sum": total,
+        "provider_confidence": answer.get("confidence"),
+    }
+
+
+def save_bundle(directory, cases, metadata):
+    directory = Path(directory)
+    if directory.exists():
+        raise ValueError(f"output already exists: {directory}; use a new directory")
+    ids = set()
+    for case in cases:
+        validate_case(case)
+        if case["id"] in ids:
+            raise ValueError(f"duplicate case id: {case['id']}")
+        ids.add(case["id"])
+    if not cases:
+        raise ValueError("empty benchmark")
+    content = "".join(dumps(c) + "\n" for c in cases).encode()
+    directory.mkdir(parents=True)
+    (directory / "cases.jsonl").write_bytes(content)
+    manifest = {
+        "format_version": 1,
+        "sha256": hashlib.sha256(content).hexdigest(),
+        "cases": len(cases),
+        "decisions": sum(len(c["questions"]) for c in cases),
+        **metadata,
+    }
+    write_json(directory / "manifest.json", manifest)
+    return manifest
+
+
+def load_bundle(directory):
+    directory = Path(directory)
+    manifest = json.loads((directory / "manifest.json").read_text())
+    content = (directory / "cases.jsonl").read_bytes()
+    if hashlib.sha256(content).hexdigest() != manifest["sha256"]:
+        raise ValueError("case file does not match its manifest hash")
+    cases = read_jsonl(directory / "cases.jsonl")
+    ids = set()
+    for case in cases:
+        validate_case(case)
+        if case["id"] in ids:
+            raise ValueError("duplicate case id")
+        ids.add(case["id"])
+    if (
+        not cases
+        or len(cases) != manifest["cases"]
+        or sum(len(c["questions"]) for c in cases) != manifest["decisions"]
+    ):
+        raise ValueError("manifest counts disagree with cases")
+    return cases, manifest

benchmarks/sources.lock.json

@@ -0,0 +1,102 @@
+{
+  "LocalLLaMA/typed-decisions": {
+    "revision": "f7a2487edd7a043a5441a5e9ccc7fe5ddbd9ebe8",
+    "license": "apache-2.0",
+    "url": "https://huggingface.co/datasets/LocalLLaMA/typed-decisions"
+  },
+  "SetFit/enron_spam": {
+    "revision": "1916f66c89d52221ae33eb57d44498b4f3a5df22",
+    "license": "see source dataset card",
+    "url": "https://huggingface.co/datasets/SetFit/enron_spam"
+  },
+  "SetFit/sst5": {
+    "revision": "e51bdcd8cd3a30da231967c1a249ba59361279a3",
+    "license": "see source dataset card",
+    "url": "https://huggingface.co/datasets/SetFit/sst5"
+  },
+  "Tobi-Bueck/customer-support-tickets": {
+    "revision": "ddf1c81a5475992c4fa6752bf1e8b4e31f07bbeb",
+    "license": "cc-by-nc-4.0",
+    "url": "https://huggingface.co/datasets/Tobi-Bueck/customer-support-tickets"
+  },
+  "btzsc/btzsc": {
+    "revision": "fef2a2ac62b69c58670047dddf045c53d7c3cb5e",
+    "license": "see source dataset card",
+    "url": "https://huggingface.co/datasets/btzsc/btzsc"
+  },
+  "dair-ai/emotion": {
+    "revision": "cab853a1dbdf4c42c2b3ef2173804746df8825fe",
+    "license": [
+      "other"
+    ],
+    "url": "https://huggingface.co/datasets/dair-ai/emotion"
+  },
+  "deepset/prompt-injections": {
+    "revision": "4f61ecb038e9c3fb77e21034b22511b523772cdd",
+    "license": "apache-2.0",
+    "url": "https://huggingface.co/datasets/deepset/prompt-injections"
+  },
+  "facebook/xnli": {
+    "revision": "b8dd5d7af51114dbda02c0e3f6133f332186418e",
+    "license": "see source dataset card",
+    "url": "https://huggingface.co/datasets/facebook/xnli"
+  },
+  "fancyzhx/ag_news": {
+    "revision": "eb185aade064a813bc0b7f42de02595523103ca4",
+    "license": [
+      "unknown"
+    ],
+    "url": "https://huggingface.co/datasets/fancyzhx/ag_news"
+  },
+  "google-research-datasets/mbpp": {
+    "revision": "4bb6404fdc6cacfda99d4ac4205087b89d32030c",
+    "license": [
+      "cc-by-4.0"
+    ],
+    "url": "https://huggingface.co/datasets/google-research-datasets/mbpp"
+  },
+  "google/boolq": {
+    "revision": "35b264d03638db9f4ce671b711558bf7ff0f80d5",
+    "license": [
+      "cc-by-sa-3.0"
+    ],
+    "url": "https://huggingface.co/datasets/google/boolq"
+  },
+  "lmsys/toxic-chat": {
+    "revision": "29df8e4dba60e1f4af4b4075c0705c5b313548a8",
+    "license": "cc-by-nc-4.0",
+    "url": "https://huggingface.co/datasets/lmsys/toxic-chat"
+  },
+  "microsoft/ms_marco": {
+    "revision": "a47ee7aae8d7d466ba15f9f0bfac3b3681087b3a",
+    "license": "see source dataset card",
+    "url": "https://huggingface.co/datasets/microsoft/ms_marco"
+  },
+  "mteb/amazon_massive_intent": {
+    "revision": "940fd47a81eaa7f2cc7b129674d945d618ac38c2",
+    "license": "apache-2.0",
+    "url": "https://huggingface.co/datasets/mteb/amazon_massive_intent"
+  },
+  "mteb/amazon_massive_scenario": {
+    "revision": "58871793b91addb7c5f7afff26ccf08737fb6697",
+    "license": "apache-2.0",
+    "url": "https://huggingface.co/datasets/mteb/amazon_massive_scenario"
+  },
+  "mteb/banking77": {
+    "revision": "18072d2685ea682290f7b8924d94c62acc19c0b2",
+    "license": "mit",
+    "url": "https://huggingface.co/datasets/mteb/banking77"
+  },
+  "openai/gsm8k": {
+    "revision": "740312add88f781978c0658806c59bc2815b9866",
+    "license": [
+      "mit"
+    ],
+    "url": "https://huggingface.co/datasets/openai/gsm8k"
+  },
+  "zefang-liu/phishing-email-dataset": {
+    "revision": "34085a032c123ca237f314a01a67909cdea35e34",
+    "license": "lgpl-3.0",
+    "url": "https://huggingface.co/datasets/zefang-liu/phishing-email-dataset"
+  }
+}

environment.json

@@ -0,0 +1,44 @@
+{
+  "torch_version": "2.13.0+cu130",
+  "transformers_version": "5.17.0",
+  "laya_version": "0.3.20",
+  "dtype": "torch.bfloat16",
+  "gpu_name": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+  "reproducibility": {
+    "deterministic_algorithms": true,
+    "warn_only": false,
+    "cublas_workspace_config": ":4096:8",
+    "python_hash_seed": "0",
+    "cudnn_benchmark": false,
+    "cudnn_deterministic": true,
+    "cudnn_allow_tf32": false,
+    "float32_matmul_precision": "highest",
+    "cudnn_sdp_enabled": false,
+    "flash_sdp_enabled": true,
+    "mem_efficient_sdp_enabled": true,
+    "math_sdp_enabled": true,
+    "cuda_version": "13.0",
+    "cudnn_version": 92000,
+    "scope": "fixed hardware/runtime; cross-platform bitwise agreement is not promised"
+  },
+  "benchmark_runtime": {
+    "python": "3.12.3",
+    "platform": "Linux-7.0.0-34-generic-x86_64-with-glibc2.39",
+    "processor": "x86_64",
+    "packages": {
+      "laya": "0.3.20",
+      "torch": "2.13.0+cu130",
+      "transformers": "5.17.0",
+      "datasets": "5.0.1",
+      "huggingface-hub": "1.33.0"
+    },
+    "harness": "0.1.0"
+  },
+  "process_environment": {
+    "PYTHONHASHSEED": "0",
+    "CUBLAS_WORKSPACE_CONFIG": ":4096:8",
+    "USE_TF": "0",
+    "OMP_NUM_THREADS": "4",
+    "TOKENIZERS_PARALLELISM": "false"
+  }
+}

evidence/adapter-equivalence.json

@@ -0,0 +1,39 @@
+{
+  "device": "cuda:1",
+  "deterministic_algorithms": true,
+  "trained_checkpoint_reference": "runs/base-frozen-linear-seeds/seed-0/training/training.json",
+  "diagnostic_only_no_optimizer_updates": true,
+  "gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+  "forward_passed": true,
+  "all_frozen_modules_in_eval_mode": true,
+  "native_attention_implementation": "sdpa",
+  "all_gradients_exact": true,
+  "comparisons": [
+    {
+      "stage": "initial",
+      "decisions": 8,
+      "loss_difference": 0.0,
+      "max_logit_difference": 0.0,
+      "max_gradient_difference": 0.0,
+      "same_wrapper_repeat_gradient_difference": 0.0,
+      "legacy_repeat_gradient_difference": 0.0,
+      "max_gradient_magnitude": 0.4774176776409149,
+      "cuda_rng_unchanged": true,
+      "forward_exact": true,
+      "exact_match": true
+    },
+    {
+      "stage": "trained_seed_0",
+      "decisions": 8,
+      "loss_difference": 0.0,
+      "max_logit_difference": 0.0,
+      "max_gradient_difference": 0.0,
+      "same_wrapper_repeat_gradient_difference": 0.0,
+      "legacy_repeat_gradient_difference": 0.0,
+      "max_gradient_magnitude": 0.01077677309513092,
+      "cuda_rng_unchanged": true,
+      "forward_exact": true,
+      "exact_match": true
+    }
+  ]
+}

evidence/audit.json

@@ -0,0 +1,59 @@
+{
+  "frozen_state_sha256": "7560d83b1e1c17cce7eb67b25674fc8e7f0958bac4b47f559c725f8695c4d59b",
+  "verified_decisions": 25580,
+  "initial_identity_verified": true,
+  "runs": [
+    {
+      "seed": 0,
+      "checkpoint_sha256": "bcb37755de54782bc4d1f79615539f48940aedb2b733dadc1033fba2f2304cf8",
+      "checkpoint_bytes": 4198616,
+      "all_pretrained_tensors_unchanged": true,
+      "independent_scoring_matches": true,
+      "min_epochs": 10,
+      "stop_epoch": 23,
+      "minimum_epoch_rule_verified": true
+    },
+    {
+      "seed": 1,
+      "checkpoint_sha256": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
+      "checkpoint_bytes": 4198616,
+      "all_pretrained_tensors_unchanged": true,
+      "independent_scoring_matches": true,
+      "min_epochs": 10,
+      "stop_epoch": 17,
+      "minimum_epoch_rule_verified": true
+    },
+    {
+      "seed": 2,
+      "checkpoint_sha256": "61391c3869c1f4a8449444db6a9e3b660692eefa25e3c149cee0d6af5daf264c",
+      "checkpoint_bytes": 4198616,
+      "all_pretrained_tensors_unchanged": true,
+      "independent_scoring_matches": true,
+      "min_epochs": 10,
+      "stop_epoch": 18,
+      "minimum_epoch_rule_verified": true
+    },
+    {
+      "seed": 3,
+      "checkpoint_sha256": "26ed2443b8ea146d3fab98e94ab55356557b9ef10ac5b8ba656b1a37cf4f0f64",
+      "checkpoint_bytes": 4198616,
+      "all_pretrained_tensors_unchanged": true,
+      "independent_scoring_matches": true,
+      "min_epochs": 10,
+      "stop_epoch": 27,
+      "minimum_epoch_rule_verified": true
+    },
+    {
+      "seed": 4,
+      "checkpoint_sha256": "603a83ca4996e32b4753250ea6b07856e7cc5196b41e635cdad35ee4270eca38",
+      "checkpoint_bytes": 4198616,
+      "all_pretrained_tensors_unchanged": true,
+      "independent_scoring_matches": true,
+      "min_epochs": 10,
+      "stop_epoch": 10,
+      "minimum_epoch_rule_verified": true
+    }
+  ],
+  "verified_runs": 5,
+  "statistics_verified": true
+}

evidence/candidate-uncertainty.json

@@ -0,0 +1,76 @@
+{
+  "candidate_seed": 1,
+  "selected_epoch": 14,
+  "benchmark_cases_sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
+  "resamples": 10000,
+  "bootstrap_seed_per_suite": 42,
+  "unit": "cases, preserving all decisions in each case",
+  "interval": "paired percentile 95%; percentage points",
+  "limits": "descriptive resampling of the measured cases for a fixed model; no multiple-comparison correction or training-seed uncertainty; not used for checkpoint selection",
+  "suites": {
+    "typed-decisions": {
+      "cases": 400,
+      "accuracy_difference_pp": 40.7,
+      "case_bootstrap_95_interval_pp": [
+        37.3,
+        44.101249999999986
+      ]
+    },
+    "ag-news": {
+      "cases": 600,
+      "accuracy_difference_pp": -1.6666666666666667,
+      "case_bootstrap_95_interval_pp": [
+        -3.5,
+        0.16666666666666666
+      ]
+    },
+    "boolq": {
+      "cases": 600,
+      "accuracy_difference_pp": -3.1666666666666665,
+      "case_bootstrap_95_interval_pp": [
+        -5.833333333333333,
+        -0.6666666666666666
+      ]
+    },
+    "emotion": {
+      "cases": 600,
+      "accuracy_difference_pp": 0.0,
+      "case_bootstrap_95_interval_pp": [
+        -2.5,
+        2.6666666666666665
+      ]
+    },
+    "prompt-injections": {
+      "cases": 116,
+      "accuracy_difference_pp": 2.586206896551724,
+      "case_bootstrap_95_interval_pp": [
+        -5.172413793103448,
+        10.344827586206897
+      ]
+    },
+    "sst5": {
+      "cases": 600,
+      "accuracy_difference_pp": 5.0,
+      "case_bootstrap_95_interval_pp": [
+        1.1666666666666667,
+        9.0
+      ]
+    },
+    "massive-intent.en": {
+      "cases": 300,
+      "accuracy_difference_pp": -4.333333333333333,
+      "case_bootstrap_95_interval_pp": [
+        -8.333333333333334,
+        -0.3333333333333333
+      ]
+    },
+    "xnli.en": {
+      "cases": 300,
+      "accuracy_difference_pp": 1.6666666666666667,
+      "case_bootstrap_95_interval_pp": [
+        -1.0,
+        4.666666666666667
+      ]
+    }
+  }
+}

evidence/identity-check.json

@@ -0,0 +1,6 @@
+{
+  "passed": true,
+  "cases": 3516,
+  "decisions": 5116,
+  "mismatched_cases": []
+}

evidence/replay-check.json

@@ -0,0 +1,32 @@
+{
+  "passed": true,
+  "seed": 0,
+  "separate_processes": true,
+  "epochs_per_process": 2,
+  "optimizer_updates_per_process": 338,
+  "initial_validation_exact": true,
+  "epoch_losses_and_metrics_exact": true,
+  "selected_checkpoint_tensors_bitwise_equal": true,
+  "checkpoint_sha256": [
+    "3a30ce9881420cebc61829e52b41a9e669a114b4dffe0f8f836831572bd70331",
+    "3a30ce9881420cebc61829e52b41a9e669a114b4dffe0f8f836831572bd70331"
+  ],
+  "reproducibility": {
+    "deterministic_algorithms": true,
+    "warn_only": false,
+    "cublas_workspace_config": ":4096:8",
+    "python_hash_seed": "0",
+    "cudnn_benchmark": false,
+    "cudnn_deterministic": true,
+    "cudnn_allow_tf32": false,
+    "float32_matmul_precision": "highest",
+    "cudnn_sdp_enabled": false,
+    "flash_sdp_enabled": true,
+    "mem_efficient_sdp_enabled": true,
+    "math_sdp_enabled": true,
+    "cuda_version": "13.0",
+    "cudnn_version": 92000,
+    "scope": "fixed hardware/runtime; cross-platform bitwise agreement is not promised"
+  },
+  "limit": "replay verified for this data, device, runtime, and two-epoch prefix"
+}

evidence/seed-0-history.json

@@ -0,0 +1,416 @@
+{
+  "seed": 0,
+  "initial_validation": {
+    "soft_cross_entropy": 1.544869564374288,
+    "accuracy": 0.36833333333333335,
+    "brier_soft": 0.43051501592000324,
+    "decisions": 600
+  },
+  "epochs": [
+    {
+      "epoch": 1,
+      "train_soft_cross_entropy": 1.1371864740936843,
+      "validation": {
+        "soft_cross_entropy": 1.195652896563212,
+        "accuracy": 0.3433333333333333,
+        "brier_soft": 0.2543176457285881,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.195652896563212,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 2,
+      "train_soft_cross_entropy": 1.1867035720966481,
+      "validation": {
+        "soft_cross_entropy": 1.1503166087468466,
+        "accuracy": 0.43666666666666665,
+        "brier_soft": 0.2273359453678131,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.1503166087468466,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 3,
+      "train_soft_cross_entropy": 1.1479326088340194,
+      "validation": {
+        "soft_cross_entropy": 1.1359616955121359,
+        "accuracy": 0.4583333333333333,
+        "brier_soft": 0.21933125893274943,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.1359616955121359,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 4,
+      "train_soft_cross_entropy": 1.17192115077266,
+      "validation": {
+        "soft_cross_entropy": 1.1432366434733072,
+        "accuracy": 0.4533333333333333,
+        "brier_soft": 0.22181885927915573,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.1359616955121359,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 5,
+      "train_soft_cross_entropy": 1.1521123818114951,
+      "validation": {
+        "soft_cross_entropy": 1.1532048479715984,
+        "accuracy": 0.42,
+        "brier_soft": 0.22427557816108068,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 1.1359616955121359,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 6,
+      "train_soft_cross_entropy": 1.139600450904281,
+      "validation": {
+        "soft_cross_entropy": 1.1171702599525453,
+        "accuracy": 0.465,
+        "brier_soft": 0.21267332146565118,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.1171702599525453,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 7,
+      "train_soft_cross_entropy": 1.1084874327094467,
+      "validation": {
+        "soft_cross_entropy": 1.101010274887085,
+        "accuracy": 0.48333333333333334,
+        "brier_soft": 0.20341493268807728,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.101010274887085,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 8,
+      "train_soft_cross_entropy": 1.0972168815577472,
+      "validation": {
+        "soft_cross_entropy": 1.0927943936983744,
+        "accuracy": 0.4866666666666667,
+        "brier_soft": 0.20212342927853266,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0927943936983744,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 9,
+      "train_soft_cross_entropy": 1.0959268622045164,
+      "validation": {
+        "soft_cross_entropy": 1.0886747964223227,
+        "accuracy": 0.485,
+        "brier_soft": 0.1981955200433731,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0886747964223227,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 10,
+      "train_soft_cross_entropy": 1.0879168816849036,
+      "validation": {
+        "soft_cross_entropy": 1.0776302735010783,
+        "accuracy": 0.5133333333333333,
+        "brier_soft": 0.19176260660092037,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0776302735010783,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 11,
+      "train_soft_cross_entropy": 1.0794475747037817,
+      "validation": {
+        "soft_cross_entropy": 1.0989113728205362,
+        "accuracy": 0.49666666666666665,
+        "brier_soft": 0.20668872609734534,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.0776302735010783,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 12,
+      "train_soft_cross_entropy": 1.0683085640271506,
+      "validation": {
+        "soft_cross_entropy": 1.0758668931325277,
+        "accuracy": 0.5016666666666667,
+        "brier_soft": 0.19317058285077413,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0758668931325277,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 13,
+      "train_soft_cross_entropy": 1.052984880871243,
+      "validation": {
+        "soft_cross_entropy": 1.0600487383206685,
+        "accuracy": 0.5316666666666666,
+        "brier_soft": 0.18257314254840215,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0600487383206685,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 14,
+      "train_soft_cross_entropy": 1.0401919462062694,
+      "validation": {
+        "soft_cross_entropy": 1.0525298221906025,
+        "accuracy": 0.5366666666666666,
+        "brier_soft": 0.1811353324353695,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0525298221906025,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 15,
+      "train_soft_cross_entropy": 1.0262903751267327,
+      "validation": {
+        "soft_cross_entropy": 1.0581993921597799,
+        "accuracy": 0.5233333333333333,
+        "brier_soft": 0.18550985043247542,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.0525298221906025,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 16,
+      "train_soft_cross_entropy": 1.0127459985238534,
+      "validation": {
+        "soft_cross_entropy": 1.0545752588907877,
+        "accuracy": 0.54,
+        "brier_soft": 0.18113683501879374,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 1.0525298221906025,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 17,
+      "train_soft_cross_entropy": 1.0096984642523306,
+      "validation": {
+        "soft_cross_entropy": 1.0468252456188203,
+        "accuracy": 0.5483333333333333,
+        "brier_soft": 0.17509076982736588,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0468252456188203,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 18,
+      "train_soft_cross_entropy": 1.0022709012914588,
+      "validation": {
+        "soft_cross_entropy": 1.0376164364814757,
+        "accuracy": 0.5766666666666667,
+        "brier_soft": 0.17226109830041728,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0376164364814757,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 19,
+      "train_soft_cross_entropy": 0.9934269437083492,
+      "validation": {
+        "soft_cross_entropy": 1.0526774628957112,
+        "accuracy": 0.5516666666666666,
+        "brier_soft": 0.17908830145994822,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.0376164364814757,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 20,
+      "train_soft_cross_entropy": 0.9915926403911025,
+      "validation": {
+        "soft_cross_entropy": 1.0292588464419048,
+        "accuracy": 0.57,
+        "brier_soft": 0.16918850486477216,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0292588464419048,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 21,
+      "train_soft_cross_entropy": 0.9833765982698511,
+      "validation": {
+        "soft_cross_entropy": 1.0493251442909242,
+        "accuracy": 0.555,
+        "brier_soft": 0.17917159788310527,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.0292588464419048,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 22,
+      "train_soft_cross_entropy": 0.9786927858988445,
+      "validation": {
+        "soft_cross_entropy": 1.0412022503217062,
+        "accuracy": 0.555,
+        "brier_soft": 0.17517984464764594,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 1.0292588464419048,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 23,
+      "train_soft_cross_entropy": 0.9695542351404826,
+      "validation": {
+        "soft_cross_entropy": 1.0376375365257262,
+        "accuracy": 0.5583333333333333,
+        "brier_soft": 0.17371407074232897,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 1.0292588464419048,
+        "improved": false
+      }
+    }
+  ],
+  "stopping": {
+    "reason": "early_stopping",
+    "epochs_completed": 23,
+    "patience": 3,
+    "min_epochs": 10,
+    "epochs_without_improvement": 3,
+    "metric": "validation.soft_cross_entropy",
+    "min_delta": 0.0
+  },
+  "trainable_names": [
+    "encoder.embeddings.interface.proj.weight",
+    "encoder.embeddings.interface.proj.bias"
+  ],
+  "frozen_parameters_unchanged": true
+}

evidence/seed-1-history.json

@@ -0,0 +1,314 @@
+{
+  "seed": 1,
+  "initial_validation": {
+    "soft_cross_entropy": 1.544869564374288,
+    "accuracy": 0.36833333333333335,
+    "brier_soft": 0.43051501592000324,
+    "decisions": 600
+  },
+  "epochs": [
+    {
+      "epoch": 1,
+      "train_soft_cross_entropy": 1.0002122403074194,
+      "validation": {
+        "soft_cross_entropy": 0.9555757478872935,
+        "accuracy": 0.665,
+        "brier_soft": 0.13395084381103517,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9555757478872935,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 2,
+      "train_soft_cross_entropy": 0.895550852616628,
+      "validation": {
+        "soft_cross_entropy": 0.8875669745604197,
+        "accuracy": 0.7216666666666667,
+        "brier_soft": 0.08987226173281669,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8875669745604197,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 3,
+      "train_soft_cross_entropy": 0.8399523215382187,
+      "validation": {
+        "soft_cross_entropy": 0.8776892550786336,
+        "accuracy": 0.725,
+        "brier_soft": 0.0829127719004949,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8776892550786336,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 4,
+      "train_soft_cross_entropy": 0.8150597869908368,
+      "validation": {
+        "soft_cross_entropy": 0.8523939752578735,
+        "accuracy": 0.7666666666666667,
+        "brier_soft": 0.07098765720923741,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8523939752578735,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 5,
+      "train_soft_cross_entropy": 0.7998251422246297,
+      "validation": {
+        "soft_cross_entropy": 0.853373521566391,
+        "accuracy": 0.745,
+        "brier_soft": 0.07141184127579132,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8523939752578735,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 6,
+      "train_soft_cross_entropy": 0.7924317064991704,
+      "validation": {
+        "soft_cross_entropy": 0.8612222145001094,
+        "accuracy": 0.7516666666666667,
+        "brier_soft": 0.07590873730679353,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8523939752578735,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 7,
+      "train_soft_cross_entropy": 0.7880261039733887,
+      "validation": {
+        "soft_cross_entropy": 0.8462035346031189,
+        "accuracy": 0.7466666666666667,
+        "brier_soft": 0.06817000946650903,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8462035346031189,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 8,
+      "train_soft_cross_entropy": 0.7816576555923179,
+      "validation": {
+        "soft_cross_entropy": 0.8440792632102966,
+        "accuracy": 0.7633333333333333,
+        "brier_soft": 0.06713677939027547,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8440792632102966,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.7783506816404837,
+      "validation": {
+        "soft_cross_entropy": 0.8438087296485901,
+        "accuracy": 0.7683333333333333,
+        "brier_soft": 0.06672837336858113,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8438087296485901,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.7754799175262451,
+      "validation": {
+        "soft_cross_entropy": 0.8468067542711893,
+        "accuracy": 0.755,
+        "brier_soft": 0.06876023932515334,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8438087296485901,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 11,
+      "train_soft_cross_entropy": 0.7734219932114637,
+      "validation": {
+        "soft_cross_entropy": 0.8521546524763107,
+        "accuracy": 0.7783333333333333,
+        "brier_soft": 0.07055049358556668,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8438087296485901,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 12,
+      "train_soft_cross_entropy": 0.7719193545094243,
+      "validation": {
+        "soft_cross_entropy": 0.842797059615453,
+        "accuracy": 0.7766666666666666,
+        "brier_soft": 0.06609537469533583,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.842797059615453,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 13,
+      "train_soft_cross_entropy": 0.7722371352601934,
+      "validation": {
+        "soft_cross_entropy": 0.848033101161321,
+        "accuracy": 0.7633333333333333,
+        "brier_soft": 0.06775808438037832,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.842797059615453,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 14,
+      "train_soft_cross_entropy": 0.7689926949695305,
+      "validation": {
+        "soft_cross_entropy": 0.8377540612220764,
+        "accuracy": 0.765,
+        "brier_soft": 0.06276483290052662,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8377540612220764,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 15,
+      "train_soft_cross_entropy": 0.7658979249000549,
+      "validation": {
+        "soft_cross_entropy": 0.8519912085930507,
+        "accuracy": 0.765,
+        "brier_soft": 0.07007166295622785,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8377540612220764,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 16,
+      "train_soft_cross_entropy": 0.7651108237107594,
+      "validation": {
+        "soft_cross_entropy": 0.8408213845888773,
+        "accuracy": 0.7633333333333333,
+        "brier_soft": 0.06544813718336324,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8377540612220764,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 17,
+      "train_soft_cross_entropy": 0.7817914198063038,
+      "validation": {
+        "soft_cross_entropy": 0.8468167595068614,
+        "accuracy": 0.7683333333333333,
+        "brier_soft": 0.06789324807934463,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8377540612220764,
+        "improved": false
+      }
+    }
+  ],
+  "stopping": {
+    "reason": "early_stopping",
+    "epochs_completed": 17,
+    "patience": 3,
+    "min_epochs": 10,
+    "epochs_without_improvement": 3,
+    "metric": "validation.soft_cross_entropy",
+    "min_delta": 0.0
+  },
+  "trainable_names": [
+    "encoder.embeddings.interface.proj.weight",
+    "encoder.embeddings.interface.proj.bias"
+  ],
+  "frozen_parameters_unchanged": true
+}

evidence/seed-2-history.json

@@ -0,0 +1,331 @@
+{
+  "seed": 2,
+  "initial_validation": {
+    "soft_cross_entropy": 1.544869564374288,
+    "accuracy": 0.36833333333333335,
+    "brier_soft": 0.43051501592000324,
+    "decisions": 600
+  },
+  "epochs": [
+    {
+      "epoch": 1,
+      "train_soft_cross_entropy": 1.1849676999339351,
+      "validation": {
+        "soft_cross_entropy": 1.1535689290364584,
+        "accuracy": 0.4266666666666667,
+        "brier_soft": 0.2362905572851499,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.1535689290364584,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 2,
+      "train_soft_cross_entropy": 1.1660638781830117,
+      "validation": {
+        "soft_cross_entropy": 1.1240008862813313,
+        "accuracy": 0.44666666666666666,
+        "brier_soft": 0.2163119477033615,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.1240008862813313,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 3,
+      "train_soft_cross_entropy": 1.1152002820262203,
+      "validation": {
+        "soft_cross_entropy": 1.111793461640676,
+        "accuracy": 0.4816666666666667,
+        "brier_soft": 0.21397013902664186,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.111793461640676,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 4,
+      "train_soft_cross_entropy": 1.0905909954177009,
+      "validation": {
+        "soft_cross_entropy": 1.0822320612271628,
+        "accuracy": 0.49833333333333335,
+        "brier_soft": 0.19861485362052916,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0822320612271628,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 5,
+      "train_soft_cross_entropy": 1.0488388820047732,
+      "validation": {
+        "soft_cross_entropy": 1.0466791025797526,
+        "accuracy": 0.5383333333333333,
+        "brier_soft": 0.18209601615866025,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0466791025797526,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 6,
+      "train_soft_cross_entropy": 1.0179926924352292,
+      "validation": {
+        "soft_cross_entropy": 1.0302869494756062,
+        "accuracy": 0.5633333333333334,
+        "brier_soft": 0.1717192947367827,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0302869494756062,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 7,
+      "train_soft_cross_entropy": 1.0011521806540313,
+      "validation": {
+        "soft_cross_entropy": 1.0306965279579163,
+        "accuracy": 0.5533333333333333,
+        "brier_soft": 0.17337830337385338,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.0302869494756062,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 8,
+      "train_soft_cross_entropy": 0.9898686430189345,
+      "validation": {
+        "soft_cross_entropy": 1.0352703229586284,
+        "accuracy": 0.5533333333333333,
+        "brier_soft": 0.17590213686227799,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 1.0302869494756062,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.9732429999775357,
+      "validation": {
+        "soft_cross_entropy": 1.0375931040445963,
+        "accuracy": 0.56,
+        "brier_soft": 0.1798627228786548,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 1.0302869494756062,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.9616305458987201,
+      "validation": {
+        "soft_cross_entropy": 1.0139233843485513,
+        "accuracy": 0.6116666666666667,
+        "brier_soft": 0.16307140870640674,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0139233843485513,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 11,
+      "train_soft_cross_entropy": 0.9440543747831274,
+      "validation": {
+        "soft_cross_entropy": 1.0138032054901123,
+        "accuracy": 0.6166666666666667,
+        "brier_soft": 0.165894419302543,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0138032054901123,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 12,
+      "train_soft_cross_entropy": 0.9309376364284091,
+      "validation": {
+        "soft_cross_entropy": 0.9929217569033305,
+        "accuracy": 0.6266666666666667,
+        "brier_soft": 0.15395031906664372,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9929217569033305,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 13,
+      "train_soft_cross_entropy": 0.9144250082969666,
+      "validation": {
+        "soft_cross_entropy": 0.9964303588867187,
+        "accuracy": 0.6266666666666667,
+        "brier_soft": 0.1560174826408426,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9929217569033305,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 14,
+      "train_soft_cross_entropy": 0.9063218560925237,
+      "validation": {
+        "soft_cross_entropy": 1.005851571559906,
+        "accuracy": 0.6133333333333333,
+        "brier_soft": 0.1608497215807438,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.9929217569033305,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 15,
+      "train_soft_cross_entropy": 0.9014872776137458,
+      "validation": {
+        "soft_cross_entropy": 0.9867499772707621,
+        "accuracy": 0.6316666666666667,
+        "brier_soft": 0.14927835414807003,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9867499772707621,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 16,
+      "train_soft_cross_entropy": 0.8920391595805133,
+      "validation": {
+        "soft_cross_entropy": 0.9926939264933268,
+        "accuracy": 0.6283333333333333,
+        "brier_soft": 0.15533705055713654,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9867499772707621,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 17,
+      "train_soft_cross_entropy": 0.8795346831833875,
+      "validation": {
+        "soft_cross_entropy": 1.02633438428243,
+        "accuracy": 0.6583333333333333,
+        "brier_soft": 0.16615503872434298,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.9867499772707621,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 18,
+      "train_soft_cross_entropy": 0.879185232453876,
+      "validation": {
+        "soft_cross_entropy": 1.0007199597358705,
+        "accuracy": 0.65,
+        "brier_soft": 0.15312831775595745,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 0.9867499772707621,
+        "improved": false
+      }
+    }
+  ],
+  "stopping": {
+    "reason": "early_stopping",
+    "epochs_completed": 18,
+    "patience": 3,
+    "min_epochs": 10,
+    "epochs_without_improvement": 3,
+    "metric": "validation.soft_cross_entropy",
+    "min_delta": 0.0
+  },
+  "trainable_names": [
+    "encoder.embeddings.interface.proj.weight",
+    "encoder.embeddings.interface.proj.bias"
+  ],
+  "frozen_parameters_unchanged": true
+}

evidence/seed-3-history.json

@@ -0,0 +1,484 @@
+{
+  "seed": 3,
+  "initial_validation": {
+    "soft_cross_entropy": 1.544869564374288,
+    "accuracy": 0.36833333333333335,
+    "brier_soft": 0.43051501592000324,
+    "decisions": 600
+  },
+  "epochs": [
+    {
+      "epoch": 1,
+      "train_soft_cross_entropy": 1.1980749452555621,
+      "validation": {
+        "soft_cross_entropy": 1.16634921391805,
+        "accuracy": 0.4266666666666667,
+        "brier_soft": 0.2427163557211558,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.16634921391805,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 2,
+      "train_soft_cross_entropy": 1.1646321240177862,
+      "validation": {
+        "soft_cross_entropy": 1.1217502697308859,
+        "accuracy": 0.4533333333333333,
+        "brier_soft": 0.2186164912581444,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.1217502697308859,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 3,
+      "train_soft_cross_entropy": 1.1056555556367944,
+      "validation": {
+        "soft_cross_entropy": 1.093972578048706,
+        "accuracy": 0.505,
+        "brier_soft": 0.199787005285422,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.093972578048706,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 4,
+      "train_soft_cross_entropy": 1.0671771099832323,
+      "validation": {
+        "soft_cross_entropy": 1.0549713468551636,
+        "accuracy": 0.5516666666666666,
+        "brier_soft": 0.18068428387244542,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0549713468551636,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 5,
+      "train_soft_cross_entropy": 1.0494003568755257,
+      "validation": {
+        "soft_cross_entropy": 1.0567963059743246,
+        "accuracy": 0.54,
+        "brier_soft": 0.18257476242880027,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.0549713468551636,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 6,
+      "train_soft_cross_entropy": 1.038703513851872,
+      "validation": {
+        "soft_cross_entropy": 1.0705021890004476,
+        "accuracy": 0.525,
+        "brier_soft": 0.19120527582863966,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 1.0549713468551636,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 7,
+      "train_soft_cross_entropy": 1.0313762316880404,
+      "validation": {
+        "soft_cross_entropy": 1.0447989773750306,
+        "accuracy": 0.5466666666666666,
+        "brier_soft": 0.17501757830381393,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0447989773750306,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 8,
+      "train_soft_cross_entropy": 1.017818791601393,
+      "validation": {
+        "soft_cross_entropy": 1.024161856174469,
+        "accuracy": 0.5816666666666667,
+        "brier_soft": 0.1630909529576699,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.024161856174469,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.9899537578335514,
+      "validation": {
+        "soft_cross_entropy": 1.0265704361597696,
+        "accuracy": 0.6083333333333333,
+        "brier_soft": 0.16808087141563496,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 1.024161856174469,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.9632394627288535,
+      "validation": {
+        "soft_cross_entropy": 1.0183856010437011,
+        "accuracy": 0.5983333333333334,
+        "brier_soft": 0.1608257148663203,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0183856010437011,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 11,
+      "train_soft_cross_entropy": 0.9342622081438701,
+      "validation": {
+        "soft_cross_entropy": 1.0024159566561381,
+        "accuracy": 0.6333333333333333,
+        "brier_soft": 0.14996731283764045,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0024159566561381,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 12,
+      "train_soft_cross_entropy": 0.9137524432606168,
+      "validation": {
+        "soft_cross_entropy": 0.9873138956228892,
+        "accuracy": 0.6516666666666666,
+        "brier_soft": 0.14277802929282188,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9873138956228892,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 13,
+      "train_soft_cross_entropy": 0.8881368139496556,
+      "validation": {
+        "soft_cross_entropy": 0.9626709421475729,
+        "accuracy": 0.6716666666666666,
+        "brier_soft": 0.12283751085400581,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9626709421475729,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 14,
+      "train_soft_cross_entropy": 0.865500467883216,
+      "validation": {
+        "soft_cross_entropy": 0.9580995849768321,
+        "accuracy": 0.685,
+        "brier_soft": 0.12383397127191226,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9580995849768321,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 15,
+      "train_soft_cross_entropy": 0.8549449095461104,
+      "validation": {
+        "soft_cross_entropy": 0.9462178750832876,
+        "accuracy": 0.6916666666666667,
+        "brier_soft": 0.1187205430244406,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9462178750832876,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 16,
+      "train_soft_cross_entropy": 0.8394836834183446,
+      "validation": {
+        "soft_cross_entropy": 0.9463119049866994,
+        "accuracy": 0.6933333333333334,
+        "brier_soft": 0.11875337022046248,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9462178750832876,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 17,
+      "train_soft_cross_entropy": 0.8286281617040987,
+      "validation": {
+        "soft_cross_entropy": 0.941375896135966,
+        "accuracy": 0.685,
+        "brier_soft": 0.11455494280904531,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.941375896135966,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 18,
+      "train_soft_cross_entropy": 0.8222652976601212,
+      "validation": {
+        "soft_cross_entropy": 0.9331588689486185,
+        "accuracy": 0.6866666666666666,
+        "brier_soft": 0.11237604923546314,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9331588689486185,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 19,
+      "train_soft_cross_entropy": 0.8160834203826056,
+      "validation": {
+        "soft_cross_entropy": 0.9387085942427317,
+        "accuracy": 0.6916666666666667,
+        "brier_soft": 0.11558652246991793,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9331588689486185,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 20,
+      "train_soft_cross_entropy": 0.8087538785846146,
+      "validation": {
+        "soft_cross_entropy": 0.9408066284656524,
+        "accuracy": 0.6783333333333333,
+        "brier_soft": 0.11930533437679211,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.9331588689486185,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 21,
+      "train_soft_cross_entropy": 0.8048860243956248,
+      "validation": {
+        "soft_cross_entropy": 0.9236933688322703,
+        "accuracy": 0.7016666666666667,
+        "brier_soft": 0.10792215374608835,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9236933688322703,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 22,
+      "train_soft_cross_entropy": 0.8001213801790167,
+      "validation": {
+        "soft_cross_entropy": 0.9347231006622314,
+        "accuracy": 0.6983333333333334,
+        "brier_soft": 0.11313647958139579,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9236933688322703,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 23,
+      "train_soft_cross_entropy": 0.7940637088263476,
+      "validation": {
+        "soft_cross_entropy": 0.9299098292986552,
+        "accuracy": 0.7033333333333334,
+        "brier_soft": 0.11475709093113741,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.9236933688322703,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 24,
+      "train_soft_cross_entropy": 0.7884197377717054,
+      "validation": {
+        "soft_cross_entropy": 0.9105557099978129,
+        "accuracy": 0.715,
+        "brier_soft": 0.1013037375236551,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9105557099978129,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 25,
+      "train_soft_cross_entropy": 0.784954049454795,
+      "validation": {
+        "soft_cross_entropy": 0.923290346066157,
+        "accuracy": 0.71,
+        "brier_soft": 0.10530036373684803,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9105557099978129,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 26,
+      "train_soft_cross_entropy": 0.7828840752442677,
+      "validation": {
+        "soft_cross_entropy": 0.9299345835049947,
+        "accuracy": 0.7066666666666667,
+        "brier_soft": 0.11173659774164359,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.9105557099978129,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 27,
+      "train_soft_cross_entropy": 0.7790651953661883,
+      "validation": {
+        "soft_cross_entropy": 0.9137676254908244,
+        "accuracy": 0.7216666666666667,
+        "brier_soft": 0.10172271362195412,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 0.9105557099978129,
+        "improved": false
+      }
+    }
+  ],
+  "stopping": {
+    "reason": "early_stopping",
+    "epochs_completed": 27,
+    "patience": 3,
+    "min_epochs": 10,
+    "epochs_without_improvement": 3,
+    "metric": "validation.soft_cross_entropy",
+    "min_delta": 0.0
+  },
+  "trainable_names": [
+    "encoder.embeddings.interface.proj.weight",
+    "encoder.embeddings.interface.proj.bias"
+  ],
+  "frozen_parameters_unchanged": true
+}

evidence/seed-4-history.json

@@ -0,0 +1,195 @@
+{
+  "seed": 4,
+  "initial_validation": {
+    "soft_cross_entropy": 1.544869564374288,
+    "accuracy": 0.36833333333333335,
+    "brier_soft": 0.43051501592000324,
+    "decisions": 600
+  },
+  "epochs": [
+    {
+      "epoch": 1,
+      "train_soft_cross_entropy": 1.002756692568461,
+      "validation": {
+        "soft_cross_entropy": 0.9602992224693299,
+        "accuracy": 0.6783333333333333,
+        "brier_soft": 0.1296700432151556,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9602992224693299,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 2,
+      "train_soft_cross_entropy": 0.8854288237624698,
+      "validation": {
+        "soft_cross_entropy": 0.8732180511951446,
+        "accuracy": 0.7316666666666667,
+        "brier_soft": 0.08001670623819034,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8732180511951446,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 3,
+      "train_soft_cross_entropy": 0.8375967553809837,
+      "validation": {
+        "soft_cross_entropy": 0.8618760740756989,
+        "accuracy": 0.7566666666666667,
+        "brier_soft": 0.07448753335823616,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8618760740756989,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 4,
+      "train_soft_cross_entropy": 0.8120107175244226,
+      "validation": {
+        "soft_cross_entropy": 0.8502214312553406,
+        "accuracy": 0.7683333333333333,
+        "brier_soft": 0.06676056392490864,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8502214312553406,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 5,
+      "train_soft_cross_entropy": 0.7984476970301734,
+      "validation": {
+        "soft_cross_entropy": 0.8527251331011454,
+        "accuracy": 0.765,
+        "brier_soft": 0.07089023986210426,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8502214312553406,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 6,
+      "train_soft_cross_entropy": 0.7902782445042221,
+      "validation": {
+        "soft_cross_entropy": 0.8496846203009287,
+        "accuracy": 0.765,
+        "brier_soft": 0.06887457605761786,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8496846203009287,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 7,
+      "train_soft_cross_entropy": 0.7847388537283297,
+      "validation": {
+        "soft_cross_entropy": 0.8448993345101674,
+        "accuracy": 0.7533333333333333,
+        "brier_soft": 0.0668264798882107,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8448993345101674,
+        "improved": true
+      }
+    },
+    {
+      "epoch": 8,
+      "train_soft_cross_entropy": 0.780297756018462,
+      "validation": {
+        "soft_cross_entropy": 0.8471182523171107,
+        "accuracy": 0.7683333333333333,
+        "brier_soft": 0.06819427679603299,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8448993345101674,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.7801684511590887,
+      "validation": {
+        "soft_cross_entropy": 0.8466147363185883,
+        "accuracy": 0.775,
+        "brier_soft": 0.06677873468647401,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8448993345101674,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.7775638208565888,
+      "validation": {
+        "soft_cross_entropy": 0.8455785755316416,
+        "accuracy": 0.78,
+        "brier_soft": 0.06686171248555184,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8448993345101674,
+        "improved": false
+      }
+    }
+  ],
+  "stopping": {
+    "reason": "early_stopping",
+    "epochs_completed": 10,
+    "patience": 3,
+    "min_epochs": 10,
+    "epochs_without_improvement": 3,
+    "metric": "validation.soft_cross_entropy",
+    "min_delta": 0.0
+  },
+  "trainable_names": [
+    "encoder.embeddings.interface.proj.weight",
+    "encoder.embeddings.interface.proj.bias"
+  ],
+  "frozen_parameters_unchanged": true
+}

export-verification.json

@@ -0,0 +1,42 @@
+{
+  "passed": true,
+  "candidate_seed": 1,
+  "selected_epoch": 14,
+  "cases": 3516,
+  "decisions": 5116,
+  "benchmark_cases_sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
+  "checkpoint_sha256": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
+  "all_answer_objects_exact": true,
+  "all_suite_metrics_exact": true,
+  "bundled_code_imported_from_outside_workspace": true,
+  "local_model_card_metadata_valid": true,
+  "device": "cuda:1",
+  "model": {
+    "gpu_name": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+    "dtype": "torch.bfloat16",
+    "torch_version": "2.13.0+cu130",
+    "reproducibility": {
+      "deterministic_algorithms": true,
+      "warn_only": false,
+      "cublas_workspace_config": ":4096:8",
+      "python_hash_seed": "0",
+      "cudnn_benchmark": false,
+      "cudnn_deterministic": true,
+      "cudnn_allow_tf32": false,
+      "float32_matmul_precision": "highest",
+      "cudnn_sdp_enabled": false,
+      "flash_sdp_enabled": true,
+      "mem_efficient_sdp_enabled": true,
+      "math_sdp_enabled": true,
+      "cuda_version": "13.0",
+      "cudnn_version": 92000,
+      "scope": "fixed hardware/runtime; cross-platform bitwise agreement is not promised"
+    },
+    "portable_adapter": {
+      "base_model": "convaiinnovations/laya",
+      "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+      "seed": 1,
+      "selected_epoch": 14
+    }
+  }
+}

licenses/jev-benchmarks-LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

licenses/laya-LICENSE

@@ -0,0 +1,176 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS

load_adapter.py

@@ -0,0 +1,14 @@
+"""Local entry point for the bundled, pinned Laya input adapter."""
+
+from pathlib import Path
+import sys
+
+PACKAGE = Path(__file__).resolve().parent
+if str(PACKAGE) not in sys.path:
+    sys.path.insert(0, str(PACKAGE))
+
+from ariadne_bench.portable import load_release  # noqa: E402
+
+
+def load(directory=PACKAGE, **options):
+    return load_release(directory, **options)

metrics.json

@@ -0,0 +1,984 @@
+{
+  "baseline": {
+    "typed-decisions": 0.3625,
+    "ag-news": 0.9483333333333334,
+    "boolq": 0.83,
+    "emotion": 0.5733333333333334,
+    "prompt-injections": 0.6896551724137931,
+    "sst5": 0.37,
+    "massive-intent.en": 0.7866666666666666,
+    "xnli.en": 0.86
+  },
+  "runs": {
+    "0": {
+      "seed": 0,
+      "selected_epoch": 20,
+      "elapsed_s": 1038.9482859019772,
+      "initial_validation": {
+        "soft_cross_entropy": 1.544869564374288,
+        "accuracy": 0.36833333333333335,
+        "brier_soft": 0.43051501592000324,
+        "decisions": 600
+      },
+      "selected_validation": {
+        "soft_cross_entropy": 1.0292588464419048,
+        "accuracy": 0.57,
+        "brier_soft": 0.16918850486477216,
+        "decisions": 600
+      },
+      "stopping": {
+        "reason": "early_stopping",
+        "epochs_completed": 23,
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "metric": "validation.soft_cross_entropy",
+        "min_delta": 0.0
+      },
+      "frozen_parameters_unchanged": true,
+      "suites": {
+        "typed-decisions": {
+          "attempted": 2000,
+          "valid": 2000,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.61,
+          "accuracy_valid": 0.61,
+          "ece_top_label": 0.15303136435282014,
+          "mean_confidence": 0.4574502381919959,
+          "brier_hard": 0.549798004153015,
+          "brier_hard_n": 2000,
+          "nll_hard": 0.9521041848093966,
+          "nll_hard_n": 2000,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.3883819759829014,
+          "soft_accuracy_n": 2000,
+          "brier_soft": 0.15223049605383382,
+          "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.2712886346630264,
+          "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.2797391333751494,
+          "total_variation_n": 2000,
+          "score_mae": 0.45095113683604626,
+          "score_mae_n": 800,
+          "within_one_level": 0.915,
+          "within_one_level_n": 800,
+          "macro_f1": 0.4150680787086972
+        },
+        "ag-news": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.31333333333333335,
+          "accuracy_valid": 0.31333333333333335,
+          "ece_top_label": 0.025934925132124264,
+          "mean_confidence": 0.28915707486787573,
+          "brier_hard": 0.7441339007069468,
+          "brier_hard_n": 600,
+          "nll_hard": 1.3750333163467814,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.29084523228646375
+        },
+        "emotion": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.195,
+          "accuracy_valid": 0.195,
+          "ece_top_label": 0.08537782470783518,
+          "mean_confidence": 0.2764666886168944,
+          "brier_hard": 0.8415061960524678,
+          "brier_hard_n": 600,
+          "nll_hard": 1.810555559185162,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.12883846057517262
+        },
+        "boolq": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.5666666666666667,
+          "accuracy_valid": 0.5666666666666667,
+          "ece_top_label": 0.08635800000000002,
+          "mean_confidence": 0.5258943333333334,
+          "brier_hard": 0.5020554209333333,
+          "brier_hard_n": 600,
+          "nll_hard": 0.6953013099942155,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.5074762578298646
+        },
+        "sst5": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.15,
+          "accuracy_valid": 0.15,
+          "ece_top_label": 0.11939667407180776,
+          "mean_confidence": 0.2693966740718078,
+          "brier_hard": 0.8207044411796734,
+          "brier_hard_n": 600,
+          "nll_hard": 1.6589117409852412,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "score_mae": 1.2247300886774997,
+          "score_mae_n": 600,
+          "within_one_level": 0.42,
+          "within_one_level_n": 600,
+          "macro_f1": 0.10629896865332711
+        },
+        "prompt-injections": {
+          "attempted": 116,
+          "valid": 116,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.5344827586206896,
+          "accuracy_valid": 0.5344827586206896,
+          "ece_top_label": 0.02089741379310345,
+          "mean_confidence": 0.5135853448275862,
+          "brier_hard": 0.5003080198275862,
+          "brier_hard_n": 116,
+          "nll_hard": 0.6934547870034389,
+          "nll_hard_n": 116,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 116,
+          "macro_f1": 0.513664596273292
+        },
+        "massive-intent.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.023333333333333334,
+          "accuracy_valid": 0.023333333333333334,
+          "ece_top_label": 0.09797495336730408,
+          "mean_confidence": 0.12130828670063742,
+          "brier_hard": 0.9672041702729498,
+          "brier_hard_n": 300,
+          "nll_hard": 3.1313647377288074,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.011215849106652133
+        },
+        "xnli.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.3433333333333333,
+          "accuracy_valid": 0.3433333333333333,
+          "ece_top_label": 0.05212130996336308,
+          "mean_confidence": 0.39545464329669644,
+          "brier_hard": 0.6747449551934047,
+          "brier_hard_n": 300,
+          "nll_hard": 1.1111148411541347,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.24805362074756226
+        }
+      }
+    },
+    "1": {
+      "seed": 1,
+      "selected_epoch": 14,
+      "elapsed_s": 766.9112328969641,
+      "initial_validation": {
+        "soft_cross_entropy": 1.544869564374288,
+        "accuracy": 0.36833333333333335,
+        "brier_soft": 0.43051501592000324,
+        "decisions": 600
+      },
+      "selected_validation": {
+        "soft_cross_entropy": 0.8377540612220764,
+        "accuracy": 0.765,
+        "brier_soft": 0.06276483290052662,
+        "decisions": 600
+      },
+      "stopping": {
+        "reason": "early_stopping",
+        "epochs_completed": 17,
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "metric": "validation.soft_cross_entropy",
+        "min_delta": 0.0
+      },
+      "frozen_parameters_unchanged": true,
+      "suites": {
+        "typed-decisions": {
+          "attempted": 2000,
+          "valid": 2000,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7695,
+          "accuracy_valid": 0.7695,
+          "ece_top_label": 0.2142237525419539,
+          "mean_confidence": 0.555276247458046,
+          "brier_hard": 0.3977842163576973,
+          "brier_hard_n": 2000,
+          "nll_hard": 0.7027635604099817,
+          "nll_hard_n": 2000,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.47103605907759827,
+          "soft_accuracy_n": 2000,
+          "brier_soft": 0.06255403710877039,
+          "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.11643450461088396,
+          "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.17146599324503523,
+          "total_variation_n": 2000,
+          "score_mae": 0.2299944493828981,
+          "score_mae_n": 800,
+          "within_one_level": 0.98875,
+          "within_one_level_n": 800,
+          "macro_f1": 0.6487623885802034
+        },
+        "ag-news": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.9316666666666666,
+          "accuracy_valid": 0.9316666666666666,
+          "ece_top_label": 0.03387674147756575,
+          "mean_confidence": 0.904708562141314,
+          "brier_hard": 0.10911433540423408,
+          "brier_hard_n": 600,
+          "nll_hard": 0.21361121678114303,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.9283987145646934
+        },
+        "emotion": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.5733333333333334,
+          "accuracy_valid": 0.5733333333333334,
+          "ece_top_label": 0.3169210441851814,
+          "mean_confidence": 0.8880587177380197,
+          "brier_hard": 0.7264584486728032,
+          "brier_hard_n": 600,
+          "nll_hard": 2.146154603203373,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.008333333333333333,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.4862286665693279
+        },
+        "boolq": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7983333333333333,
+          "accuracy_valid": 0.7983333333333333,
+          "ece_top_label": 0.09755433333333334,
+          "mean_confidence": 0.8900093333333333,
+          "brier_hard": 0.29822958146666667,
+          "brier_hard_n": 600,
+          "nll_hard": 0.4711253960780479,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.7813983878883868
+        },
+        "sst5": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.42,
+          "accuracy_valid": 0.42,
+          "ece_top_label": 0.1643168667044709,
+          "mean_confidence": 0.5761336745094923,
+          "brier_hard": 0.722245716690302,
+          "brier_hard_n": 600,
+          "nll_hard": 1.412054753482227,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "score_mae": 0.7360310433394476,
+          "score_mae_n": 600,
+          "within_one_level": 0.7466666666666667,
+          "within_one_level_n": 600,
+          "macro_f1": 0.41387920942262574
+        },
+        "prompt-injections": {
+          "attempted": 116,
+          "valid": 116,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7155172413793104,
+          "accuracy_valid": 0.7155172413793104,
+          "ece_top_label": 0.20010517241379305,
+          "mean_confidence": 0.9105,
+          "brier_hard": 0.43610344172413795,
+          "brier_hard_n": 116,
+          "nll_hard": 1.2355666268201304,
+          "nll_hard_n": 116,
+          "zero_probability_gold": 0.008620689655172414,
+          "zero_probability_gold_n": 116,
+          "macro_f1": 0.7038756091900673
+        },
+        "massive-intent.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7433333333333333,
+          "accuracy_valid": 0.7433333333333333,
+          "ece_top_label": 0.1906540683935104,
+          "mean_confidence": 0.9339874017268438,
+          "brier_hard": 0.43706654758878133,
+          "brier_hard_n": 300,
+          "nll_hard": 2.7801978001907903,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.07,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.44346732036749054
+        },
+        "xnli.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.8766666666666667,
+          "accuracy_valid": 0.8766666666666667,
+          "ece_top_label": 0.055158283453726184,
+          "mean_confidence": 0.913468962679153,
+          "brier_hard": 0.19180470102420566,
+          "brier_hard_n": 300,
+          "nll_hard": 0.3708980570966955,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.8772028178860477
+        }
+      }
+    },
+    "2": {
+      "seed": 2,
+      "selected_epoch": 15,
+      "elapsed_s": 813.5451362769818,
+      "initial_validation": {
+        "soft_cross_entropy": 1.544869564374288,
+        "accuracy": 0.36833333333333335,
+        "brier_soft": 0.43051501592000324,
+        "decisions": 600
+      },
+      "selected_validation": {
+        "soft_cross_entropy": 0.9867499772707621,
+        "accuracy": 0.6316666666666667,
+        "brier_soft": 0.14927835414807003,
+        "decisions": 600
+      },
+      "stopping": {
+        "reason": "early_stopping",
+        "epochs_completed": 18,
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "metric": "validation.soft_cross_entropy",
+        "min_delta": 0.0
+      },
+      "frozen_parameters_unchanged": true,
+      "suites": {
+        "typed-decisions": {
+          "attempted": 2000,
+          "valid": 2000,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.661,
+          "accuracy_valid": 0.661,
+          "ece_top_label": 0.17250655060654457,
+          "mean_confidence": 0.48883668313682976,
+          "brier_hard": 0.5040496940863629,
+          "brier_hard_n": 2000,
+          "nll_hard": 0.8694405673021295,
+          "nll_hard_n": 2000,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.41427322765683283,
+          "soft_accuracy_n": 2000,
+          "brier_soft": 0.12329331823446005,
+          "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.2167589625120186,
+          "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.24557641844905673,
+          "total_variation_n": 2000,
+          "score_mae": 0.38257377026959916,
+          "score_mae_n": 800,
+          "within_one_level": 0.92625,
+          "within_one_level_n": 800,
+          "macro_f1": 0.511489281675723
+        },
+        "ag-news": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.2633333333333333,
+          "accuracy_valid": 0.2633333333333333,
+          "ece_top_label": 0.0415880758215024,
+          "mean_confidence": 0.30492140915483573,
+          "brier_hard": 0.7533353201524136,
+          "brier_hard_n": 600,
+          "nll_hard": 1.392051963909145,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.2196261943166739
+        },
+        "emotion": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.04833333333333333,
+          "accuracy_valid": 0.04833333333333333,
+          "ece_top_label": 0.21796334391120661,
+          "mean_confidence": 0.2662966772445399,
+          "brier_hard": 0.9083949711964834,
+          "brier_hard_n": 600,
+          "nll_hard": 2.0132867760168445,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.03431938431938432
+        },
+        "boolq": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.49833333333333335,
+          "accuracy_valid": 0.49833333333333335,
+          "ece_top_label": 0.03133150000000007,
+          "mean_confidence": 0.5290545,
+          "brier_hard": 0.5040687543666666,
+          "brier_hard_n": 600,
+          "nll_hard": 0.6972355680599187,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.49597983919356775
+        },
+        "sst5": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.22666666666666666,
+          "accuracy_valid": 0.22666666666666666,
+          "ece_top_label": 0.08382145968032653,
+          "mean_confidence": 0.30788872429147607,
+          "brier_hard": 0.8111226172703578,
+          "brier_hard_n": 600,
+          "nll_hard": 1.6364255257258482,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "score_mae": 1.223873285603664,
+          "score_mae_n": 600,
+          "within_one_level": 0.39,
+          "within_one_level_n": 600,
+          "macro_f1": 0.12552908285983264
+        },
+        "prompt-injections": {
+          "attempted": 116,
+          "valid": 116,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.3448275862068966,
+          "accuracy_valid": 0.3448275862068966,
+          "ece_top_label": 0.1914112068965517,
+          "mean_confidence": 0.5362387931034482,
+          "brier_hard": 0.5323824856896552,
+          "brier_hard_n": 116,
+          "nll_hard": 0.7258214322004152,
+          "nll_hard_n": 116,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 116,
+          "macro_f1": 0.275
+        },
+        "massive-intent.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.06333333333333334,
+          "accuracy_valid": 0.06333333333333334,
+          "ece_top_label": 0.03411377590323972,
+          "mean_confidence": 0.09744710923657304,
+          "brier_hard": 0.9608971275390831,
+          "brier_hard_n": 300,
+          "nll_hard": 3.135262799945936,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.02907709642455108
+        },
+        "xnli.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.3433333333333333,
+          "accuracy_valid": 0.3433333333333333,
+          "ece_top_label": 0.04294870734165375,
+          "mean_confidence": 0.38628204067498706,
+          "brier_hard": 0.6761361630010466,
+          "brier_hard_n": 300,
+          "nll_hard": 1.1128104653939794,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.3099922839506173
+        }
+      }
+    },
+    "3": {
+      "seed": 3,
+      "selected_epoch": 24,
+      "elapsed_s": 1226.4128511130111,
+      "initial_validation": {
+        "soft_cross_entropy": 1.544869564374288,
+        "accuracy": 0.36833333333333335,
+        "brier_soft": 0.43051501592000324,
+        "decisions": 600
+      },
+      "selected_validation": {
+        "soft_cross_entropy": 0.9105557099978129,
+        "accuracy": 0.715,
+        "brier_soft": 0.1013037375236551,
+        "decisions": 600
+      },
+      "stopping": {
+        "reason": "early_stopping",
+        "epochs_completed": 27,
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "metric": "validation.soft_cross_entropy",
+        "min_delta": 0.0
+      },
+      "frozen_parameters_unchanged": true,
+      "suites": {
+        "typed-decisions": {
+          "attempted": 2000,
+          "valid": 2000,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7145,
+          "accuracy_valid": 0.7145,
+          "ece_top_label": 0.18500231281217855,
+          "mean_confidence": 0.5300184130804106,
+          "brier_hard": 0.45082257101028994,
+          "brier_hard_n": 2000,
+          "nll_hard": 0.7891005479512444,
+          "nll_hard_n": 2000,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.4447570518885741,
+          "soft_accuracy_n": 2000,
+          "brier_soft": 0.09325671264191153,
+          "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.17157487793178008,
+          "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.21097575386758916,
+          "total_variation_n": 2000,
+          "score_mae": 0.32394138130337413,
+          "score_mae_n": 800,
+          "within_one_level": 0.96375,
+          "within_one_level_n": 800,
+          "macro_f1": 0.5630278027844289
+        },
+        "ag-news": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.285,
+          "accuracy_valid": 0.285,
+          "ece_top_label": 0.03904648151058483,
+          "mean_confidence": 0.3240464815105848,
+          "brier_hard": 0.7625373960165811,
+          "brier_hard_n": 600,
+          "nll_hard": 1.4140716749662658,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.25772655273033623
+        },
+        "emotion": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.135,
+          "accuracy_valid": 0.135,
+          "ece_top_label": 0.31863940813484304,
+          "mean_confidence": 0.45363940813484305,
+          "brier_hard": 0.9777635802768799,
+          "brier_hard_n": 600,
+          "nll_hard": 2.137352366621794,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.06590661903472231
+        },
+        "boolq": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.48833333333333334,
+          "accuracy_valid": 0.48833333333333334,
+          "ece_top_label": 0.10787050000000001,
+          "mean_confidence": 0.5912141666666667,
+          "brier_hard": 0.5250116893,
+          "brier_hard_n": 600,
+          "nll_hard": 0.719837552287629,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.48696381173075903
+        },
+        "sst5": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.25666666666666665,
+          "accuracy_valid": 0.25666666666666665,
+          "ece_top_label": 0.05066152500332892,
+          "mean_confidence": 0.3073281916699956,
+          "brier_hard": 0.8215936069738192,
+          "brier_hard_n": 600,
+          "nll_hard": 1.6775697808435297,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "score_mae": 1.245226132804566,
+          "score_mae_n": 600,
+          "within_one_level": 0.44333333333333336,
+          "within_one_level_n": 600,
+          "macro_f1": 0.13260195835186606
+        },
+        "prompt-injections": {
+          "attempted": 116,
+          "valid": 116,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.5689655172413793,
+          "accuracy_valid": 0.5689655172413793,
+          "ece_top_label": 0.07609396551724136,
+          "mean_confidence": 0.6450594827586207,
+          "brier_hard": 0.4881634129310345,
+          "brier_hard_n": 116,
+          "nll_hard": 0.6835747994579267,
+          "nll_hard_n": 116,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 116,
+          "macro_f1": 0.5461658841940532
+        },
+        "massive-intent.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.023333333333333334,
+          "accuracy_valid": 0.023333333333333334,
+          "ece_top_label": 0.14827139054632427,
+          "mean_confidence": 0.1716047238796576,
+          "brier_hard": 1.012268483556247,
+          "brier_hard_n": 300,
+          "nll_hard": 3.540188562309456,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.014601824457593688
+        },
+        "xnli.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.33666666666666667,
+          "accuracy_valid": 0.33666666666666667,
+          "ece_top_label": 0.21980800490067007,
+          "mean_confidence": 0.5564746715673368,
+          "brier_hard": 0.7709342547858443,
+          "brier_hard_n": 300,
+          "nll_hard": 1.2650663651832452,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.20238216957239646
+        }
+      }
+    },
+    "4": {
+      "seed": 4,
+      "selected_epoch": 7,
+      "elapsed_s": 452.7137592760264,
+      "initial_validation": {
+        "soft_cross_entropy": 1.544869564374288,
+        "accuracy": 0.36833333333333335,
+        "brier_soft": 0.43051501592000324,
+        "decisions": 600
+      },
+      "selected_validation": {
+        "soft_cross_entropy": 0.8448993345101674,
+        "accuracy": 0.7533333333333333,
+        "brier_soft": 0.0668264798882107,
+        "decisions": 600
+      },
+      "stopping": {
+        "reason": "early_stopping",
+        "epochs_completed": 10,
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "metric": "validation.soft_cross_entropy",
+        "min_delta": 0.0
+      },
+      "frozen_parameters_unchanged": true,
+      "suites": {
+        "typed-decisions": {
+          "attempted": 2000,
+          "valid": 2000,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7565,
+          "accuracy_valid": 0.7565,
+          "ece_top_label": 0.20311433485058888,
+          "mean_confidence": 0.5533856651494111,
+          "brier_hard": 0.4063906772103004,
+          "brier_hard_n": 2000,
+          "nll_hard": 0.7152837956058272,
+          "nll_hard_n": 2000,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.46913207673904656,
+          "soft_accuracy_n": 2000,
+          "brier_soft": 0.06497074467877281,
+          "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.12038107186141371,
+          "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.17619801386776057,
+          "total_variation_n": 2000,
+          "score_mae": 0.2447599606823941,
+          "score_mae_n": 800,
+          "within_one_level": 0.99375,
+          "within_one_level_n": 800,
+          "macro_f1": 0.6349242079343602
+        },
+        "ag-news": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.935,
+          "accuracy_valid": 0.935,
+          "ece_top_label": 0.0277188639855619,
+          "mean_confidence": 0.9106097216911906,
+          "brier_hard": 0.10223891825148304,
+          "brier_hard_n": 600,
+          "nll_hard": 0.2046134786862912,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.9319475717225341
+        },
+        "emotion": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.565,
+          "accuracy_valid": 0.565,
+          "ece_top_label": 0.3127789762695219,
+          "mean_confidence": 0.8750361088494466,
+          "brier_hard": 0.7278820816911535,
+          "brier_hard_n": 600,
+          "nll_hard": 2.179428847187102,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.01,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.47063728412574746
+        },
+        "boolq": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.8116666666666666,
+          "accuracy_valid": 0.8116666666666666,
+          "ece_top_label": 0.08128733333333335,
+          "mean_confidence": 0.8882866666666667,
+          "brier_hard": 0.2800035268666667,
+          "brier_hard_n": 600,
+          "nll_hard": 0.4526075086973918,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "macro_f1": 0.7989317880539385
+        },
+        "sst5": {
+          "attempted": 600,
+          "valid": 600,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.47333333333333333,
+          "accuracy_valid": 0.47333333333333333,
+          "ece_top_label": 0.11648579870821457,
+          "mean_confidence": 0.5717307193165773,
+          "brier_hard": 0.6735168048802583,
+          "brier_hard_n": 600,
+          "nll_hard": 1.2841517570209144,
+          "nll_hard_n": 600,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 600,
+          "score_mae": 0.6724125988264632,
+          "score_mae_n": 600,
+          "within_one_level": 0.7883333333333333,
+          "within_one_level_n": 600,
+          "macro_f1": 0.46002232794452175
+        },
+        "prompt-injections": {
+          "attempted": 116,
+          "valid": 116,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7241379310344828,
+          "accuracy_valid": 0.7241379310344828,
+          "ece_top_label": 0.22484482758620694,
+          "mean_confidence": 0.9346275862068966,
+          "brier_hard": 0.44889866172413795,
+          "brier_hard_n": 116,
+          "nll_hard": 1.682957724359292,
+          "nll_hard_n": 116,
+          "zero_probability_gold": 0.02586206896551724,
+          "zero_probability_gold_n": 116,
+          "macro_f1": 0.7118012422360249
+        },
+        "massive-intent.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.7733333333333333,
+          "accuracy_valid": 0.7733333333333333,
+          "ece_top_label": 0.1762748719939652,
+          "mean_confidence": 0.9448706790799233,
+          "brier_hard": 0.3914895786578462,
+          "brier_hard_n": 300,
+          "nll_hard": 2.594218156353015,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.06333333333333334,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.4611733462473961
+        },
+        "xnli.en": {
+          "attempted": 300,
+          "valid": 300,
+          "failed": 0,
+          "coverage": 1.0,
+          "accuracy_all": 0.8766666666666667,
+          "accuracy_valid": 0.8766666666666667,
+          "ece_top_label": 0.059115758529190925,
+          "mean_confidence": 0.9151327612640476,
+          "brier_hard": 0.1964252077913328,
+          "brier_hard_n": 300,
+          "nll_hard": 0.36246740512671244,
+          "nll_hard_n": 300,
+          "zero_probability_gold": 0.0,
+          "zero_probability_gold_n": 300,
+          "macro_f1": 0.8760253913421515
+        }
+      }
+    }
+  },
+  "summary": {
+    "typed-decisions": {
+      "n": 5,
+      "mean_percent": 70.23,
+      "sample_variance_pp_squared": 44.56824999999997,
+      "sample_std_pp": 6.675945625901995
+    },
+    "ag-news": {
+      "n": 5,
+      "mean_percent": 54.56666666666666,
+      "sample_variance_pp_squared": 1255.536111111111,
+      "sample_std_pp": 35.433544997799906
+    },
+    "boolq": {
+      "n": 5,
+      "mean_percent": 63.266666666666666,
+      "sample_variance_pp_squared": 256.79999999999995,
+      "sample_std_pp": 16.0249804992081
+    },
+    "emotion": {
+      "n": 5,
+      "mean_percent": 30.333333333333332,
+      "sample_variance_pp_squared": 616.1666666666666,
+      "sample_std_pp": 24.82270466058577
+    },
+    "prompt-injections": {
+      "n": 5,
+      "mean_percent": 57.758620689655174,
+      "sample_variance_pp_squared": 241.5279429250891,
+      "sample_std_pp": 15.541169290793055
+    },
+    "sst5": {
+      "n": 5,
+      "mean_percent": 30.53333333333333,
+      "sample_variance_pp_squared": 185.14444444444447,
+      "sample_std_pp": 13.606779356057938
+    },
+    "massive-intent.en": {
+      "n": 5,
+      "mean_percent": 32.53333333333333,
+      "sample_variance_pp_squared": 1566.1999999999998,
+      "sample_std_pp": 39.57524478761944
+    },
+    "xnli.en": {
+      "n": 5,
+      "mean_percent": 55.53333333333334,
+      "sample_variance_pp_squared": 860.5333333333334,
+      "sample_std_pp": 29.33484844571953
+    }
+  },
+  "candidate_seed": 1,
+  "variance_definition": "sample variance, n-1, percentage points squared",
+  "benchmark_manifest": {
+    "format_version": 1,
+    "sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
+    "cases": 3516,
+    "decisions": 5116,
+    "profile": "heldout-eight",
+    "role": "test",
+    "parent_bundles": {
+      "core": "477c9cbd4f9eaf8844fa4d6870560e751960735e7441b803306d6e011e689435",
+      "language_en": "a3d41f5060826770f0f28b16a0aff26b4294691423624349db6b008789ea9096"
+    },
+    "purpose": "Base vs official typed-decisions specialist vs trained Laya-only input interface; no new training."
+  }
+}

requirements.txt

@@ -0,0 +1,8 @@
+torch==2.13.0
+laya==0.3.20
+transformers==5.17.0
+huggingface-hub==1.33.0
+safetensors==0.8.0
+numpy==2.4.4
+tokenizers==0.23.2
+datasets==5.0.1

training_protocol.json

@@ -0,0 +1,92 @@
+{
+  "experiment": "E0i",
+  "seeds": [
+    0,
+    1,
+    2,
+    3,
+    4
+  ],
+  "created_at": "2026-09-27T13:49:50.006241+00:00",
+  "data": {
+    "train": {
+      "format_version": 1,
+      "sha256": "5b273d74d4eed80f75763af64cfaf3fabae1d2a31a0b67551a5a753b6972e22e",
+      "cases": 1080,
+      "decisions": 5400,
+      "profile": "hybrid-train",
+      "role": "train",
+      "source": "LocalLLaMA/typed-decisions",
+      "revision": "f7a2487edd7a043a5441a5e9ccc7fe5ddbd9ebe8",
+      "source_split": "train",
+      "split_seed": 42
+    },
+    "validation": {
+      "format_version": 1,
+      "sha256": "95fe23fc9e82f9992c5056e7ab7ff76888aa614f49aa6d48929aba5d4605812a",
+      "cases": 120,
+      "decisions": 600,
+      "profile": "hybrid-validation",
+      "role": "validation",
+      "source": "LocalLLaMA/typed-decisions",
+      "revision": "f7a2487edd7a043a5441a5e9ccc7fe5ddbd9ebe8",
+      "source_split": "train",
+      "split_seed": 42
+    },
+    "test": {
+      "format_version": 1,
+      "sha256": "dd34a1df70013b7835b57aa474e36bf432f6e586a98e960e2b43b80ab7b75d06",
+      "cases": 400,
+      "decisions": 2000,
+      "profile": "typed-decisions-test",
+      "role": "test",
+      "source_split": "test",
+      "parent_sha256": "477c9cbd4f9eaf8844fa4d6870560e751960735e7441b803306d6e011e689435"
+    }
+  },
+  "heldout_manifest": {
+    "format_version": 1,
+    "sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
+    "cases": 3516,
+    "decisions": 5116,
+    "profile": "heldout-eight",
+    "role": "test",
+    "parent_bundles": {
+      "core": "477c9cbd4f9eaf8844fa4d6870560e751960735e7441b803306d6e011e689435",
+      "language_en": "a3d41f5060826770f0f28b16a0aff26b4294691423624349db6b008789ea9096"
+    },
+    "purpose": "Base vs official typed-decisions specialist vs trained Laya-only input interface; no new training."
+  },
+  "max_len": 1024,
+  "head_max_len": 256,
+  "epochs": null,
+  "min_epochs": 10,
+  "early_stopping_patience": 3,
+  "early_stopping_metric": "validation.soft_cross_entropy",
+  "early_stopping_min_delta": 0.0,
+  "microbatch": 8,
+  "accumulation": 4,
+  "effective_batch": 32,
+  "interface_lr": 0.0003,
+  "weight_decay": 0.01,
+  "gradient_clip": 1,
+  "training_scope": "bridge",
+  "training_dropout": false,
+  "selection": "minimum validation soft cross-entropy among trained epochs",
+  "release_candidate_selection": "lowest selected validation soft cross-entropy across all five seeds; tie goes to the lowest seed; no test-based selection",
+  "device": "cuda:1",
+  "deterministic": true,
+  "native_sdk_calibration": true,
+  "publish_policy": "draft locally; upload only after user and assistant agree",
+  "baseline_predictions_sha256": "37dff666a00a340cd6e37f775dc60df33a3c6eb78f76bfee7f094443c225b6e8",
+  "base_model": "convaiinnovations/laya",
+  "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+  "training_sources_sha256": {
+    "ariadne_bench/reproducibility.py": "5b0c161c596f25277b3326c645df53d5d1f2dc0272da9e77bc4a4c0d56188b6e",
+    "ariadne_bench/frozen_input_interface.py": "253f0186dac6e4a7f4b0fcf3c33449c3b3882cf8be538dca6eeca8d9e4ce9b83",
+    "ariadne_bench/full_input_interface.py": "a8f440a2185085c64896300a243ac52a9619c4d05a1b4e43c50c2c32aa1da996",
+    "ariadne_bench/full_finetune.py": "6067be8cf837d254b919c841c9a9851e1546a413dbf6324ff242e5d2f56c9734",
+    "ariadne_bench/interfaces.py": "19700fb8170b2460401900cb96ebdff22bd7bf778ca5132e98d56ba68f344187",
+    "ariadne_bench/experiments/align.py": "31e2733dcc223a1e1a6250d41c4158f84091f1cb4067d61388e06b5b01ae34c4"
+  }
+}

upload-manifest.json

@@ -0,0 +1,55 @@
+{
+  "published_by_assistant": false,
+  "candidate_seed": 1,
+  "selected_epoch": 14,
+  "format": "compact affine adapter requiring pinned base Laya",
+  "existing_model_card_preserved": true,
+  "inference_code_and_weights_match_verified_export": true,
+  "removed_only_other_seed_loading_entries": true,
+  "files_sha256": {
+    "LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
+    "THIRD_PARTY.md": "3dd2683344d45ce64c15f45c723f4b0db720deee84446365fdf4084a73ebeceb",
+    "USAGE.md": "e6d76a3a450fd589e52cab5190a065cbd783082a98d3524071bcf60a7015ebef",
+    "adapter.safetensors": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
+    "adapter_config.json": "8d7450b812f696e1dbeb3d7abfa13fbee0cde95aae0349a94e5f22c9b458ed1f",
+    "ariadne_bench/__init__.py": "a73c9fdf21413382cb93b523bfb3925aa3fd354e18faeead4650e86a30645411",
+    "ariadne_bench/__main__.py": "935a1c1166b0c1ea35a82256345000bf2c73ded718d77773bc27a71ecce28f7d",
+    "ariadne_bench/adapters.py": "3bd4820165be077b3b98fe83ac23e649e1ad69186ad5f2a4f5ee7addf1427cf5",
+    "ariadne_bench/cli.py": "de267c0502489eb4bb37e4b8f6faa8da8be37a56ced270f754424a40404d1af6",
+    "ariadne_bench/datasets.py": "31c722c64895519a4635b501da88862b446384682e3e8ee884495941d6990912",
+    "ariadne_bench/experiments/__init__.py": "95ad0a7f30c4e0a194d0a6e2ae8905abae5d622edbb3d4e5d791b4bc460aee5d",
+    "ariadne_bench/experiments/align.py": "31e2733dcc223a1e1a6250d41c4158f84091f1cb4067d61388e06b5b01ae34c4",
+    "ariadne_bench/experiments/prepare_alignment.py": "c4f39c10ce46ee13c0d89128d18a85ec4a86729e81e070f5834ed01fc1c405a9",
+    "ariadne_bench/experiments/spectrum.py": "22d4801b585c539d0d579a9d2a00076e1002fd337bfe31d1906d8ae3fb8d4320",
+    "ariadne_bench/frozen_input_interface.py": "253f0186dac6e4a7f4b0fcf3c33449c3b3882cf8be538dca6eeca8d9e4ce9b83",
+    "ariadne_bench/full_finetune.py": "6067be8cf837d254b919c841c9a9851e1546a413dbf6324ff242e5d2f56c9734",
+    "ariadne_bench/full_input_interface.py": "a8f440a2185085c64896300a243ac52a9619c4d05a1b4e43c50c2c32aa1da996",
+    "ariadne_bench/hybrid.py": "90fbbdab9b8157a32854d27c2d93089be6dc88dfe9320426f102eae4e03d9ffd",
+    "ariadne_bench/hybrid_control.py": "1b53678bea73fe7608570b78d7215d1dff2e38d6ad3627615f26c06ae74d8b4c",
+    "ariadne_bench/interfaces.py": "19700fb8170b2460401900cb96ebdff22bd7bf778ca5132e98d56ba68f344187",
+    "ariadne_bench/metrics.py": "fc6fde20c95d5052de52d05d6fb9b6eb69b780c46966f5bf1e517738b276a4b4",
+    "ariadne_bench/portable.py": "259f633a8bb6663d0594d9639ea8d80ef493f5a63d4b11444a4003f2f9a318e8",
+    "ariadne_bench/reproducibility.py": "5b0c161c596f25277b3326c645df53d5d1f2dc0272da9e77bc4a4c0d56188b6e",
+    "ariadne_bench/runner.py": "b20e2af4ba254e2d48a07563356f95728c4807018c3c989b429e9922674c64cd",
+    "ariadne_bench/schema.py": "67794f1172c1ef9d9625b84c8096fd48b8fa7cd10e3f88fc13ba3a98ef7c7a76",
+    "benchmarks/sources.lock.json": "6e6eaab2778092c8127257fce202c9fa8d4625dd0d029490dd9f137da91d134c",
+    "environment.json": "c5b9c796d97f3dec6d242caf79aa5d2ac69c91717f2be6ed5a51627bf6d58c17",
+    "evidence/adapter-equivalence.json": "c77faa691bd273d363dbb196155545b4f72697b9d0f4c80451ffb06b0090d200",
+    "evidence/audit.json": "1fb474a47923e22374a47e06f545e747420c0a7bd269bbbeaabb5869e89a8d70",
+    "evidence/candidate-uncertainty.json": "39eba4a205eaea5f13384da24ca8563de364c9edb6858c1db38eb4111960ce40",
+    "evidence/identity-check.json": "0cc8904dc58e1e0ed49496c7ae01980c52e3a1395d61ef91286cec2c9dcdfa25",
+    "evidence/replay-check.json": "4f246eb3d6afc0c0b1d306a20b82060ae564d476ec22451322b065edb946d15e",
+    "evidence/seed-0-history.json": "bc3de468970bf350426d9c4281456e258984838feb4d8dbbfe54c8752ebb24b7",
+    "evidence/seed-1-history.json": "8ee78b5685138712512b39035ef48a491ed52813c2131a8e9b80715871ab9405",
+    "evidence/seed-2-history.json": "d9bf7ab3991b85b6e3877b9ebd4ca2fc113cba1716f56da5aa16f4018ac4b197",
+    "evidence/seed-3-history.json": "7c0f19638e31f28e438af879ceaa68bbe06480e6f7fb72e8e749b4aabb304341",
+    "evidence/seed-4-history.json": "601c5bd91dcbb004ff8a07621f3f125ba578cc4107a9884e8fe69ac17751a741",
+    "export-verification.json": "b9f8667626e2facd03f310c0c67c100ddde9481218c51cbead7972fd048d4afb",
+    "licenses/jev-benchmarks-LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
+    "licenses/laya-LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
+    "load_adapter.py": "2f305ef95fef2b2482e9214836fc0071a59cfd6c6d3aaf42cebdbfb131272081",
+    "metrics.json": "dd45292f04dc5b81cc958a39f618aa8748b5198e39b5effee6c95520c92d94b6",
+    "requirements.txt": "285a8ae6529abeefd058f2e9679e0e314b19fda38169e685ea16aec59b053bd9",
+    "training_protocol.json": "dccf81e3d8a70c756a95a88280cbec139eea43bc391d67fa7b40e384a8824f57"
+  }
+}