GoatHerder commited on 5 days ago

Commit

5f2a11f

verified ·

1 Parent(s): 086613e

Upload 49 files

Browse files

Files changed (20) hide show

README.md +121 -261
USAGE.md +2 -47
adapter.safetensors +1 -1
adapter_config.json +4 -4
evidence/adapter-equivalence.json +6 -1
evidence/audit.json +10 -10
evidence/candidate-uncertainty.json +25 -25
evidence/lr-comparison.json +315 -0
evidence/replay-check.json +6 -3
evidence/seed-0-history.json +79 -232
evidence/seed-1-history.json +67 -152
evidence/seed-2-history.json +93 -127
evidence/seed-3-history.json +61 -333
evidence/seed-4-history.json +75 -58
export-verification.json +5 -5
make_training_config.py +37 -0
metrics.json +391 -391
release_manifest.json +54 -0
training_protocol.json +19 -4
upload-manifest.json +22 -20

README.md CHANGED Viewed

@@ -2,327 +2,187 @@
 license: apache-2.0
 language:
 - en
-base_model:
-- convaiinnovations/laya
-base_model_relation: adapter
 pipeline_tag: text-classification
 tags:
 - laya
-- typed-decisions
-- system-one
-- classification
 - adapter
-- linear-interface
-- representation-alignment
-datasets:
-- LocalLLaMA/typed-decisions
-metrics:
-- accuracy
-- brier_score
-library_name: transformers
 ---
-# Ariadne-Laya-TD
-**Ariadne** is a lightweight representation-alignment adapter for the Laya base model.
-Ariadne keeps the underlying Laya encoder frozen and trains only a single identity-initialized affine transformation immediately before the encoder stack:
-$$
-h' = Wh + b,\qquad W_0 = I,\qquad b_0 = 0
-$$
-The interface contains just **1,049,600 trainable parameters**.
-The experiment asks a simple question:
-> **Instead of adapting the representation to the task, can we adapt the task to the representation?**
-Early results suggest that much of the discrimination required for typed decisions is already accessible in the frozen Laya base representation.
-Ariadne learns a small transformation that allows the existing model to use that structure without modifying the encoder itself.
-This makes specialisms extremely lightweight: a deployment can keep one frozen Laya base resident and switch between task-specific Ariadne interfaces containing only 1.05M parameters—roughly 2 MB in BF16—rather than loading a separate ~421M-parameter model for each specialism.
 ## Architecture
-Ariadne adds one trainable affine transformation before the frozen Laya encoder:
-```text
-input
-  ↓
-Laya embeddings
-  ↓
-Ariadne linear interface
-  ↓
-frozen Laya encoder
-  ↓
-existing decision pathway
-```
-The interface is deliberately minimal:
-- one 1024 × 1024 linear transformation
-- one 1024-dimensional bias
-- exact identity initialization
-- **1,049,600 trainable parameters**
-- no LoRA matrices
-- no additional transformer blocks
-- no modification of the Laya encoder during interface training
-At initialization, Ariadne is exactly equivalent to the unmodified base model.
-Training therefore learns only how to transform the representation presented to the existing frozen network.
-## Base model
-Ariadne-Laya-TD is an adapter for:
-`convaiinnovations/laya`
-The underlying Laya base model remains frozen during Ariadne training.
-The principal comparison model is the official Laya typed-decisions specialist:
-`convaiinnovations/laya-typed-decisions`
-That specialist is obtained through conventional full-model specialization and serves as the reference for how much typed-decisions capability can be recovered without modifying the base encoder.
-## Typed-decisions result
-The primary experiment trains Ariadne on the `LocalLLaMA/typed-decisions` training data while keeping the entire Laya base model frozen.
-Only the 1.05M-parameter interface is updated.
-### Preliminary deterministic results
-The deterministic multi-seed sweep is currently being completed.
-Individual runs have already demonstrated typed-decisions performance at or above the official Laya specialist's approximately **76.7%** reference accuracy while leaving the underlying encoder frozen.
-One deterministic run reached:
-| Model | Trainable parameters | Typed-decisions accuracy |
 |---|---:|---:|
-| Official Laya typed-decisions specialist | ~421M during specialization | 76.70% |
-| **Ariadne on frozen Laya base** | **1,049,600** | **76.95%** |
-This is currently a **single-run result**, not the final multi-seed mean.
-Ariadne training has shown significant convergence sensitivity between runs. The final five-seed deterministic statistics will replace this preliminary result before the benchmark is treated as stable.
-## What this result means
-The goal is not simply to reduce the number of trainable parameters.
-The frozen-base experiment tests whether task-specific discrimination requires rewriting the internal representation of the model at all.
-The working hypothesis is:
-> **The base model already contains much of the discriminative structure required by typed decisions. Ariadne learns how to align the task with that existing structure rather than modifying hundreds of millions of encoder parameters to align the representation with the task.**
-A successful interface therefore does not teach the frozen encoder new contextual computation.
-Instead, it changes the coordinates in which the existing computation receives its input.
-This can be viewed as **representation alignment** rather than conventional representation adaptation.
-## Post-specialization experiment
-A second experiment applies the same interface concept to the official Laya typed-decisions specialist.
-In this experiment:
-- the specialist itself is frozen
-- only the Ariadne interface is trained
-- five seeds are evaluated under the same benchmark bundle
-Results:
-| Method | Trainable parameters | Typed-decisions accuracy |
 |---|---:|---:|
-| Untouched Laya specialist | — | 76.70% |
-| **Specialist + Ariadne linear interface** | **1,049,600** | **78.10 ± 0.31%** |
-| Residual-interface ablation | 1,049,600 | 78.24 ± 0.75% |
-| LayerNorm + Linear ablation | ~1.05M | 77.72 ± 0.53% |
-| Continued specialist full fine-tune | 421,029,889 | 78.42 ± 0.75% |
-Values after ± are sample standard deviations across five seeds.
-The plain linear interface was the most stable adaptation method tested.
-Continued full-model fine-tuning produced a mean accuracy only **0.32 percentage points higher** than the linear interface while updating roughly **400× more parameters**.
-With only five seeds, this small difference does not establish a reliable ordering between the two methods.
-## Cross-task retention after specialist alignment
-The post-specialization Ariadne interface was also evaluated on seven additional English benchmarks.
-| Benchmark | Untouched specialist | Specialist + Ariadne |
-|---|---:|---:|
-| Typed decisions | 76.70% | **78.10 ± 0.31%** |
-| AG News | 94.67% | 94.17 ± 0.37% |
-| BoolQ | 82.00% | 81.10 ± 0.90% |
-| DAIR Emotion | 58.17% | 57.40 ± 0.65% |
-| Prompt injections | 67.24% | 67.41 ± 1.87% |
-| SST-5 | 48.00% | 49.33 ± 1.67% |
-| MASSIVE intent EN | 81.00% | 79.40 ± 1.23% |
-| XNLI EN | 86.33% | 88.33 ± 0.53% |
-Across the seven non-target tasks, the mean accuracy change relative to the untouched specialist is approximately **−0.04 percentage points**.
-This does not imply that every capability is preserved identically. Individual tasks move in both directions.
-It does indicate that the gain on typed decisions is not accompanied by a broad average collapse across this evaluation bundle.
 ## Training
-### Frozen-base Ariadne
-The released Ariadne checkpoint is trained from the official Laya base model.
-During training:
-- all Laya parameters remain frozen
-- the frozen encoder is run in evaluation mode
-- only the Ariadne interface is trainable
-- initialization is exact identity
-- training uses soft typed-decision targets
-- checkpoint selection uses validation soft cross-entropy
-Current training experiments use deterministic execution settings to better separate seed effects from GPU numerical nondeterminism.
-The final training configuration and five-seed statistics will be added once the current deterministic sweep is complete.
-### Post-specialist Ariadne experiment
-The completed post-specialist sweep used:
-- trainable parameters: 1,049,600
-- effective batch size: 32
-- optimizer: AdamW
-- learning rate: `3e-4`
-- weight decay: `0.01`
-- gradient clipping: `1.0`
-- objective: soft-target decision cross-entropy
-- seeds: 0, 1, 2, 3, 4
-- checkpoint selection: lowest validation soft cross-entropy
-## Dataset
-Ariadne is trained on:
-`LocalLLaMA/typed-decisions`
-The primary benchmark contains:
-- **400 test cases**
-- **2,000 individual decisions**
-Training and validation use data derived from the public training split.
-The official test split is excluded from training and checkpoint selection.
-The test set was inspected during development, so results should be interpreted as research benchmark results rather than performance on a completely untouched external test set.
-## Evaluation
-Accuracy counts invalid or missing decisions as incorrect.
-Probability-based metrics are calculated over valid decisions.
-The evaluation harness tracks:
-- hard accuracy
-- soft accuracy
-- soft cross-entropy
-- hard Brier score
-- soft Brier score
-- expected calibration error (ECE)
-ECE uses the predicted option probability rather than a separate provider confidence field.
-Brier scores sum over classes.
-Evaluation bundle SHA-256:
-```text
-10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854
 ```
-## Calibration
-Hard decision accuracy and probability calibration should be treated separately.
-The upstream Laya checkpoint may emit warnings for stored temperature values outside the runtime-supported calibration range. Laya clamps affected values and warns that confidence for those entries should be treated as uncalibrated.
-Ariadne's improvements in hard accuracy should therefore **not** be interpreted as evidence of improved calibration.
-Soft cross-entropy, Brier scores and ECE are reported separately where available.
-## Why "Ariadne"?
-Ariadne is named after the thread through the labyrinth.
-The metaphor reflects the central idea of the method: the useful representational structure may already exist inside the model; the interface learns a path through that structure rather than reconstructing it.
-In practical terms:
-> **Conventional fine-tuning moves the representation toward the task. Ariadne attempts to move the task toward the representation.**
-## Intended use
-Ariadne is intended for research and experimentation involving:
-- representation alignment
-- frozen encoders
-- parameter-efficient adaptation
-- System-1 and selector models
-- typed decision systems
-- reuse of pretrained representations
-- comparison with conventional full fine-tuning
-- comparison with PEFT methods such as LoRA and adapters
-The approach may be particularly useful when many specialist tasks need to share a large common frozen encoder.
-## Limitations
-- The current release focuses on English typed decisions.
-- The primary benchmark consists of synthetic decision workflows.
-- Frozen-base training currently shows significant convergence sensitivity.
-- Individual high-performing runs should not be confused with multi-seed average performance.
-- The deterministic frozen-base five-seed sweep is still being completed.
-- The typed-decisions test set was inspected during development.
-- Cross-task results should not be assumed to generalize to arbitrary domains.
-- Hard accuracy does not imply good probability calibration.
-- Ariadne depends on the upstream Laya model and is not a standalone foundation model.
-- The model should not be used for safety-critical decisions without independent task-specific validation.
-## Reproducibility
-The experimental artifacts include:
-- deterministic and non-deterministic training histories
-- per-seed validation metrics
-- per-task predictions
-- checkpoint hashes
-- frozen-parameter verification
-- machine-readable benchmark reports
-- per-seed/per-task CSV results
-The test suite verifies that interface-only experiments do not alter the frozen upstream model parameters.
-The current audited experimental suite includes **43 passing tests**.
-## Relationship to Laya
-Ariadne is independently developed and is not an official Convai Innovations model.
-The upstream Laya models and code are released under Apache-2.0.
-Ariadne's interface weights and accompanying code are also released under Apache-2.0.
-## Citation
-A technical report describing Ariadne and the representation-alignment experiments is in preparation.
-Until then, please cite this Hugging Face repository and the upstream Laya project when using Ariadne.

 license: apache-2.0
 language:
 - en
 pipeline_tag: text-classification
+base_model: convaiinnovations/laya
+base_model_relation: adapter
+datasets:
+- LocalLLaMA/typed-decisions
 tags:
 - laya
+- modernbert
 - adapter
+- typed-decisions
+- custom-code
 ---
+# Ariadne-Laya-TD — stable training with a small input adapter
+**A small typed-decision adapter for frozen Laya, with a more stable training recipe.**
+A 1,049,600-parameter affine input adapter trained on typed decisions while every parameter of base Laya stays frozen. It requires the pinned base model; the small adapter file is not a standalone model.
+The included checkpoint scores **75.45%** on the local Typed Decisions test, compared with **36.25%** for the matched base. It is seed 4, epoch 8, selected by validation loss. This release uses LR 0.0001.
+Across five deterministic runs, typed-decision accuracy was **75.28% ± 0.93 percentage points** (sample SD). The matched native base scored **36.25%**. The table below reports all eight tasks and every seed.
+**Lower learning rate improved training stability.** Changing LR from 3e-4 to 1e-4 raised mean typed-decision accuracy from 70.23% to 75.28% and reduced the across-seed standard deviation from 6.68 to 0.93 points. Sample variance fell by 98.1% in this five-seed comparison. The complete learning-rate comparison is below.
+**Consistency across seeds.** Typed-decision accuracy ranged from 74.25% to 76.55%. AG News stayed between 93.67% and 95.17%, and MASSIVE between 75.00% and 79.00%. These are measurements of this recipe on five seeds and the fixed benchmark bundle.
 ## Architecture
+The map is `x′ = Wx + b`, initialized with `W = I`, `b = 0`, immediately after Laya's original token embedding and normalization. Original embeddings, normalization, all 28 ModernBERT blocks, and decision heads remain frozen and in evaluation mode. Gradients pass through the frozen stack to update only the map. There is no added LLM and no extra BERT forward pass.
+The combined model has 422,343,427 parameters; 1,049,600 were trainable (about 0.25%). Each FP32 adapter checkpoint is approximately 4.2 MB. The original model and tokenizer are downloaded separately.
+Base: [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya/tree/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851), revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Source weights were hashed before training and at checkpoint saves; all pretrained tensors remained unchanged.
+## Evaluation
+All scores below are local measurements on identical frozen cases. Accuracy is the highest-probability label accuracy, including ordinal questions; failed decisions count as incorrect. Values are percentages; variance is sample variance in percentage-points squared. Five seeds are used for the mean and SD. These are not the upstream model card's historical benchmark numbers.
+| Benchmark | Decisions | Native base | Adapter mean ± SD | Variance | Selected checkpoint |
+|---|---:|---:|---:|---:|---:|
+| Typed decisions | 2000 | 36.25 | 75.28 ± 0.93 | 0.8620 | 75.45 |
+| AG News | 600 | 94.83 | 94.47 ± 0.58 | 0.3389 | 95.17 |
+| BoolQ | 600 | 83.00 | 82.10 ± 0.89 | 0.7861 | 83.17 |
+| DAIR Emotion | 600 | 57.33 | 57.20 ± 1.00 | 1.0056 | 57.83 |
+| Prompt injections | 116 | 68.97 | 74.48 ± 3.93 | 15.4578 | 74.14 |
+| SST-5 | 600 | 37.00 | 41.00 ± 2.46 | 6.0278 | 41.00 |
+| MASSIVE intent EN | 300 | 78.67 | 76.80 ± 1.56 | 2.4222 | 77.00 |
+| XNLI EN | 300 | 86.00 | 87.27 ± 1.04 | 1.0778 | 86.33 |
+### What changed from the first release
+The earlier adapter used LR 3e-4. This version uses 1e-4 with the same seeds, data, initialization, batch, stopping rule and deterministic runtime. The table compares all five runs of each recipe. Different stopping epochs are a consequence of the shared validation rule.
+| Benchmark | Earlier LR 3e-4 mean ± SD | This LR 1e-4 mean ± SD |
 |---|---:|---:|
+| Typed decisions | 70.23 ± 6.68 | 75.28 ± 0.93 |
+| AG News | 54.57 ± 35.43 | 94.47 ± 0.58 |
+| BoolQ | 63.27 ± 16.02 | 82.10 ± 0.89 |
+| DAIR Emotion | 30.33 ± 24.82 | 57.20 ± 1.00 |
+| Prompt injections | 57.76 ± 15.54 | 74.48 ± 3.93 |
+| SST-5 | 30.53 ± 13.61 | 41.00 ± 2.46 |
+| MASSIVE intent EN | 32.53 ± 39.58 | 76.80 ± 1.56 |
+| XNLI EN | 55.53 ± 29.33 | 87.27 ± 1.04 |
+The earlier published checkpoint was seed 1, epoch 14. The new validation-selected checkpoint improves AG News from 93.17% to 95.17%, BoolQ from 79.83% to 83.17%, and MASSIVE from 74.33% to 77.00%. Its typed-decision score is 75.45%, compared with 76.95% for the earlier checkpoint. The five-seed means above measure the improvement in the training recipe.
+### Individual seeds
+| Seed | Typed | AG News | BoolQ | Emotion | Injection | SST-5 | MASSIVE | XNLI | Best epoch | Stop epoch |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 0 | 74.25 | 94.17 | 80.83 | 58.50 | 69.83 | 38.17 | 79.00 | 89.00 | 11 | 14 |
+| 1 | 75.65 | 94.83 | 82.67 | 57.17 | 78.45 | 44.83 | 75.67 | 87.00 | 9 | 12 |
+| 2 | 74.50 | 94.50 | 81.83 | 56.00 | 71.55 | 39.83 | 75.00 | 87.33 | 13 | 16 |
+| 3 | 76.55 | 93.67 | 82.00 | 56.50 | 78.45 | 41.17 | 77.33 | 86.67 | 8 | 11 |
+| 4 | 75.45 | 95.17 | 83.17 | 57.83 | 74.14 | 41.00 | 77.00 | 86.33 | 8 | 11 |
+### Selected checkpoint versus base: paired uncertainty
+These descriptive 95% intervals resample cases 10,000 times, keeping each typed case's five decisions together. They describe benchmark-case uncertainty for the selected model, separately from the across-seed SD above. They have no multiple-comparison correction and were not used to select the checkpoint.
+| Benchmark | Change from base (pp) | Paired 95% interval (pp) |
 |---|---:|---:|
+| Typed decisions | +39.20 | [+35.90, +42.50] |
+| AG News | +0.33 | [-0.50, +1.33] |
+| BoolQ | +0.17 | [-1.50, +1.83] |
+| DAIR Emotion | +0.50 | [-1.50, +2.67] |
+| Prompt injections | +5.17 | [+1.72, +9.48] |
+| SST-5 | +4.00 | [+1.00, +6.83] |
+| MASSIVE intent EN | -1.67 | [-4.33, +1.00] |
+| XNLI EN | +0.33 | [-1.67, +2.33] |
+### Checkpoint selection
+The default package contains **seed 4, epoch 8**, selected by the lowest validation soft cross-entropy across the five seeds (0.854020). This selection rule was recorded before the sweep. Test performance was not used to choose the checkpoint. Only the selected adapter weights are included; all five seeds’ results are reported.
+### Benchmark scope
+- Typed Decisions: 400 test cases / 2,000 decisions from four synthetic workflows. Training and validation case IDs are disjoint from test IDs. The test task itself is the adaptation task.
+- AG News, BoolQ, Emotion and SST-5: fixed prefixes of 600 examples each. Prompt injections: all 116 test examples. MASSIVE and XNLI: first 300 English test examples each.
+- MASSIVE uses the gold intent plus 19 seeded distractors (20 options), not all intents. XNLI uses three labels.
+- These seven retention tasks were excluded from adapter training; some were present in the base model's training. These are held-out examples for this adaptation, not proof of wholly unseen-task generalization.
+- The test suite was inspected during earlier exploratory experiments. This recipe was selected adaptively after earlier results; seed 0 began as the lower-LR pilot and was retained for the five-seed confirmation. Checkpoint selection used validation only, but the research process is not a blind test. Earlier runs with changing stopping rules are excluded from the main table.
+- No downstream deployment, multilingual retention, or broad safety claim is established by these small English benchmark subsets.
 ## Training
+Training data: [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions), revision `f7a2487edd7a043a5441a5e9ccc7fe5ddbd9ebe8`. A case-level split with seed 42 yields 1,080 training cases (5,400 decisions) and 120 validation cases (600 decisions). Questions from one case never cross splits.
+- Seeds 0, 1, 2, 3, 4; RTX PRO 6000 Blackwell only, sequential runs.
+- Every map starts at the same identity transformation and native dropout stays off. Seeds control training-example shuffling; this comparison measures sensitivity to data order with deterministic computation.
+- AdamW, LR 0.0001, weight decay 0.01, gradient clipping at norm 1; effective batch 32 (microbatch 8 × accumulation 4). No learning-rate warmup or scheduler.
+- Soft-target cross-entropy on raw logits. The pretrained stack stays in eval mode; its dropout is off. The affine map is computed in FP32; the remaining CUDA forward uses BF16 autocast.
+- Minimum 10 completed epochs; stop after three consecutive epochs without a strictly lower validation loss. Patience counts from epoch 1, with stopping gated by the minimum. No fixed maximum. Keep the best trained checkpoint; epoch zero is a separate baseline.
+- Input/head token budgets: 1024/256, shared by native baseline and adapter. These differ from the original base config's defaults, so this card reports matched-budget local baselines.
+## Reproducibility
+Strict deterministic algorithms were enabled with errors on unsupported operations, cuDNN benchmarking and cuDNN SDPA were disabled, and float32 matmul precision was set to highest. Python, NumPy and PyTorch seeds were set. Processes used `PYTHONHASHSEED=0` and `CUBLAS_WORKSPACE_CONFIG=:4096:8`.
+Two independent two-epoch seed-0 verification runs (338 optimizer updates each) matched losses, validation metrics, and checkpoint tensors bit for bit. The reported seed-0 run matched that two-epoch metric prefix. This verifies the checked environment and prefix; it does not promise bitwise agreement across GPUs, libraries or operating systems. See `environment.json` and `evidence/replay-check.json`.
+Before training, the identity adapter matched every native-base answer object on all 5,116 benchmark decisions. All five final runs were independently rescored and checked for unchanged pretrained weights. Export verification is recorded separately in `export-verification.json` after testing the portable package.
+## Confidence and temperature warning
+The pinned base checkpoint ships `choice:11+` temperature 0.1005828. Laya 0.3.20 clamps it to 0.5 and emits a warning on load. Both baseline and adapters use that same behavior. Only the 300 MASSIVE examples use this bucket in this suite. These temperatures are not used in the training loss.
+Temperature scaling preserves class order mathematically but changes probability sharpness and calibration metrics. Calibration was not refitted after adaptation. Confidence calibration should be evaluated for the intended application; the base model's calibration claims have not been established for this adapter.
+## Load locally
+This is a custom adapter. Use the bundled loader: `laya.load(repo_id)` and `AutoModel.from_pretrained(repo_id)` do not install this input transformation.
+Download the repository files with the Hugging Face UI or `hf download GoatHerder/Ariadne-Laya-TD --local-dir ariadne-laya-td`, then change into that directory. The learning rate and selected seed are recorded in `training_protocol.json` and `adapter_config.json`.
+Install a PyTorch build for your GPU and the versions in `requirements.txt`. The measured environment used PyTorch 2.13.0+cu130; package versions and CUDA/cuDNN details are in `environment.json`. Run from this downloaded repository directory:
+```python
+from load_adapter import load
+model = load(device="cuda:0")  # defaults to the validation-selected seed
+answer = model.predict("The customer says the delivery arrived damaged.", {
+    "route": {"type": "choice", "instructions": "Choose the support queue.",
+              "criteria": {"delivery": "Delivery and damaged items",
+                           "billing": "Payments and invoices"}}
+})
+print(answer)
+model.close()
 ```
+Only the validation-selected seed is included in this compact package. Use `device='cpu'` for CPU inference. The loader downloads the pinned base, verifies base configuration/tokenizer hashes and model tensors, checks adapter weights, then freezes all parameters for inference. `local_files_only=True` uses cached weights; `base_path` can point to an existing copy of the pinned base.
+## Reproduce preparation and one training run
+The bundled `ariadne_bench` source contains the actual tokenizer/prompt preparation, training loop, adapters and scorer. Dataset files are downloaded from their pinned sources; evaluation data is not redistributed.
+```bash
+python -m ariadne_bench.experiments.prepare_alignment --out data/alignment
+python -m ariadne_bench prepare --profile laya-core \
+  --suites typed-decisions,ag-news,emotion,boolq,sst5,prompt-injections,massive-intent.en,xnli.en \
+  --out data/heldout-eight
+```
+The expected case-file hashes and immutable dataset revisions are in `training_protocol.json` and `benchmarks/sources.lock.json`. `make_training_config.py` resolves the pinned base into a local training config:
+```bash
+python make_training_config.py --seed 0 --device cuda:0 --out train-config.json
+PYTHONHASHSEED=0 CUBLAS_WORKSPACE_CONFIG=:4096:8 USE_TF=0 OMP_NUM_THREADS=4 \
+  python -m ariadne_bench.experiments.align --config train-config.json \
+  --train data/alignment/train --validation data/alignment/validation \
+  --out training-seed-0 --epochs 0 --min-epochs 10 --early-stopping-patience 3 \
+  --batch-size 8 --accumulation 4 --lr 0.0001 --keep-best-only
+PYTHONHASHSEED=0 CUBLAS_WORKSPACE_CONFIG=:4096:8 USE_TF=0 OMP_NUM_THREADS=4 \
+  python -m ariadne_bench run --data data/heldout-eight \
+  --config training-seed-0/benchmark-config.json --out evaluation-seed-0 --warmup 5
+```
+## License and attribution
+Apache-2.0. The [base-model card](https://huggingface.co/convaiinnovations/laya/blob/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851/README.md) and [Typed Decisions dataset card](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/blob/f7a2487edd7a043a5441a5e9ccc7fe5ddbd9ebe8/README.md) list Apache-2.0. Base weights are downloaded from their original repository. Evaluation datasets retain their own licenses. Prompt preparation credits and upstream notices are in `THIRD_PARTY.md` and `licenses/`.
+This is an independent adaptation of Laya. The upstream authors did not produce these adapter weights or experimental results.

USAGE.md CHANGED Viewed

@@ -1,48 +1,3 @@
-# Load the Laya linear adapter
-This package contains the validation-selected **seed 1, epoch 14** checkpoint from the deterministic frozen-base experiment. It trained 1,049,600 affine parameters at LR 3e-4 while all base Laya weights remained frozen. Its 4.2 MB weight file requires the pinned base model, which the loader downloads separately.
-The ongoing lower-learning-rate experiment is separate. These are the already verified weights, not a claim that the training stability investigation is complete.
-## Upload
-Extract the ZIP and upload the files and folders **at the root of your existing Hugging Face model repository**. There is deliberately no README.md in this bundle, so it can coexist with your existing model card. The ZIP is a transport archive; upload its extracted contents for normal use.
-This is a custom input adapter. Loading requires the bundled loader; bare `laya.load(repo_id)` and `AutoModel.from_pretrained(repo_id)` do not install the extra affine layer.
-## Load a downloaded copy
-Install an appropriate PyTorch build and the versions in `requirements.txt`. The measured GPU environment is recorded in `environment.json`. From the downloaded repository directory:
-```python
-from load_adapter import load
-model = load(device="cuda:0")
-answer = model.predict("The delivery arrived damaged.", {
-    "route": {
-        "type": "choice",
-        "instructions": "Choose the customer-support queue.",
-        "criteria": {"delivery": "Delivery and damaged items", "billing": "Payments and invoices"},
-    }
-})
-print(answer)
-model.close()
-```
-After you upload, first download your repository with `huggingface_hub.snapshot_download("YOUR_ACCOUNT/YOUR_REPOSITORY")`, then run the example from that downloaded directory. Use `device="cpu"` for CPU inference. `base_path` can point to the pinned base snapshot already on disk; `local_files_only=True` uses cached base files.
-The loader verifies the adapter checksum, base configuration/tokenizer hashes and pretrained tensor hash. Strict determinism is enabled by default and requires the CUBLAS workspace environment to be set before CUDA is initialized. A fresh process handles this automatically. Numerical agreement across other hardware or software is not guaranteed.
-## Measured performance and limits
-Selected checkpoint accuracy: typed decisions **76.95%**, AG News **93.17%**, BoolQ **79.83%**, Emotion **57.33%**, prompt injections **71.55%**, SST-5 **42.00%**, MASSIVE EN **74.33%**, XNLI EN **87.67%**. Native base typed accuracy was 36.25% under the matched protocol.
-Across all five training seeds, typed accuracy was **70.23% ± 6.68 pp** (sample SD). Three seeds had severe retention losses. The selected model also lost 3.17 pp on BoolQ and 4.33 pp on MASSIVE versus base. Full results and paired uncertainty are in `metrics.json` and `evidence/candidate-uncertainty.json`. Other seeds' scores are included for transparency; this bundle contains only seed 1's weights.
-The benchmark uses 2,000 typed decisions plus fixed subsets of seven other tasks, totaling 5,116 decisions. MASSIVE uses 20 candidate intents. The same test subsets were inspected in earlier experiments. These results describe an experimental task adapter, not established broad task improvement.
-The base checkpoint's choice:11+ temperature is clamped from 0.1005828 to 0.5 by Laya 0.3.20. This expected load warning concerns confidence calibration; raw-logit training and class ordering are unaffected. Calibration was not refitted after adaptation.
-All 5,116 answer objects matched the original selected checkpoint when the portable export was tested from outside the workspace. `export-verification.json` records that check. `upload-manifest.json` verifies that this bundle keeps the same weights and inference code, with the other seeds removed from its loading configuration.
-License: Apache-2.0. Base model: `convaiinnovations/laya`, pinned to revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`. Attribution is in `THIRD_PARTY.md` and `licenses/`.


1	+ # Using this adapter
2
3	+ This package uses LR 0.0001, seed 4, epoch 8. See [the model card](README.md#load-locally) for the loading example, all five seeds' results, requirements and limitations. Load with the bundled `load_adapter.py`; the adapter weights require the pinned base Laya.

adapter.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a
 size 4198616

 version https://git-lfs.github.com/spec/v1
+oid sha256:f23ad342a969773bbbcb18619ce8a8843bbb105d94fe64c5a837d68f26829081
 size 4198616

adapter_config.json CHANGED Viewed

@@ -4,7 +4,7 @@
   "position": "after native embeddings, before ModernBERT block 0",
   "base_model": "convaiinnovations/laya",
   "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
-  "candidate_seed": 1,
   "max_len": 1024,
   "head_max_len": 256,
   "trainable_parameters_during_training": 1049600,
@@ -16,10 +16,10 @@
     "tokenizer/tokenizer_config.json": "50044de60daaa73df97d262e15a40d4faf0160e7d742df64b377877a1320dd12"
   },
   "seeds": {
-    "1": {
       "weights_file": "adapter.safetensors",
-      "weights_sha256": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
-      "selected_epoch": 14
     }
   }
 }

   "position": "after native embeddings, before ModernBERT block 0",
   "base_model": "convaiinnovations/laya",
   "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+  "candidate_seed": 4,
   "max_len": 1024,
   "head_max_len": 256,
   "trainable_parameters_during_training": 1049600,
     "tokenizer/tokenizer_config.json": "50044de60daaa73df97d262e15a40d4faf0160e7d742df64b377877a1320dd12"
   },
   "seeds": {
+    "4": {
       "weights_file": "adapter.safetensors",
+      "weights_sha256": "f23ad342a969773bbbcb18619ce8a8843bbb105d94fe64c5a837d68f26829081",
+      "selected_epoch": 8
     }
   }
 }

evidence/adapter-equivalence.json CHANGED Viewed

@@ -35,5 +35,10 @@
       "forward_exact": true,
       "exact_match": true
     }
-  ]
 }

       "forward_exact": true,
       "exact_match": true
     }
+  ],
+  "reuse_provenance": {
+    "experiment": "E0i",
+    "source_sha256": "c77faa691bd273d363dbb196155545b4f72697b9d0f4c80451ffb06b0090d200",
+    "reason": "unchanged adapter computation and strict runtime; this forward/backward diagnostic has no optimizer step or learning-rate dependence"
+  }
 }

evidence/audit.json CHANGED Viewed

@@ -5,52 +5,52 @@
   "runs": [
     {
       "seed": 0,
-      "checkpoint_sha256": "bcb37755de54782bc4d1f79615539f48940aedb2b733dadc1033fba2f2304cf8",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
-      "stop_epoch": 23,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 1,
-      "checkpoint_sha256": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
-      "stop_epoch": 17,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 2,
-      "checkpoint_sha256": "61391c3869c1f4a8449444db6a9e3b660692eefa25e3c149cee0d6af5daf264c",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
-      "stop_epoch": 18,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 3,
-      "checkpoint_sha256": "26ed2443b8ea146d3fab98e94ab55356557b9ef10ac5b8ba656b1a37cf4f0f64",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
-      "stop_epoch": 27,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 4,
-      "checkpoint_sha256": "603a83ca4996e32b4753250ea6b07856e7cc5196b41e635cdad35ee4270eca38",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
-      "stop_epoch": 10,
       "minimum_epoch_rule_verified": true
     }
   ],

   "runs": [
     {
       "seed": 0,
+      "checkpoint_sha256": "4794ce39ed5fe621ef8deb7e40900590951f7382e9c7e15543f8d0ef4b41df6b",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
+      "stop_epoch": 14,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 1,
+      "checkpoint_sha256": "0441f00f59c24e9b6efe2e01e71fefc31ca5e4b0da70ed2a5a2051a385782ddc",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
+      "stop_epoch": 12,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 2,
+      "checkpoint_sha256": "1a4cebb05017866daa8f339671599ac21ff3125fcbfac8605845813ef540c007",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
+      "stop_epoch": 16,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 3,
+      "checkpoint_sha256": "1ce28c37f42b247b9bdb51d126bc189161f33d8ff28990b922a3041e5a3ff911",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
+      "stop_epoch": 11,
       "minimum_epoch_rule_verified": true
     },
     {
       "seed": 4,
+      "checkpoint_sha256": "f23ad342a969773bbbcb18619ce8a8843bbb105d94fe64c5a837d68f26829081",
       "checkpoint_bytes": 4198616,
       "all_pretrained_tensors_unchanged": true,
       "independent_scoring_matches": true,
       "min_epochs": 10,
+      "stop_epoch": 11,
       "minimum_epoch_rule_verified": true
     }
   ],

evidence/candidate-uncertainty.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
-  "candidate_seed": 1,
-  "selected_epoch": 14,
   "benchmark_cases_sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
   "resamples": 10000,
   "bootstrap_seed_per_suite": 42,
@@ -10,66 +10,66 @@
   "suites": {
     "typed-decisions": {
       "cases": 400,
-      "accuracy_difference_pp": 40.7,
       "case_bootstrap_95_interval_pp": [
-        37.3,
-        44.101249999999986
       ]
     },
     "ag-news": {
       "cases": 600,
-      "accuracy_difference_pp": -1.6666666666666667,
       "case_bootstrap_95_interval_pp": [
-        -3.5,
-        0.16666666666666666
       ]
     },
     "boolq": {
       "cases": 600,
-      "accuracy_difference_pp": -3.1666666666666665,
       "case_bootstrap_95_interval_pp": [
-        -5.833333333333333,
-        -0.6666666666666666
       ]
     },
     "emotion": {
       "cases": 600,
-      "accuracy_difference_pp": 0.0,
       "case_bootstrap_95_interval_pp": [
-        -2.5,
         2.6666666666666665
       ]
     },
     "prompt-injections": {
       "cases": 116,
-      "accuracy_difference_pp": 2.586206896551724,
       "case_bootstrap_95_interval_pp": [
-        -5.172413793103448,
-        10.344827586206897
       ]
     },
     "sst5": {
       "cases": 600,
-      "accuracy_difference_pp": 5.0,
       "case_bootstrap_95_interval_pp": [
-        1.1666666666666667,
-        9.0
       ]
     },
     "massive-intent.en": {
       "cases": 300,
-      "accuracy_difference_pp": -4.333333333333333,
       "case_bootstrap_95_interval_pp": [
-        -8.333333333333334,
-        -0.3333333333333333
       ]
     },
     "xnli.en": {
       "cases": 300,
-      "accuracy_difference_pp": 1.6666666666666667,
       "case_bootstrap_95_interval_pp": [
-        -1.0,
-        4.666666666666667
       ]
     }
   }

 {
+  "candidate_seed": 4,
+  "selected_epoch": 8,
   "benchmark_cases_sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
   "resamples": 10000,
   "bootstrap_seed_per_suite": 42,
   "suites": {
     "typed-decisions": {
       "cases": 400,
+      "accuracy_difference_pp": 39.2,
       "case_bootstrap_95_interval_pp": [
+        35.9,
+        42.5
       ]
     },
     "ag-news": {
       "cases": 600,
+      "accuracy_difference_pp": 0.3333333333333333,
       "case_bootstrap_95_interval_pp": [
+        -0.5,
+        1.3333333333333333
       ]
     },
     "boolq": {
       "cases": 600,
+      "accuracy_difference_pp": 0.16666666666666666,
       "case_bootstrap_95_interval_pp": [
+        -1.5,
+        1.8333333333333333
       ]
     },
     "emotion": {
       "cases": 600,
+      "accuracy_difference_pp": 0.5,
       "case_bootstrap_95_interval_pp": [
+        -1.5,
         2.6666666666666665
       ]
     },
     "prompt-injections": {
       "cases": 116,
+      "accuracy_difference_pp": 5.172413793103448,
       "case_bootstrap_95_interval_pp": [
+        1.7241379310344827,
+        9.482758620689655
       ]
     },
     "sst5": {
       "cases": 600,
+      "accuracy_difference_pp": 4.0,
       "case_bootstrap_95_interval_pp": [
+        1.0,
+        6.833333333333333
       ]
     },
     "massive-intent.en": {
       "cases": 300,
+      "accuracy_difference_pp": -1.6666666666666667,
       "case_bootstrap_95_interval_pp": [
+        -4.333333333333333,
+        1.0
       ]
     },
     "xnli.en": {
       "cases": 300,
+      "accuracy_difference_pp": 0.3333333333333333,
       "case_bootstrap_95_interval_pp": [
+        -1.6666666666666667,
+        2.3333333333333335
       ]
     }
   }

evidence/lr-comparison.json ADDED Viewed

	@@ -0,0 +1,315 @@

+{
+  "lr_3e4": {
+    "typed-decisions": {
+      "n": 5,
+      "mean_percent": 70.23,
+      "sample_variance_pp_squared": 44.56824999999997,
+      "sample_std_pp": 6.675945625901995
+    },
+    "ag-news": {
+      "n": 5,
+      "mean_percent": 54.56666666666666,
+      "sample_variance_pp_squared": 1255.536111111111,
+      "sample_std_pp": 35.433544997799906
+    },
+    "boolq": {
+      "n": 5,
+      "mean_percent": 63.266666666666666,
+      "sample_variance_pp_squared": 256.79999999999995,
+      "sample_std_pp": 16.0249804992081
+    },
+    "emotion": {
+      "n": 5,
+      "mean_percent": 30.333333333333332,
+      "sample_variance_pp_squared": 616.1666666666666,
+      "sample_std_pp": 24.82270466058577
+    },
+    "prompt-injections": {
+      "n": 5,
+      "mean_percent": 57.758620689655174,
+      "sample_variance_pp_squared": 241.5279429250891,
+      "sample_std_pp": 15.541169290793055
+    },
+    "sst5": {
+      "n": 5,
+      "mean_percent": 30.53333333333333,
+      "sample_variance_pp_squared": 185.14444444444447,
+      "sample_std_pp": 13.606779356057938
+    },
+    "massive-intent.en": {
+      "n": 5,
+      "mean_percent": 32.53333333333333,
+      "sample_variance_pp_squared": 1566.1999999999998,
+      "sample_std_pp": 39.57524478761944
+    },
+    "xnli.en": {
+      "n": 5,
+      "mean_percent": 55.53333333333334,
+      "sample_variance_pp_squared": 860.5333333333334,
+      "sample_std_pp": 29.33484844571953
+    }
+  },
+  "lr_1e4": {
+    "typed-decisions": {
+      "n": 5,
+      "mean_percent": 75.28,
+      "sample_variance_pp_squared": 0.8619999999999957,
+      "sample_std_pp": 0.9284395510748105
+    },
+    "ag-news": {
+      "n": 5,
+      "mean_percent": 94.46666666666667,
+      "sample_variance_pp_squared": 0.3388888888888897,
+      "sample_std_pp": 0.5821416398857667
+    },
+    "boolq": {
+      "n": 5,
+      "mean_percent": 82.10000000000001,
+      "sample_variance_pp_squared": 0.7861111111111168,
+      "sample_std_pp": 0.8866290718846956
+    },
+    "emotion": {
+      "n": 5,
+      "mean_percent": 57.2,
+      "sample_variance_pp_squared": 1.0055555555555546,
+      "sample_std_pp": 1.0027739304327543
+    },
+    "prompt-injections": {
+      "n": 5,
+      "mean_percent": 74.48275862068965,
+      "sample_variance_pp_squared": 15.457788347205712,
+      "sample_std_pp": 3.93163939689358
+    },
+    "sst5": {
+      "n": 5,
+      "mean_percent": 41.0,
+      "sample_variance_pp_squared": 6.027777777777775,
+      "sample_std_pp": 2.4551533104427055
+    },
+    "massive-intent.en": {
+      "n": 5,
+      "mean_percent": 76.8,
+      "sample_variance_pp_squared": 2.422222222222218,
+      "sample_std_pp": 1.556349003990499
+    },
+    "xnli.en": {
+      "n": 5,
+      "mean_percent": 87.26666666666667,
+      "sample_variance_pp_squared": 1.0777777777777784,
+      "sample_std_pp": 1.038160766826496
+    }
+  },
+  "paired_seed_accuracy_differences_pp": {
+    "0": {
+      "typed-decisions": 13.250000000000007,
+      "ag-news": 62.83333333333333,
+      "emotion": 38.99999999999999,
+      "boolq": 24.16666666666667,
+      "sst5": 23.166666666666664,
+      "prompt-injections": 16.37931034482759,
+      "massive-intent.en": 76.66666666666667,
+      "xnli.en": 54.666666666666664
+    },
+    "1": {
+      "typed-decisions": -1.3000000000000012,
+      "ag-news": 1.6666666666666718,
+      "emotion": -0.16666666666667052,
+      "boolq": 2.833333333333332,
+      "sst5": 2.833333333333332,
+      "prompt-injections": 6.896551724137923,
+      "massive-intent.en": 1.333333333333342,
+      "xnli.en": -0.666666666666671
+    },
+    "2": {
+      "typed-decisions": 8.399999999999997,
+      "ag-news": 68.16666666666666,
+      "emotion": 51.16666666666667,
+      "boolq": 32.0,
+      "sst5": 17.166666666666668,
+      "prompt-injections": 37.06896551724138,
+      "massive-intent.en": 68.66666666666667,
+      "xnli.en": 53.0
+    },
+    "3": {
+      "typed-decisions": 5.099999999999993,
+      "ag-news": 65.16666666666666,
+      "emotion": 42.99999999999999,
+      "boolq": 33.166666666666664,
+      "sst5": 15.500000000000004,
+      "prompt-injections": 21.55172413793103,
+      "massive-intent.en": 75.0,
+      "xnli.en": 53.0
+    },
+    "4": {
+      "typed-decisions": -0.20000000000000018,
+      "ag-news": 1.6666666666666607,
+      "emotion": 1.333333333333342,
+      "boolq": 2.0000000000000018,
+      "sst5": -6.333333333333336,
+      "prompt-injections": 1.7241379310344862,
+      "massive-intent.en": -0.33333333333332993,
+      "xnli.en": -1.333333333333342
+    }
+  },
+  "previous_release": {
+    "seed": 1,
+    "epoch": 14,
+    "interface_lr": 0.0003,
+    "suites": {
+      "typed-decisions": {
+        "attempted": 2000,
+        "valid": 2000,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.7695,
+        "accuracy_valid": 0.7695,
+        "ece_top_label": 0.2142237525419539,
+        "mean_confidence": 0.555276247458046,
+        "brier_hard": 0.3977842163576973,
+        "brier_hard_n": 2000,
+        "nll_hard": 0.7027635604099817,
+        "nll_hard_n": 2000,
+        "zero_probability_gold": 0.0,
+        "zero_probability_gold_n": 2000,
+        "soft_accuracy": 0.47103605907759827,
+        "soft_accuracy_n": 2000,
+        "brier_soft": 0.06255403710877039,
+        "brier_soft_n": 2000,
+        "kl_gold_to_prediction": 0.11643450461088396,
+        "kl_gold_to_prediction_n": 2000,
+        "total_variation": 0.17146599324503523,
+        "total_variation_n": 2000,
+        "score_mae": 0.2299944493828981,
+        "score_mae_n": 800,
+        "within_one_level": 0.98875,
+        "within_one_level_n": 800,
+        "macro_f1": 0.6487623885802034
+      },
+      "ag-news": {
+        "attempted": 600,
+        "valid": 600,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.9316666666666666,
+        "accuracy_valid": 0.9316666666666666,
+        "ece_top_label": 0.03387674147756575,
+        "mean_confidence": 0.904708562141314,
+        "brier_hard": 0.10911433540423408,
+        "brier_hard_n": 600,
+        "nll_hard": 0.21361121678114303,
+        "nll_hard_n": 600,
+        "zero_probability_gold": 0.0,
+        "zero_probability_gold_n": 600,
+        "macro_f1": 0.9283987145646934
+      },
+      "emotion": {
+        "attempted": 600,
+        "valid": 600,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.5733333333333334,
+        "accuracy_valid": 0.5733333333333334,
+        "ece_top_label": 0.3169210441851814,
+        "mean_confidence": 0.8880587177380197,
+        "brier_hard": 0.7264584486728032,
+        "brier_hard_n": 600,
+        "nll_hard": 2.146154603203373,
+        "nll_hard_n": 600,
+        "zero_probability_gold": 0.008333333333333333,
+        "zero_probability_gold_n": 600,
+        "macro_f1": 0.4862286665693279
+      },
+      "boolq": {
+        "attempted": 600,
+        "valid": 600,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.7983333333333333,
+        "accuracy_valid": 0.7983333333333333,
+        "ece_top_label": 0.09755433333333334,
+        "mean_confidence": 0.8900093333333333,
+        "brier_hard": 0.29822958146666667,
+        "brier_hard_n": 600,
+        "nll_hard": 0.4711253960780479,
+        "nll_hard_n": 600,
+        "zero_probability_gold": 0.0,
+        "zero_probability_gold_n": 600,
+        "macro_f1": 0.7813983878883868
+      },
+      "sst5": {
+        "attempted": 600,
+        "valid": 600,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.42,
+        "accuracy_valid": 0.42,
+        "ece_top_label": 0.1643168667044709,
+        "mean_confidence": 0.5761336745094923,
+        "brier_hard": 0.722245716690302,
+        "brier_hard_n": 600,
+        "nll_hard": 1.412054753482227,
+        "nll_hard_n": 600,
+        "zero_probability_gold": 0.0,
+        "zero_probability_gold_n": 600,
+        "score_mae": 0.7360310433394476,
+        "score_mae_n": 600,
+        "within_one_level": 0.7466666666666667,
+        "within_one_level_n": 600,
+        "macro_f1": 0.41387920942262574
+      },
+      "prompt-injections": {
+        "attempted": 116,
+        "valid": 116,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.7155172413793104,
+        "accuracy_valid": 0.7155172413793104,
+        "ece_top_label": 0.20010517241379305,
+        "mean_confidence": 0.9105,
+        "brier_hard": 0.43610344172413795,
+        "brier_hard_n": 116,
+        "nll_hard": 1.2355666268201304,
+        "nll_hard_n": 116,
+        "zero_probability_gold": 0.008620689655172414,
+        "zero_probability_gold_n": 116,
+        "macro_f1": 0.7038756091900673
+      },
+      "massive-intent.en": {
+        "attempted": 300,
+        "valid": 300,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.7433333333333333,
+        "accuracy_valid": 0.7433333333333333,
+        "ece_top_label": 0.1906540683935104,
+        "mean_confidence": 0.9339874017268438,
+        "brier_hard": 0.43706654758878133,
+        "brier_hard_n": 300,
+        "nll_hard": 2.7801978001907903,
+        "nll_hard_n": 300,
+        "zero_probability_gold": 0.07,
+        "zero_probability_gold_n": 300,
+        "macro_f1": 0.44346732036749054
+      },
+      "xnli.en": {
+        "attempted": 300,
+        "valid": 300,
+        "failed": 0,
+        "coverage": 1.0,
+        "accuracy_all": 0.8766666666666667,
+        "accuracy_valid": 0.8766666666666667,
+        "ece_top_label": 0.055158283453726184,
+        "mean_confidence": 0.913468962679153,
+        "brier_hard": 0.19180470102420566,
+        "brier_hard_n": 300,
+        "nll_hard": 0.3708980570966955,
+        "nll_hard_n": 300,
+        "zero_probability_gold": 0.0,
+        "zero_probability_gold_n": 300,
+        "macro_f1": 0.8772028178860477
+      }
+    },
+    "repository_revision": "eaea15162eac93240e5d70f263920eea3093e6c2"
+  },
+  "scope": "adaptive follow-up on a repeatedly inspected benchmark bundle"
+}

evidence/replay-check.json CHANGED Viewed

@@ -1,15 +1,17 @@
 {
   "passed": true,
   "seed": 0,
   "separate_processes": true,
   "epochs_per_process": 2,
   "optimizer_updates_per_process": 338,
   "initial_validation_exact": true,
   "epoch_losses_and_metrics_exact": true,
   "selected_checkpoint_tensors_bitwise_equal": true,
   "checkpoint_sha256": [
-    "3a30ce9881420cebc61829e52b41a9e669a114b4dffe0f8f836831572bd70331",
-    "3a30ce9881420cebc61829e52b41a9e669a114b4dffe0f8f836831572bd70331"
   ],
   "reproducibility": {
     "deterministic_algorithms": true,
@@ -28,5 +30,6 @@
     "cudnn_version": 92000,
     "scope": "fixed hardware/runtime; cross-platform bitwise agreement is not promised"
   },
-  "limit": "replay verified for this data, device, runtime, and two-epoch prefix"
 }

 {
   "passed": true,
   "seed": 0,
+  "interface_lr": 0.0001,
   "separate_processes": true,
   "epochs_per_process": 2,
   "optimizer_updates_per_process": 338,
   "initial_validation_exact": true,
   "epoch_losses_and_metrics_exact": true,
   "selected_checkpoint_tensors_bitwise_equal": true,
+  "reported_seed_0_two_epoch_metrics_exact": true,
   "checkpoint_sha256": [
+    "3c52574d070140a55de8929922302c193d4493f76f35a502edf419ee5005881e",
+    "3c52574d070140a55de8929922302c193d4493f76f35a502edf419ee5005881e"
   ],
   "reproducibility": {
     "deterministic_algorithms": true,
     "cudnn_version": 92000,
     "scope": "fixed hardware/runtime; cross-platform bitwise agreement is not promised"
   },
+  "timing": "verification after the completed five-seed sweep",
+  "limit": "same runtime, hardware, data and two-epoch prefix; not a full-trajectory or cross-platform replay"
 }

evidence/seed-0-history.json CHANGED Viewed

@@ -9,399 +9,246 @@
   "epochs": [
     {
       "epoch": 1,
-      "train_soft_cross_entropy": 1.1371864740936843,
       "validation": {
-        "soft_cross_entropy": 1.195652896563212,
-        "accuracy": 0.3433333333333333,
-        "brier_soft": 0.2543176457285881,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.195652896563212,
         "improved": true
       }
     },
     {
       "epoch": 2,
-      "train_soft_cross_entropy": 1.1867035720966481,
       "validation": {
-        "soft_cross_entropy": 1.1503166087468466,
-        "accuracy": 0.43666666666666665,
-        "brier_soft": 0.2273359453678131,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.1503166087468466,
         "improved": true
       }
     },
     {
       "epoch": 3,
-      "train_soft_cross_entropy": 1.1479326088340194,
       "validation": {
-        "soft_cross_entropy": 1.1359616955121359,
-        "accuracy": 0.4583333333333333,
-        "brier_soft": 0.21933125893274943,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.1359616955121359,
         "improved": true
       }
     },
     {
       "epoch": 4,
-      "train_soft_cross_entropy": 1.17192115077266,
       "validation": {
-        "soft_cross_entropy": 1.1432366434733072,
-        "accuracy": 0.4533333333333333,
-        "brier_soft": 0.22181885927915573,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 1.1359616955121359,
         "improved": false
       }
     },
     {
       "epoch": 5,
-      "train_soft_cross_entropy": 1.1521123818114951,
       "validation": {
-        "soft_cross_entropy": 1.1532048479715984,
-        "accuracy": 0.42,
-        "brier_soft": 0.22427557816108068,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 1.1359616955121359,
-        "improved": false
       }
     },
     {
       "epoch": 6,
-      "train_soft_cross_entropy": 1.139600450904281,
       "validation": {
-        "soft_cross_entropy": 1.1171702599525453,
-        "accuracy": 0.465,
-        "brier_soft": 0.21267332146565118,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.1171702599525453,
         "improved": true
       }
     },
     {
       "epoch": 7,
-      "train_soft_cross_entropy": 1.1084874327094467,
       "validation": {
-        "soft_cross_entropy": 1.101010274887085,
-        "accuracy": 0.48333333333333334,
-        "brier_soft": 0.20341493268807728,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.101010274887085,
         "improved": true
       }
     },
     {
       "epoch": 8,
-      "train_soft_cross_entropy": 1.0972168815577472,
-      "validation": {
-        "soft_cross_entropy": 1.0927943936983744,
-        "accuracy": 0.4866666666666667,
-        "brier_soft": 0.20212342927853266,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0927943936983744,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 9,
-      "train_soft_cross_entropy": 1.0959268622045164,
       "validation": {
-        "soft_cross_entropy": 1.0886747964223227,
-        "accuracy": 0.485,
-        "brier_soft": 0.1981955200433731,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0886747964223227,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 10,
-      "train_soft_cross_entropy": 1.0879168816849036,
-      "validation": {
-        "soft_cross_entropy": 1.0776302735010783,
-        "accuracy": 0.5133333333333333,
-        "brier_soft": 0.19176260660092037,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0776302735010783,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 11,
-      "train_soft_cross_entropy": 1.0794475747037817,
-      "validation": {
-        "soft_cross_entropy": 1.0989113728205362,
-        "accuracy": 0.49666666666666665,
-        "brier_soft": 0.20668872609734534,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 1.0776302735010783,
         "improved": false
       }
     },
     {
-      "epoch": 12,
-      "train_soft_cross_entropy": 1.0683085640271506,
-      "validation": {
-        "soft_cross_entropy": 1.0758668931325277,
-        "accuracy": 0.5016666666666667,
-        "brier_soft": 0.19317058285077413,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0758668931325277,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 13,
-      "train_soft_cross_entropy": 1.052984880871243,
-      "validation": {
-        "soft_cross_entropy": 1.0600487383206685,
-        "accuracy": 0.5316666666666666,
-        "brier_soft": 0.18257314254840215,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0600487383206685,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 14,
-      "train_soft_cross_entropy": 1.0401919462062694,
-      "validation": {
-        "soft_cross_entropy": 1.0525298221906025,
-        "accuracy": 0.5366666666666666,
-        "brier_soft": 0.1811353324353695,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0525298221906025,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 15,
-      "train_soft_cross_entropy": 1.0262903751267327,
-      "validation": {
-        "soft_cross_entropy": 1.0581993921597799,
-        "accuracy": 0.5233333333333333,
-        "brier_soft": 0.18550985043247542,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 1.0525298221906025,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 16,
-      "train_soft_cross_entropy": 1.0127459985238534,
-      "validation": {
-        "soft_cross_entropy": 1.0545752588907877,
-        "accuracy": 0.54,
-        "brier_soft": 0.18113683501879374,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 1.0525298221906025,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 17,
-      "train_soft_cross_entropy": 1.0096984642523306,
-      "validation": {
-        "soft_cross_entropy": 1.0468252456188203,
-        "accuracy": 0.5483333333333333,
-        "brier_soft": 0.17509076982736588,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0468252456188203,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 18,
-      "train_soft_cross_entropy": 1.0022709012914588,
       "validation": {
-        "soft_cross_entropy": 1.0376164364814757,
-        "accuracy": 0.5766666666666667,
-        "brier_soft": 0.17226109830041728,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0376164364814757,
         "improved": true
       }
     },
     {
-      "epoch": 19,
-      "train_soft_cross_entropy": 0.9934269437083492,
       "validation": {
-        "soft_cross_entropy": 1.0526774628957112,
-        "accuracy": 0.5516666666666666,
-        "brier_soft": 0.17908830145994822,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 1.0376164364814757,
         "improved": false
       }
     },
     {
-      "epoch": 20,
-      "train_soft_cross_entropy": 0.9915926403911025,
       "validation": {
-        "soft_cross_entropy": 1.0292588464419048,
-        "accuracy": 0.57,
-        "brier_soft": 0.16918850486477216,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0292588464419048,
         "improved": true
       }
     },
     {
-      "epoch": 21,
-      "train_soft_cross_entropy": 0.9833765982698511,
       "validation": {
-        "soft_cross_entropy": 1.0493251442909242,
-        "accuracy": 0.555,
-        "brier_soft": 0.17917159788310527,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 1.0292588464419048,
         "improved": false
       }
     },
     {
-      "epoch": 22,
-      "train_soft_cross_entropy": 0.9786927858988445,
       "validation": {
-        "soft_cross_entropy": 1.0412022503217062,
-        "accuracy": 0.555,
-        "brier_soft": 0.17517984464764594,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
-        "best_validation_loss": 1.0292588464419048,
         "improved": false
       }
     },
     {
-      "epoch": 23,
-      "train_soft_cross_entropy": 0.9695542351404826,
       "validation": {
-        "soft_cross_entropy": 1.0376375365257262,
-        "accuracy": 0.5583333333333333,
-        "brier_soft": 0.17371407074232897,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
-        "best_validation_loss": 1.0292588464419048,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
-    "epochs_completed": 23,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

   "epochs": [
     {
       "epoch": 1,
+      "train_soft_cross_entropy": 1.087307652897305,
       "validation": {
+        "soft_cross_entropy": 1.0358176565170287,
+        "accuracy": 0.5516666666666666,
+        "brier_soft": 0.17500559210777283,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 1.0358176565170287,
         "improved": true
       }
     },
     {
       "epoch": 2,
+      "train_soft_cross_entropy": 0.9998172007666694,
       "validation": {
+        "soft_cross_entropy": 1.013402551015218,
+        "accuracy": 0.6066666666666667,
+        "brier_soft": 0.16418362177908422,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 1.013402551015218,
         "improved": true
       }
     },
     {
       "epoch": 3,
+      "train_soft_cross_entropy": 0.961990624445456,
       "validation": {
+        "soft_cross_entropy": 0.9978651634852092,
+        "accuracy": 0.6116666666666667,
+        "brier_soft": 0.15542398323615392,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9978651634852092,
         "improved": true
       }
     },
     {
       "epoch": 4,
+      "train_soft_cross_entropy": 0.9616141191235295,
       "validation": {
+        "soft_cross_entropy": 1.1574800237019858,
+        "accuracy": 0.48,
+        "brier_soft": 0.23033222297827402,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9978651634852092,
         "improved": false
       }
     },
     {
       "epoch": 5,
+      "train_soft_cross_entropy": 0.9278180778468097,
       "validation": {
+        "soft_cross_entropy": 0.9557415008544922,
+        "accuracy": 0.6833333333333333,
+        "brier_soft": 0.12863569288204113,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9557415008544922,
+        "improved": true
       }
     },
     {
       "epoch": 6,
+      "train_soft_cross_entropy": 0.8944051215383741,
       "validation": {
+        "soft_cross_entropy": 0.9157323372364045,
+        "accuracy": 0.6916666666666667,
+        "brier_soft": 0.1038101188838482,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9157323372364045,
         "improved": true
       }
     },
     {
       "epoch": 7,
+      "train_soft_cross_entropy": 0.858607970961818,
       "validation": {
+        "soft_cross_entropy": 0.9070962381362915,
+        "accuracy": 0.7016666666666667,
+        "brier_soft": 0.09971169379850228,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9070962381362915,
         "improved": true
       }
     },
     {
       "epoch": 8,
+      "train_soft_cross_entropy": 0.8490360851199539,
       "validation": {
+        "soft_cross_entropy": 0.9083957560857137,
+        "accuracy": 0.7116666666666667,
+        "brier_soft": 0.10123874122897784,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.9070962381362915,
         "improved": false
       }
     },
     {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.8271938294393045,
       "validation": {
+        "soft_cross_entropy": 0.8859062685569128,
+        "accuracy": 0.7233333333333334,
+        "brier_soft": 0.08866231886049111,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8859062685569128,
         "improved": true
       }
     },
     {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.8117916690861737,
       "validation": {
+        "soft_cross_entropy": 0.8980106647809346,
+        "accuracy": 0.72,
+        "brier_soft": 0.09496299955993891,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8859062685569128,
         "improved": false
       }
     },
     {
+      "epoch": 11,
+      "train_soft_cross_entropy": 0.8015254776124601,
       "validation": {
+        "soft_cross_entropy": 0.8771124110619227,
+        "accuracy": 0.7116666666666667,
+        "brier_soft": 0.0828241604194045,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8771124110619227,
         "improved": true
       }
     },
     {
+      "epoch": 12,
+      "train_soft_cross_entropy": 0.7930714415620874,
       "validation": {
+        "soft_cross_entropy": 0.8903550871213277,
+        "accuracy": 0.7233333333333334,
+        "brier_soft": 0.09005908486122886,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8771124110619227,
         "improved": false
       }
     },
     {
+      "epoch": 13,
+      "train_soft_cross_entropy": 0.7864397717405248,
       "validation": {
+        "soft_cross_entropy": 0.8776129017273585,
+        "accuracy": 0.7283333333333334,
+        "brier_soft": 0.08296463950847587,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8771124110619227,
         "improved": false
       }
     },
     {
+      "epoch": 14,
+      "train_soft_cross_entropy": 0.7832702258339634,
       "validation": {
+        "soft_cross_entropy": 0.8776572781801224,
+        "accuracy": 0.7466666666666667,
+        "brier_soft": 0.08209236452666421,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8771124110619227,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
+    "epochs_completed": 14,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

evidence/seed-1-history.json CHANGED Viewed

@@ -9,297 +9,212 @@
   "epochs": [
     {
       "epoch": 1,
-      "train_soft_cross_entropy": 1.0002122403074194,
       "validation": {
-        "soft_cross_entropy": 0.9555757478872935,
-        "accuracy": 0.665,
-        "brier_soft": 0.13395084381103517,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9555757478872935,
         "improved": true
       }
     },
     {
       "epoch": 2,
-      "train_soft_cross_entropy": 0.895550852616628,
       "validation": {
-        "soft_cross_entropy": 0.8875669745604197,
-        "accuracy": 0.7216666666666667,
-        "brier_soft": 0.08987226173281669,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8875669745604197,
         "improved": true
       }
     },
     {
       "epoch": 3,
-      "train_soft_cross_entropy": 0.8399523215382187,
       "validation": {
-        "soft_cross_entropy": 0.8776892550786336,
-        "accuracy": 0.725,
-        "brier_soft": 0.0829127719004949,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8776892550786336,
         "improved": true
       }
     },
     {
       "epoch": 4,
-      "train_soft_cross_entropy": 0.8150597869908368,
       "validation": {
-        "soft_cross_entropy": 0.8523939752578735,
-        "accuracy": 0.7666666666666667,
-        "brier_soft": 0.07098765720923741,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8523939752578735,
         "improved": true
       }
     },
     {
       "epoch": 5,
-      "train_soft_cross_entropy": 0.7998251422246297,
       "validation": {
-        "soft_cross_entropy": 0.853373521566391,
-        "accuracy": 0.745,
-        "brier_soft": 0.07141184127579132,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 0.8523939752578735,
-        "improved": false
       }
     },
     {
       "epoch": 6,
-      "train_soft_cross_entropy": 0.7924317064991704,
       "validation": {
-        "soft_cross_entropy": 0.8612222145001094,
-        "accuracy": 0.7516666666666667,
-        "brier_soft": 0.07590873730679353,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 0.8523939752578735,
         "improved": false
       }
     },
     {
       "epoch": 7,
-      "train_soft_cross_entropy": 0.7880261039733887,
       "validation": {
-        "soft_cross_entropy": 0.8462035346031189,
         "accuracy": 0.7466666666666667,
-        "brier_soft": 0.06817000946650903,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8462035346031189,
-        "improved": true
       }
     },
     {
       "epoch": 8,
-      "train_soft_cross_entropy": 0.7816576555923179,
       "validation": {
-        "soft_cross_entropy": 0.8440792632102966,
-        "accuracy": 0.7633333333333333,
-        "brier_soft": 0.06713677939027547,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8440792632102966,
-        "improved": true
       }
     },
     {
       "epoch": 9,
-      "train_soft_cross_entropy": 0.7783506816404837,
       "validation": {
-        "soft_cross_entropy": 0.8438087296485901,
-        "accuracy": 0.7683333333333333,
-        "brier_soft": 0.06672837336858113,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8438087296485901,
         "improved": true
       }
     },
     {
       "epoch": 10,
-      "train_soft_cross_entropy": 0.7754799175262451,
       "validation": {
-        "soft_cross_entropy": 0.8468067542711893,
-        "accuracy": 0.755,
-        "brier_soft": 0.06876023932515334,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 0.8438087296485901,
         "improved": false
       }
     },
     {
       "epoch": 11,
-      "train_soft_cross_entropy": 0.7734219932114637,
       "validation": {
-        "soft_cross_entropy": 0.8521546524763107,
-        "accuracy": 0.7783333333333333,
-        "brier_soft": 0.07055049358556668,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
-        "best_validation_loss": 0.8438087296485901,
         "improved": false
       }
     },
     {
       "epoch": 12,
-      "train_soft_cross_entropy": 0.7719193545094243,
-      "validation": {
-        "soft_cross_entropy": 0.842797059615453,
-        "accuracy": 0.7766666666666666,
-        "brier_soft": 0.06609537469533583,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.842797059615453,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 13,
-      "train_soft_cross_entropy": 0.7722371352601934,
-      "validation": {
-        "soft_cross_entropy": 0.848033101161321,
-        "accuracy": 0.7633333333333333,
-        "brier_soft": 0.06775808438037832,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 0.842797059615453,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 14,
-      "train_soft_cross_entropy": 0.7689926949695305,
-      "validation": {
-        "soft_cross_entropy": 0.8377540612220764,
-        "accuracy": 0.765,
-        "brier_soft": 0.06276483290052662,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8377540612220764,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 15,
-      "train_soft_cross_entropy": 0.7658979249000549,
-      "validation": {
-        "soft_cross_entropy": 0.8519912085930507,
-        "accuracy": 0.765,
-        "brier_soft": 0.07007166295622785,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 0.8377540612220764,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 16,
-      "train_soft_cross_entropy": 0.7651108237107594,
-      "validation": {
-        "soft_cross_entropy": 0.8408213845888773,
-        "accuracy": 0.7633333333333333,
-        "brier_soft": 0.06544813718336324,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 0.8377540612220764,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 17,
-      "train_soft_cross_entropy": 0.7817914198063038,
       "validation": {
-        "soft_cross_entropy": 0.8468167595068614,
-        "accuracy": 0.7683333333333333,
-        "brier_soft": 0.06789324807934463,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
-        "best_validation_loss": 0.8377540612220764,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
-    "epochs_completed": 17,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

   "epochs": [
     {
       "epoch": 1,
+      "train_soft_cross_entropy": 1.0518495603843971,
       "validation": {
+        "soft_cross_entropy": 0.9761401589711507,
+        "accuracy": 0.6666666666666666,
+        "brier_soft": 0.14186986642579238,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9761401589711507,
         "improved": true
       }
     },
     {
       "epoch": 2,
+      "train_soft_cross_entropy": 0.912936895776678,
       "validation": {
+        "soft_cross_entropy": 0.9058315515518188,
+        "accuracy": 0.7133333333333334,
+        "brier_soft": 0.09862347106138865,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9058315515518188,
         "improved": true
       }
     },
     {
       "epoch": 3,
+      "train_soft_cross_entropy": 0.8595328189708569,
       "validation": {
+        "soft_cross_entropy": 0.8869859429200491,
+        "accuracy": 0.7333333333333333,
+        "brier_soft": 0.08666441923628251,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8869859429200491,
         "improved": true
       }
     },
     {
       "epoch": 4,
+      "train_soft_cross_entropy": 0.8311581687574033,
       "validation": {
+        "soft_cross_entropy": 0.875844070315361,
+        "accuracy": 0.7466666666666667,
+        "brier_soft": 0.08233745716512203,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.875844070315361,
         "improved": true
       }
     },
     {
       "epoch": 5,
+      "train_soft_cross_entropy": 0.812676587899526,
       "validation": {
+        "soft_cross_entropy": 0.8657626557350159,
+        "accuracy": 0.74,
+        "brier_soft": 0.07910436561020712,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8657626557350159,
+        "improved": true
       }
     },
     {
       "epoch": 6,
+      "train_soft_cross_entropy": 0.7997228077605919,
       "validation": {
+        "soft_cross_entropy": 0.8763724052906037,
+        "accuracy": 0.7433333333333333,
+        "brier_soft": 0.08383749471356472,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8657626557350159,
         "improved": false
       }
     },
     {
       "epoch": 7,
+      "train_soft_cross_entropy": 0.7919920165892,
       "validation": {
+        "soft_cross_entropy": 0.8720610602696737,
         "accuracy": 0.7466666666666667,
+        "brier_soft": 0.08182110945383708,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8657626557350159,
+        "improved": false
       }
     },
     {
       "epoch": 8,
+      "train_soft_cross_entropy": 0.7865607251061334,
       "validation": {
+        "soft_cross_entropy": 0.8708612742026647,
+        "accuracy": 0.735,
+        "brier_soft": 0.08140341289962331,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8657626557350159,
+        "improved": false
       }
     },
     {
       "epoch": 9,
+      "train_soft_cross_entropy": 0.781722169496395,
       "validation": {
+        "soft_cross_entropy": 0.8583925066391627,
+        "accuracy": 0.7466666666666667,
+        "brier_soft": 0.07437195796364297,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8583925066391627,
         "improved": true
       }
     },
     {
       "epoch": 10,
+      "train_soft_cross_entropy": 0.7776996397530591,
       "validation": {
+        "soft_cross_entropy": 0.8674025861422221,
+        "accuracy": 0.7433333333333333,
+        "brier_soft": 0.07951540602991979,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8583925066391627,
         "improved": false
       }
     },
     {
       "epoch": 11,
+      "train_soft_cross_entropy": 0.7751605097894315,
       "validation": {
+        "soft_cross_entropy": 0.86817025522391,
+        "accuracy": 0.7433333333333333,
+        "brier_soft": 0.07928484301393231,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8583925066391627,
         "improved": false
       }
     },
     {
       "epoch": 12,
+      "train_soft_cross_entropy": 0.7721157057638521,
       "validation": {
+        "soft_cross_entropy": 0.8638703493277232,
+        "accuracy": 0.7583333333333333,
+        "brier_soft": 0.0763223198801279,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8583925066391627,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
+    "epochs_completed": 12,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

evidence/seed-2-history.json CHANGED Viewed

@@ -9,314 +9,280 @@
   "epochs": [
     {
       "epoch": 1,
-      "train_soft_cross_entropy": 1.1849676999339351,
       "validation": {
-        "soft_cross_entropy": 1.1535689290364584,
-        "accuracy": 0.4266666666666667,
-        "brier_soft": 0.2362905572851499,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.1535689290364584,
         "improved": true
       }
     },
     {
       "epoch": 2,
-      "train_soft_cross_entropy": 1.1660638781830117,
       "validation": {
-        "soft_cross_entropy": 1.1240008862813313,
-        "accuracy": 0.44666666666666666,
-        "brier_soft": 0.2163119477033615,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.1240008862813313,
         "improved": true
       }
     },
     {
       "epoch": 3,
-      "train_soft_cross_entropy": 1.1152002820262203,
       "validation": {
-        "soft_cross_entropy": 1.111793461640676,
-        "accuracy": 0.4816666666666667,
-        "brier_soft": 0.21397013902664186,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.111793461640676,
         "improved": true
       }
     },
     {
       "epoch": 4,
-      "train_soft_cross_entropy": 1.0905909954177009,
       "validation": {
-        "soft_cross_entropy": 1.0822320612271628,
-        "accuracy": 0.49833333333333335,
-        "brier_soft": 0.19861485362052916,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0822320612271628,
         "improved": true
       }
     },
     {
       "epoch": 5,
-      "train_soft_cross_entropy": 1.0488388820047732,
       "validation": {
-        "soft_cross_entropy": 1.0466791025797526,
-        "accuracy": 0.5383333333333333,
-        "brier_soft": 0.18209601615866025,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0466791025797526,
-        "improved": true
       }
     },
     {
       "epoch": 6,
-      "train_soft_cross_entropy": 1.0179926924352292,
       "validation": {
-        "soft_cross_entropy": 1.0302869494756062,
-        "accuracy": 0.5633333333333334,
-        "brier_soft": 0.1717192947367827,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0302869494756062,
         "improved": true
       }
     },
     {
       "epoch": 7,
-      "train_soft_cross_entropy": 1.0011521806540313,
       "validation": {
-        "soft_cross_entropy": 1.0306965279579163,
-        "accuracy": 0.5533333333333333,
-        "brier_soft": 0.17337830337385338,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 1.0302869494756062,
-        "improved": false
       }
     },
     {
       "epoch": 8,
-      "train_soft_cross_entropy": 0.9898686430189345,
       "validation": {
-        "soft_cross_entropy": 1.0352703229586284,
-        "accuracy": 0.5533333333333333,
-        "brier_soft": 0.17590213686227799,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 1.0302869494756062,
-        "improved": false
       }
     },
     {
       "epoch": 9,
-      "train_soft_cross_entropy": 0.9732429999775357,
       "validation": {
-        "soft_cross_entropy": 1.0375931040445963,
-        "accuracy": 0.56,
-        "brier_soft": 0.1798627228786548,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 3,
-        "best_validation_loss": 1.0302869494756062,
         "improved": false
       }
     },
     {
       "epoch": 10,
-      "train_soft_cross_entropy": 0.9616305458987201,
       "validation": {
-        "soft_cross_entropy": 1.0139233843485513,
-        "accuracy": 0.6116666666666667,
-        "brier_soft": 0.16307140870640674,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0139233843485513,
         "improved": true
       }
     },
     {
       "epoch": 11,
-      "train_soft_cross_entropy": 0.9440543747831274,
-      "validation": {
-        "soft_cross_entropy": 1.0138032054901123,
-        "accuracy": 0.6166666666666667,
-        "brier_soft": 0.165894419302543,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0138032054901123,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 12,
-      "train_soft_cross_entropy": 0.9309376364284091,
       "validation": {
-        "soft_cross_entropy": 0.9929217569033305,
-        "accuracy": 0.6266666666666667,
-        "brier_soft": 0.15395031906664372,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9929217569033305,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 13,
-      "train_soft_cross_entropy": 0.9144250082969666,
-      "validation": {
-        "soft_cross_entropy": 0.9964303588867187,
-        "accuracy": 0.6266666666666667,
-        "brier_soft": 0.1560174826408426,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 0.9929217569033305,
         "improved": false
       }
     },
     {
-      "epoch": 14,
-      "train_soft_cross_entropy": 0.9063218560925237,
       "validation": {
-        "soft_cross_entropy": 1.005851571559906,
-        "accuracy": 0.6133333333333333,
-        "brier_soft": 0.1608497215807438,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
-        "best_validation_loss": 0.9929217569033305,
         "improved": false
       }
     },
     {
-      "epoch": 15,
-      "train_soft_cross_entropy": 0.9014872776137458,
       "validation": {
-        "soft_cross_entropy": 0.9867499772707621,
-        "accuracy": 0.6316666666666667,
-        "brier_soft": 0.14927835414807003,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9867499772707621,
         "improved": true
       }
     },
     {
-      "epoch": 16,
-      "train_soft_cross_entropy": 0.8920391595805133,
       "validation": {
-        "soft_cross_entropy": 0.9926939264933268,
-        "accuracy": 0.6283333333333333,
-        "brier_soft": 0.15533705055713654,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 0.9867499772707621,
         "improved": false
       }
     },
     {
-      "epoch": 17,
-      "train_soft_cross_entropy": 0.8795346831833875,
       "validation": {
-        "soft_cross_entropy": 1.02633438428243,
-        "accuracy": 0.6583333333333333,
-        "brier_soft": 0.16615503872434298,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
-        "best_validation_loss": 0.9867499772707621,
         "improved": false
       }
     },
     {
-      "epoch": 18,
-      "train_soft_cross_entropy": 0.879185232453876,
       "validation": {
-        "soft_cross_entropy": 1.0007199597358705,
-        "accuracy": 0.65,
-        "brier_soft": 0.15312831775595745,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
-        "best_validation_loss": 0.9867499772707621,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
-    "epochs_completed": 18,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

   "epochs": [
     {
       "epoch": 1,
+      "train_soft_cross_entropy": 1.0231783160456904,
       "validation": {
+        "soft_cross_entropy": 0.9674482258160909,
+        "accuracy": 0.6516666666666666,
+        "brier_soft": 0.139310811907053,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9674482258160909,
         "improved": true
       }
     },
     {
       "epoch": 2,
+      "train_soft_cross_entropy": 0.8996694993972778,
       "validation": {
+        "soft_cross_entropy": 0.8974683928489685,
+        "accuracy": 0.75,
+        "brier_soft": 0.09312087532132864,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8974683928489685,
         "improved": true
       }
     },
     {
       "epoch": 3,
+      "train_soft_cross_entropy": 0.8483260071277618,
       "validation": {
+        "soft_cross_entropy": 0.8809152638912201,
+        "accuracy": 0.7433333333333333,
+        "brier_soft": 0.08576706000914176,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8809152638912201,
         "improved": true
       }
     },
     {
       "epoch": 4,
+      "train_soft_cross_entropy": 0.8239163917523843,
       "validation": {
+        "soft_cross_entropy": 0.8699432893594106,
+        "accuracy": 0.75,
+        "brier_soft": 0.07744032381723324,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8699432893594106,
         "improved": true
       }
     },
     {
       "epoch": 5,
+      "train_soft_cross_entropy": 0.8087966615623898,
       "validation": {
+        "soft_cross_entropy": 0.8749503823121388,
+        "accuracy": 0.725,
+        "brier_soft": 0.08202173060427109,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8699432893594106,
+        "improved": false
       }
     },
     {
       "epoch": 6,
+      "train_soft_cross_entropy": 0.798634327782525,
       "validation": {
+        "soft_cross_entropy": 0.8646231130758921,
+        "accuracy": 0.7633333333333333,
+        "brier_soft": 0.07594866358985504,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8646231130758921,
         "improved": true
       }
     },
     {
       "epoch": 7,
+      "train_soft_cross_entropy": 0.7900045536182545,
       "validation": {
+        "soft_cross_entropy": 0.8598153670628865,
+        "accuracy": 0.755,
+        "brier_soft": 0.07338624291742842,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8598153670628865,
+        "improved": true
       }
     },
     {
       "epoch": 8,
+      "train_soft_cross_entropy": 0.7848307268266325,
       "validation": {
+        "soft_cross_entropy": 0.8584532356262207,
+        "accuracy": 0.7483333333333333,
+        "brier_soft": 0.07417237816999356,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8584532356262207,
+        "improved": true
       }
     },
     {
       "epoch": 9,
+      "train_soft_cross_entropy": 0.7802449210043306,
       "validation": {
+        "soft_cross_entropy": 0.863755419254303,
+        "accuracy": 0.7433333333333333,
+        "brier_soft": 0.07594012685120105,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8584532356262207,
         "improved": false
       }
     },
     {
       "epoch": 10,
+      "train_soft_cross_entropy": 0.777841743098365,
       "validation": {
+        "soft_cross_entropy": 0.8546390326817831,
+        "accuracy": 0.7566666666666667,
+        "brier_soft": 0.07055099235226711,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8546390326817831,
         "improved": true
       }
     },
     {
       "epoch": 11,
+      "train_soft_cross_entropy": 0.7738726913045954,
       "validation": {
+        "soft_cross_entropy": 0.8676082201798757,
+        "accuracy": 0.7466666666666667,
+        "brier_soft": 0.07854201994836331,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8546390326817831,
         "improved": false
       }
     },
     {
+      "epoch": 12,
+      "train_soft_cross_entropy": 0.7718432058228387,
       "validation": {
+        "soft_cross_entropy": 0.8577940479914348,
+        "accuracy": 0.7366666666666667,
+        "brier_soft": 0.07410156331335505,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8546390326817831,
         "improved": false
       }
     },
     {
+      "epoch": 13,
+      "train_soft_cross_entropy": 0.7713618384467231,
       "validation": {
+        "soft_cross_entropy": 0.8543428432941437,
+        "accuracy": 0.7583333333333333,
+        "brier_soft": 0.07187043125430743,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8543428432941437,
         "improved": true
       }
     },
     {
+      "epoch": 14,
+      "train_soft_cross_entropy": 0.7684376273331819,
       "validation": {
+        "soft_cross_entropy": 0.8578398124376932,
+        "accuracy": 0.7483333333333333,
+        "brier_soft": 0.07348543658852577,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8543428432941437,
         "improved": false
       }
     },
     {
+      "epoch": 15,
+      "train_soft_cross_entropy": 0.766730530526903,
       "validation": {
+        "soft_cross_entropy": 0.860800955692927,
+        "accuracy": 0.7383333333333333,
+        "brier_soft": 0.07522662562007705,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8543428432941437,
         "improved": false
       }
     },
     {
+      "epoch": 16,
+      "train_soft_cross_entropy": 0.7659876039734593,
       "validation": {
+        "soft_cross_entropy": 0.8564839088916778,
+        "accuracy": 0.7533333333333333,
+        "brier_soft": 0.07288925250992179,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8543428432941437,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
+    "epochs_completed": 16,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

evidence/seed-3-history.json CHANGED Viewed

@@ -9,467 +9,195 @@
   "epochs": [
     {
       "epoch": 1,
-      "train_soft_cross_entropy": 1.1980749452555621,
       "validation": {
-        "soft_cross_entropy": 1.16634921391805,
-        "accuracy": 0.4266666666666667,
-        "brier_soft": 0.2427163557211558,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.16634921391805,
         "improved": true
       }
     },
     {
       "epoch": 2,
-      "train_soft_cross_entropy": 1.1646321240177862,
       "validation": {
-        "soft_cross_entropy": 1.1217502697308859,
-        "accuracy": 0.4533333333333333,
-        "brier_soft": 0.2186164912581444,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.1217502697308859,
         "improved": true
       }
     },
     {
       "epoch": 3,
-      "train_soft_cross_entropy": 1.1056555556367944,
       "validation": {
-        "soft_cross_entropy": 1.093972578048706,
-        "accuracy": 0.505,
-        "brier_soft": 0.199787005285422,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.093972578048706,
         "improved": true
       }
     },
     {
       "epoch": 4,
-      "train_soft_cross_entropy": 1.0671771099832323,
       "validation": {
-        "soft_cross_entropy": 1.0549713468551636,
-        "accuracy": 0.5516666666666666,
-        "brier_soft": 0.18068428387244542,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0549713468551636,
         "improved": true
       }
     },
     {
       "epoch": 5,
-      "train_soft_cross_entropy": 1.0494003568755257,
       "validation": {
-        "soft_cross_entropy": 1.0567963059743246,
-        "accuracy": 0.54,
-        "brier_soft": 0.18257476242880027,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 1.0549713468551636,
         "improved": false
       }
     },
     {
       "epoch": 6,
-      "train_soft_cross_entropy": 1.038703513851872,
-      "validation": {
-        "soft_cross_entropy": 1.0705021890004476,
-        "accuracy": 0.525,
-        "brier_soft": 0.19120527582863966,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 1.0549713468551636,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 7,
-      "train_soft_cross_entropy": 1.0313762316880404,
-      "validation": {
-        "soft_cross_entropy": 1.0447989773750306,
-        "accuracy": 0.5466666666666666,
-        "brier_soft": 0.17501757830381393,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0447989773750306,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 8,
-      "train_soft_cross_entropy": 1.017818791601393,
-      "validation": {
-        "soft_cross_entropy": 1.024161856174469,
-        "accuracy": 0.5816666666666667,
-        "brier_soft": 0.1630909529576699,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.024161856174469,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 9,
-      "train_soft_cross_entropy": 0.9899537578335514,
-      "validation": {
-        "soft_cross_entropy": 1.0265704361597696,
-        "accuracy": 0.6083333333333333,
-        "brier_soft": 0.16808087141563496,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 1.024161856174469,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 10,
-      "train_soft_cross_entropy": 0.9632394627288535,
-      "validation": {
-        "soft_cross_entropy": 1.0183856010437011,
-        "accuracy": 0.5983333333333334,
-        "brier_soft": 0.1608257148663203,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0183856010437011,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 11,
-      "train_soft_cross_entropy": 0.9342622081438701,
-      "validation": {
-        "soft_cross_entropy": 1.0024159566561381,
-        "accuracy": 0.6333333333333333,
-        "brier_soft": 0.14996731283764045,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 1.0024159566561381,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 12,
-      "train_soft_cross_entropy": 0.9137524432606168,
-      "validation": {
-        "soft_cross_entropy": 0.9873138956228892,
-        "accuracy": 0.6516666666666666,
-        "brier_soft": 0.14277802929282188,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9873138956228892,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 13,
-      "train_soft_cross_entropy": 0.8881368139496556,
-      "validation": {
-        "soft_cross_entropy": 0.9626709421475729,
-        "accuracy": 0.6716666666666666,
-        "brier_soft": 0.12283751085400581,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9626709421475729,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 14,
-      "train_soft_cross_entropy": 0.865500467883216,
       "validation": {
-        "soft_cross_entropy": 0.9580995849768321,
-        "accuracy": 0.685,
-        "brier_soft": 0.12383397127191226,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9580995849768321,
         "improved": true
       }
     },
     {
-      "epoch": 15,
-      "train_soft_cross_entropy": 0.8549449095461104,
-      "validation": {
-        "soft_cross_entropy": 0.9462178750832876,
-        "accuracy": 0.6916666666666667,
-        "brier_soft": 0.1187205430244406,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9462178750832876,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 16,
-      "train_soft_cross_entropy": 0.8394836834183446,
-      "validation": {
-        "soft_cross_entropy": 0.9463119049866994,
-        "accuracy": 0.6933333333333334,
-        "brier_soft": 0.11875337022046248,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 0.9462178750832876,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 17,
-      "train_soft_cross_entropy": 0.8286281617040987,
-      "validation": {
-        "soft_cross_entropy": 0.941375896135966,
-        "accuracy": 0.685,
-        "brier_soft": 0.11455494280904531,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.941375896135966,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 18,
-      "train_soft_cross_entropy": 0.8222652976601212,
-      "validation": {
-        "soft_cross_entropy": 0.9331588689486185,
-        "accuracy": 0.6866666666666666,
-        "brier_soft": 0.11237604923546314,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9331588689486185,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 19,
-      "train_soft_cross_entropy": 0.8160834203826056,
-      "validation": {
-        "soft_cross_entropy": 0.9387085942427317,
-        "accuracy": 0.6916666666666667,
-        "brier_soft": 0.11558652246991793,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 0.9331588689486185,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 20,
-      "train_soft_cross_entropy": 0.8087538785846146,
-      "validation": {
-        "soft_cross_entropy": 0.9408066284656524,
-        "accuracy": 0.6783333333333333,
-        "brier_soft": 0.11930533437679211,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 0.9331588689486185,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 21,
-      "train_soft_cross_entropy": 0.8048860243956248,
-      "validation": {
-        "soft_cross_entropy": 0.9236933688322703,
-        "accuracy": 0.7016666666666667,
-        "brier_soft": 0.10792215374608835,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9236933688322703,
-        "improved": true
-      }
-    },
-    {
-      "epoch": 22,
-      "train_soft_cross_entropy": 0.8001213801790167,
       "validation": {
-        "soft_cross_entropy": 0.9347231006622314,
-        "accuracy": 0.6983333333333334,
-        "brier_soft": 0.11313647958139579,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 0.9236933688322703,
         "improved": false
       }
     },
     {
-      "epoch": 23,
-      "train_soft_cross_entropy": 0.7940637088263476,
-      "validation": {
-        "soft_cross_entropy": 0.9299098292986552,
-        "accuracy": 0.7033333333333334,
-        "brier_soft": 0.11475709093113741,
-        "decisions": 600
-      },
-      "early_stopping": {
-        "patience": 3,
-        "min_epochs": 10,
-        "epochs_without_improvement": 2,
-        "best_validation_loss": 0.9236933688322703,
-        "improved": false
-      }
-    },
-    {
-      "epoch": 24,
-      "train_soft_cross_entropy": 0.7884197377717054,
       "validation": {
-        "soft_cross_entropy": 0.9105557099978129,
-        "accuracy": 0.715,
-        "brier_soft": 0.1013037375236551,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9105557099978129,
         "improved": true
       }
     },
     {
-      "epoch": 25,
-      "train_soft_cross_entropy": 0.784954049454795,
       "validation": {
-        "soft_cross_entropy": 0.923290346066157,
-        "accuracy": 0.71,
-        "brier_soft": 0.10530036373684803,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 0.9105557099978129,
         "improved": false
       }
     },
     {
-      "epoch": 26,
-      "train_soft_cross_entropy": 0.7828840752442677,
       "validation": {
-        "soft_cross_entropy": 0.9299345835049947,
-        "accuracy": 0.7066666666666667,
-        "brier_soft": 0.11173659774164359,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
-        "best_validation_loss": 0.9105557099978129,
         "improved": false
       }
     },
     {
-      "epoch": 27,
-      "train_soft_cross_entropy": 0.7790651953661883,
       "validation": {
-        "soft_cross_entropy": 0.9137676254908244,
-        "accuracy": 0.7216666666666667,
-        "brier_soft": 0.10172271362195412,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
-        "best_validation_loss": 0.9105557099978129,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
-    "epochs_completed": 27,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

   "epochs": [
     {
       "epoch": 1,
+      "train_soft_cross_entropy": 1.0557677305186237,
       "validation": {
+        "soft_cross_entropy": 0.990565903186798,
+        "accuracy": 0.6166666666666667,
+        "brier_soft": 0.15243564940989018,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.990565903186798,
         "improved": true
       }
     },
     {
       "epoch": 2,
+      "train_soft_cross_entropy": 0.9286279639491328,
       "validation": {
+        "soft_cross_entropy": 0.9022069962819418,
+        "accuracy": 0.71,
+        "brier_soft": 0.0947934572895368,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9022069962819418,
         "improved": true
       }
     },
     {
       "epoch": 3,
+      "train_soft_cross_entropy": 0.8668451248716424,
       "validation": {
+        "soft_cross_entropy": 0.8912256610393524,
+        "accuracy": 0.725,
+        "brier_soft": 0.08988383966187637,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8912256610393524,
         "improved": true
       }
     },
     {
       "epoch": 4,
+      "train_soft_cross_entropy": 0.8375598757796817,
       "validation": {
+        "soft_cross_entropy": 0.8716744474569956,
+        "accuracy": 0.7566666666666667,
+        "brier_soft": 0.07936330476154883,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8716744474569956,
         "improved": true
       }
     },
     {
       "epoch": 5,
+      "train_soft_cross_entropy": 0.8176034839064986,
       "validation": {
+        "soft_cross_entropy": 0.8743925015131633,
+        "accuracy": 0.7333333333333333,
+        "brier_soft": 0.07994898026498655,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8716744474569956,
         "improved": false
       }
     },
     {
       "epoch": 6,
+      "train_soft_cross_entropy": 0.8031262465318044,
       "validation": {
+        "soft_cross_entropy": 0.8599367336432139,
+        "accuracy": 0.76,
+        "brier_soft": 0.07330287167181572,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8599367336432139,
         "improved": true
       }
     },
     {
+      "epoch": 7,
+      "train_soft_cross_entropy": 0.7926116739820551,
       "validation": {
+        "soft_cross_entropy": 0.8638877793153127,
+        "accuracy": 0.7566666666666667,
+        "brier_soft": 0.07329062350094319,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8599367336432139,
         "improved": false
       }
     },
     {
+      "epoch": 8,
+      "train_soft_cross_entropy": 0.7867316756866596,
       "validation": {
+        "soft_cross_entropy": 0.8545633804798126,
+        "accuracy": 0.77,
+        "brier_soft": 0.06962224062221746,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8545633804798126,
         "improved": true
       }
     },
     {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.7820918090696688,
       "validation": {
+        "soft_cross_entropy": 0.8555870747566223,
+        "accuracy": 0.7683333333333333,
+        "brier_soft": 0.06890943528773884,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8545633804798126,
         "improved": false
       }
     },
     {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.779588269745862,
       "validation": {
+        "soft_cross_entropy": 0.8600131020943323,
+        "accuracy": 0.76,
+        "brier_soft": 0.07248457937811811,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8545633804798126,
         "improved": false
       }
     },
     {
+      "epoch": 11,
+      "train_soft_cross_entropy": 0.7761333607302772,
       "validation": {
+        "soft_cross_entropy": 0.8604028668006262,
+        "accuracy": 0.7616666666666667,
+        "brier_soft": 0.07338622493979831,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8545633804798126,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
+    "epochs_completed": 11,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

evidence/seed-4-history.json CHANGED Viewed

@@ -9,178 +9,195 @@
   "epochs": [
     {
       "epoch": 1,
-      "train_soft_cross_entropy": 1.002756692568461,
       "validation": {
-        "soft_cross_entropy": 0.9602992224693299,
-        "accuracy": 0.6783333333333333,
-        "brier_soft": 0.1296700432151556,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.9602992224693299,
         "improved": true
       }
     },
     {
       "epoch": 2,
-      "train_soft_cross_entropy": 0.8854288237624698,
       "validation": {
-        "soft_cross_entropy": 0.8732180511951446,
-        "accuracy": 0.7316666666666667,
-        "brier_soft": 0.08001670623819034,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8732180511951446,
         "improved": true
       }
     },
     {
       "epoch": 3,
-      "train_soft_cross_entropy": 0.8375967553809837,
       "validation": {
-        "soft_cross_entropy": 0.8618760740756989,
-        "accuracy": 0.7566666666666667,
-        "brier_soft": 0.07448753335823616,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8618760740756989,
         "improved": true
       }
     },
     {
       "epoch": 4,
-      "train_soft_cross_entropy": 0.8120107175244226,
       "validation": {
-        "soft_cross_entropy": 0.8502214312553406,
-        "accuracy": 0.7683333333333333,
-        "brier_soft": 0.06676056392490864,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8502214312553406,
         "improved": true
       }
     },
     {
       "epoch": 5,
-      "train_soft_cross_entropy": 0.7984476970301734,
       "validation": {
-        "soft_cross_entropy": 0.8527251331011454,
-        "accuracy": 0.765,
-        "brier_soft": 0.07089023986210426,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 1,
-        "best_validation_loss": 0.8502214312553406,
-        "improved": false
       }
     },
     {
       "epoch": 6,
-      "train_soft_cross_entropy": 0.7902782445042221,
       "validation": {
-        "soft_cross_entropy": 0.8496846203009287,
-        "accuracy": 0.765,
-        "brier_soft": 0.06887457605761786,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
-        "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8496846203009287,
-        "improved": true
       }
     },
     {
       "epoch": 7,
-      "train_soft_cross_entropy": 0.7847388537283297,
       "validation": {
-        "soft_cross_entropy": 0.8448993345101674,
-        "accuracy": 0.7533333333333333,
-        "brier_soft": 0.0668264798882107,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
-        "best_validation_loss": 0.8448993345101674,
         "improved": true
       }
     },
     {
-      "epoch": 8,
-      "train_soft_cross_entropy": 0.780297756018462,
       "validation": {
-        "soft_cross_entropy": 0.8471182523171107,
-        "accuracy": 0.7683333333333333,
-        "brier_soft": 0.06819427679603299,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
-        "best_validation_loss": 0.8448993345101674,
         "improved": false
       }
     },
     {
-      "epoch": 9,
-      "train_soft_cross_entropy": 0.7801684511590887,
       "validation": {
-        "soft_cross_entropy": 0.8466147363185883,
-        "accuracy": 0.775,
-        "brier_soft": 0.06677873468647401,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
-        "best_validation_loss": 0.8448993345101674,
         "improved": false
       }
     },
     {
-      "epoch": 10,
-      "train_soft_cross_entropy": 0.7775638208565888,
       "validation": {
-        "soft_cross_entropy": 0.8455785755316416,
-        "accuracy": 0.78,
-        "brier_soft": 0.06686171248555184,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
-        "best_validation_loss": 0.8448993345101674,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
-    "epochs_completed": 10,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

   "epochs": [
     {
       "epoch": 1,
+      "train_soft_cross_entropy": 1.022522153324551,
       "validation": {
+        "soft_cross_entropy": 0.9845864299933116,
+        "accuracy": 0.6683333333333333,
+        "brier_soft": 0.14738356913129488,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9845864299933116,
         "improved": true
       }
     },
     {
       "epoch": 2,
+      "train_soft_cross_entropy": 0.9041571515136295,
       "validation": {
+        "soft_cross_entropy": 0.9089026947816213,
+        "accuracy": 0.7216666666666667,
+        "brier_soft": 0.10058322168886662,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.9089026947816213,
         "improved": true
       }
     },
     {
       "epoch": 3,
+      "train_soft_cross_entropy": 0.8495574627099214,
       "validation": {
+        "soft_cross_entropy": 0.8853352854649226,
+        "accuracy": 0.74,
+        "brier_soft": 0.08744065261135499,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8853352854649226,
         "improved": true
       }
     },
     {
       "epoch": 4,
+      "train_soft_cross_entropy": 0.8263491551522856,
       "validation": {
+        "soft_cross_entropy": 0.8789768069982529,
+        "accuracy": 0.7416666666666667,
+        "brier_soft": 0.08262864720076323,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8789768069982529,
         "improved": true
       }
     },
     {
       "epoch": 5,
+      "train_soft_cross_entropy": 0.8096302935812209,
       "validation": {
+        "soft_cross_entropy": 0.864648152589798,
+        "accuracy": 0.73,
+        "brier_soft": 0.07628292332092922,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 0,
+        "best_validation_loss": 0.864648152589798,
+        "improved": true
       }
     },
     {
       "epoch": 6,
+      "train_soft_cross_entropy": 0.7991735404950601,
       "validation": {
+        "soft_cross_entropy": 0.8743107922871908,
+        "accuracy": 0.74,
+        "brier_soft": 0.08211790287246307,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
+        "epochs_without_improvement": 1,
+        "best_validation_loss": 0.864648152589798,
+        "improved": false
       }
     },
     {
       "epoch": 7,
+      "train_soft_cross_entropy": 0.7918641125714337,
       "validation": {
+        "soft_cross_entropy": 0.86731658577919,
+        "accuracy": 0.7483333333333333,
+        "brier_soft": 0.07722779513647159,
+        "decisions": 600
+      },
+      "early_stopping": {
+        "patience": 3,
+        "min_epochs": 10,
+        "epochs_without_improvement": 2,
+        "best_validation_loss": 0.864648152589798,
+        "improved": false
+      }
+    },
+    {
+      "epoch": 8,
+      "train_soft_cross_entropy": 0.7861463651833711,
+      "validation": {
+        "soft_cross_entropy": 0.8540202794472377,
+        "accuracy": 0.76,
+        "brier_soft": 0.07019127607345581,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 0,
+        "best_validation_loss": 0.8540202794472377,
         "improved": true
       }
     },
     {
+      "epoch": 9,
+      "train_soft_cross_entropy": 0.7806310887248428,
       "validation": {
+        "soft_cross_entropy": 0.8646407808860143,
+        "accuracy": 0.735,
+        "brier_soft": 0.07658187456429005,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 1,
+        "best_validation_loss": 0.8540202794472377,
         "improved": false
       }
     },
     {
+      "epoch": 10,
+      "train_soft_cross_entropy": 0.7768384672094274,
       "validation": {
+        "soft_cross_entropy": 0.8618478431304296,
+        "accuracy": 0.73,
+        "brier_soft": 0.07576349244763454,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 2,
+        "best_validation_loss": 0.8540202794472377,
         "improved": false
       }
     },
     {
+      "epoch": 11,
+      "train_soft_cross_entropy": 0.7740854684511821,
       "validation": {
+        "soft_cross_entropy": 0.8627219025293986,
+        "accuracy": 0.7283333333333334,
+        "brier_soft": 0.07741292332919936,
         "decisions": 600
       },
       "early_stopping": {
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
+        "best_validation_loss": 0.8540202794472377,
         "improved": false
       }
     }
   ],
   "stopping": {
     "reason": "early_stopping",
+    "epochs_completed": 11,
     "patience": 3,
     "min_epochs": 10,
     "epochs_without_improvement": 3,

export-verification.json CHANGED Viewed

@@ -1,11 +1,11 @@
 {
   "passed": true,
-  "candidate_seed": 1,
-  "selected_epoch": 14,
   "cases": 3516,
   "decisions": 5116,
   "benchmark_cases_sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
-  "checkpoint_sha256": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
   "all_answer_objects_exact": true,
   "all_suite_metrics_exact": true,
   "bundled_code_imported_from_outside_workspace": true,
@@ -35,8 +35,8 @@
     "portable_adapter": {
       "base_model": "convaiinnovations/laya",
       "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
-      "seed": 1,
-      "selected_epoch": 14
     }
   }
 }

 {
   "passed": true,
+  "candidate_seed": 4,
+  "selected_epoch": 8,
   "cases": 3516,
   "decisions": 5116,
   "benchmark_cases_sha256": "10fb671245cdac1ee09e9c4d2b9b09829fbf1abd06e0954fddee0b9ec8519854",
+  "checkpoint_sha256": "f23ad342a969773bbbcb18619ce8a8843bbb105d94fe64c5a837d68f26829081",
   "all_answer_objects_exact": true,
   "all_suite_metrics_exact": true,
   "bundled_code_imported_from_outside_workspace": true,
     "portable_adapter": {
       "base_model": "convaiinnovations/laya",
       "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+      "seed": 4,
+      "selected_epoch": 8
     }
   }
 }

make_training_config.py ADDED Viewed

	@@ -0,0 +1,37 @@

+"""Resolve the pinned base checkpoint and create a reproducible training config."""
+import argparse
+import json
+from pathlib import Path
+from huggingface_hub import snapshot_download
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument("--seed", type=int, choices=range(5), required=True)
+parser.add_argument("--device", default="cuda:0")
+parser.add_argument("--out", required=True)
+args = parser.parse_args()
+root = Path(__file__).resolve().parent
+metadata = json.loads((root / "adapter_config.json").read_text())
+base = snapshot_download(
+    metadata["base_model"],
+    revision=metadata["base_revision"],
+    allow_patterns=["model.safetensors", "rl_agent_config.json", "encoder/*", "tokenizer/*"],
+)
+config = {
+    "name": f"Deterministic frozen base + linear, seed {args.seed} (UNTRAINED)",
+    "adapter": "python",
+    "mode": "reference",
+    "factory": "ariadne_bench.frozen_input_interface:create",
+    "options": {
+        "laya_model": str(Path(base).resolve()),
+        "device": args.device,
+        "batch_size": 16,
+        "max_len": 1024,
+        "head_max_len": 256,
+        "seed": args.seed,
+        "training_scope": "bridge",
+        "add_interface": True,
+        "deterministic": True,
+    },
+}
+Path(args.out).write_text(json.dumps(config, indent=2) + "\n")

metrics.json CHANGED Viewed

@@ -12,8 +12,8 @@
   "runs": {
     "0": {
       "seed": 0,
-      "selected_epoch": 20,
-      "elapsed_s": 1038.9482859019772,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
@@ -21,14 +21,14 @@
         "decisions": 600
       },
       "selected_validation": {
-        "soft_cross_entropy": 1.0292588464419048,
-        "accuracy": 0.57,
-        "brier_soft": 0.16918850486477216,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
-        "epochs_completed": 23,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
@@ -42,159 +42,159 @@
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.61,
-          "accuracy_valid": 0.61,
-          "ece_top_label": 0.15303136435282014,
-          "mean_confidence": 0.4574502381919959,
-          "brier_hard": 0.549798004153015,
           "brier_hard_n": 2000,
-          "nll_hard": 0.9521041848093966,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
-          "soft_accuracy": 0.3883819759829014,
           "soft_accuracy_n": 2000,
-          "brier_soft": 0.15223049605383382,
           "brier_soft_n": 2000,
-          "kl_gold_to_prediction": 0.2712886346630264,
           "kl_gold_to_prediction_n": 2000,
-          "total_variation": 0.2797391333751494,
           "total_variation_n": 2000,
-          "score_mae": 0.45095113683604626,
           "score_mae_n": 800,
-          "within_one_level": 0.915,
           "within_one_level_n": 800,
-          "macro_f1": 0.4150680787086972
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.31333333333333335,
-          "accuracy_valid": 0.31333333333333335,
-          "ece_top_label": 0.025934925132124264,
-          "mean_confidence": 0.28915707486787573,
-          "brier_hard": 0.7441339007069468,
           "brier_hard_n": 600,
-          "nll_hard": 1.3750333163467814,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.29084523228646375
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.195,
-          "accuracy_valid": 0.195,
-          "ece_top_label": 0.08537782470783518,
-          "mean_confidence": 0.2764666886168944,
-          "brier_hard": 0.8415061960524678,
           "brier_hard_n": 600,
-          "nll_hard": 1.810555559185162,
           "nll_hard_n": 600,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.12883846057517262
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.5666666666666667,
-          "accuracy_valid": 0.5666666666666667,
-          "ece_top_label": 0.08635800000000002,
-          "mean_confidence": 0.5258943333333334,
-          "brier_hard": 0.5020554209333333,
           "brier_hard_n": 600,
-          "nll_hard": 0.6953013099942155,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.5074762578298646
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.15,
-          "accuracy_valid": 0.15,
-          "ece_top_label": 0.11939667407180776,
-          "mean_confidence": 0.2693966740718078,
-          "brier_hard": 0.8207044411796734,
           "brier_hard_n": 600,
-          "nll_hard": 1.6589117409852412,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "score_mae": 1.2247300886774997,
           "score_mae_n": 600,
-          "within_one_level": 0.42,
           "within_one_level_n": 600,
-          "macro_f1": 0.10629896865332711
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.5344827586206896,
-          "accuracy_valid": 0.5344827586206896,
-          "ece_top_label": 0.02089741379310345,
-          "mean_confidence": 0.5135853448275862,
-          "brier_hard": 0.5003080198275862,
           "brier_hard_n": 116,
-          "nll_hard": 0.6934547870034389,
           "nll_hard_n": 116,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 116,
-          "macro_f1": 0.513664596273292
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.023333333333333334,
-          "accuracy_valid": 0.023333333333333334,
-          "ece_top_label": 0.09797495336730408,
-          "mean_confidence": 0.12130828670063742,
-          "brier_hard": 0.9672041702729498,
           "brier_hard_n": 300,
-          "nll_hard": 3.1313647377288074,
           "nll_hard_n": 300,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.011215849106652133
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.3433333333333333,
-          "accuracy_valid": 0.3433333333333333,
-          "ece_top_label": 0.05212130996336308,
-          "mean_confidence": 0.39545464329669644,
-          "brier_hard": 0.6747449551934047,
           "brier_hard_n": 300,
-          "nll_hard": 1.1111148411541347,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.24805362074756226
         }
       }
     },
     "1": {
       "seed": 1,
-      "selected_epoch": 14,
-      "elapsed_s": 766.9112328969641,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
@@ -202,14 +202,14 @@
         "decisions": 600
       },
       "selected_validation": {
-        "soft_cross_entropy": 0.8377540612220764,
-        "accuracy": 0.765,
-        "brier_soft": 0.06276483290052662,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
-        "epochs_completed": 17,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
@@ -223,159 +223,159 @@
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7695,
-          "accuracy_valid": 0.7695,
-          "ece_top_label": 0.2142237525419539,
-          "mean_confidence": 0.555276247458046,
-          "brier_hard": 0.3977842163576973,
           "brier_hard_n": 2000,
-          "nll_hard": 0.7027635604099817,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
-          "soft_accuracy": 0.47103605907759827,
           "soft_accuracy_n": 2000,
-          "brier_soft": 0.06255403710877039,
           "brier_soft_n": 2000,
-          "kl_gold_to_prediction": 0.11643450461088396,
           "kl_gold_to_prediction_n": 2000,
-          "total_variation": 0.17146599324503523,
           "total_variation_n": 2000,
-          "score_mae": 0.2299944493828981,
           "score_mae_n": 800,
-          "within_one_level": 0.98875,
           "within_one_level_n": 800,
-          "macro_f1": 0.6487623885802034
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.9316666666666666,
-          "accuracy_valid": 0.9316666666666666,
-          "ece_top_label": 0.03387674147756575,
-          "mean_confidence": 0.904708562141314,
-          "brier_hard": 0.10911433540423408,
           "brier_hard_n": 600,
-          "nll_hard": 0.21361121678114303,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.9283987145646934
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.5733333333333334,
-          "accuracy_valid": 0.5733333333333334,
-          "ece_top_label": 0.3169210441851814,
-          "mean_confidence": 0.8880587177380197,
-          "brier_hard": 0.7264584486728032,
           "brier_hard_n": 600,
-          "nll_hard": 2.146154603203373,
           "nll_hard_n": 600,
-          "zero_probability_gold": 0.008333333333333333,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.4862286665693279
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7983333333333333,
-          "accuracy_valid": 0.7983333333333333,
-          "ece_top_label": 0.09755433333333334,
-          "mean_confidence": 0.8900093333333333,
-          "brier_hard": 0.29822958146666667,
           "brier_hard_n": 600,
-          "nll_hard": 0.4711253960780479,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.7813983878883868
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.42,
-          "accuracy_valid": 0.42,
-          "ece_top_label": 0.1643168667044709,
-          "mean_confidence": 0.5761336745094923,
-          "brier_hard": 0.722245716690302,
           "brier_hard_n": 600,
-          "nll_hard": 1.412054753482227,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "score_mae": 0.7360310433394476,
           "score_mae_n": 600,
-          "within_one_level": 0.7466666666666667,
           "within_one_level_n": 600,
-          "macro_f1": 0.41387920942262574
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7155172413793104,
-          "accuracy_valid": 0.7155172413793104,
-          "ece_top_label": 0.20010517241379305,
-          "mean_confidence": 0.9105,
-          "brier_hard": 0.43610344172413795,
           "brier_hard_n": 116,
-          "nll_hard": 1.2355666268201304,
           "nll_hard_n": 116,
-          "zero_probability_gold": 0.008620689655172414,
           "zero_probability_gold_n": 116,
-          "macro_f1": 0.7038756091900673
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7433333333333333,
-          "accuracy_valid": 0.7433333333333333,
-          "ece_top_label": 0.1906540683935104,
-          "mean_confidence": 0.9339874017268438,
-          "brier_hard": 0.43706654758878133,
           "brier_hard_n": 300,
-          "nll_hard": 2.7801978001907903,
           "nll_hard_n": 300,
-          "zero_probability_gold": 0.07,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.44346732036749054
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.8766666666666667,
-          "accuracy_valid": 0.8766666666666667,
-          "ece_top_label": 0.055158283453726184,
-          "mean_confidence": 0.913468962679153,
-          "brier_hard": 0.19180470102420566,
           "brier_hard_n": 300,
-          "nll_hard": 0.3708980570966955,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.8772028178860477
         }
       }
     },
     "2": {
       "seed": 2,
-      "selected_epoch": 15,
-      "elapsed_s": 813.5451362769818,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
@@ -383,14 +383,14 @@
         "decisions": 600
       },
       "selected_validation": {
-        "soft_cross_entropy": 0.9867499772707621,
-        "accuracy": 0.6316666666666667,
-        "brier_soft": 0.14927835414807003,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
-        "epochs_completed": 18,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
@@ -404,159 +404,159 @@
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.661,
-          "accuracy_valid": 0.661,
-          "ece_top_label": 0.17250655060654457,
-          "mean_confidence": 0.48883668313682976,
-          "brier_hard": 0.5040496940863629,
           "brier_hard_n": 2000,
-          "nll_hard": 0.8694405673021295,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
-          "soft_accuracy": 0.41427322765683283,
           "soft_accuracy_n": 2000,
-          "brier_soft": 0.12329331823446005,
           "brier_soft_n": 2000,
-          "kl_gold_to_prediction": 0.2167589625120186,
           "kl_gold_to_prediction_n": 2000,
-          "total_variation": 0.24557641844905673,
           "total_variation_n": 2000,
-          "score_mae": 0.38257377026959916,
           "score_mae_n": 800,
-          "within_one_level": 0.92625,
           "within_one_level_n": 800,
-          "macro_f1": 0.511489281675723
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.2633333333333333,
-          "accuracy_valid": 0.2633333333333333,
-          "ece_top_label": 0.0415880758215024,
-          "mean_confidence": 0.30492140915483573,
-          "brier_hard": 0.7533353201524136,
           "brier_hard_n": 600,
-          "nll_hard": 1.392051963909145,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.2196261943166739
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.04833333333333333,
-          "accuracy_valid": 0.04833333333333333,
-          "ece_top_label": 0.21796334391120661,
-          "mean_confidence": 0.2662966772445399,
-          "brier_hard": 0.9083949711964834,
           "brier_hard_n": 600,
-          "nll_hard": 2.0132867760168445,
           "nll_hard_n": 600,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.03431938431938432
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.49833333333333335,
-          "accuracy_valid": 0.49833333333333335,
-          "ece_top_label": 0.03133150000000007,
-          "mean_confidence": 0.5290545,
-          "brier_hard": 0.5040687543666666,
           "brier_hard_n": 600,
-          "nll_hard": 0.6972355680599187,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.49597983919356775
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.22666666666666666,
-          "accuracy_valid": 0.22666666666666666,
-          "ece_top_label": 0.08382145968032653,
-          "mean_confidence": 0.30788872429147607,
-          "brier_hard": 0.8111226172703578,
           "brier_hard_n": 600,
-          "nll_hard": 1.6364255257258482,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "score_mae": 1.223873285603664,
           "score_mae_n": 600,
-          "within_one_level": 0.39,
           "within_one_level_n": 600,
-          "macro_f1": 0.12552908285983264
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.3448275862068966,
-          "accuracy_valid": 0.3448275862068966,
-          "ece_top_label": 0.1914112068965517,
-          "mean_confidence": 0.5362387931034482,
-          "brier_hard": 0.5323824856896552,
           "brier_hard_n": 116,
-          "nll_hard": 0.7258214322004152,
           "nll_hard_n": 116,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 116,
-          "macro_f1": 0.275
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.06333333333333334,
-          "accuracy_valid": 0.06333333333333334,
-          "ece_top_label": 0.03411377590323972,
-          "mean_confidence": 0.09744710923657304,
-          "brier_hard": 0.9608971275390831,
           "brier_hard_n": 300,
-          "nll_hard": 3.135262799945936,
           "nll_hard_n": 300,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.02907709642455108
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.3433333333333333,
-          "accuracy_valid": 0.3433333333333333,
-          "ece_top_label": 0.04294870734165375,
-          "mean_confidence": 0.38628204067498706,
-          "brier_hard": 0.6761361630010466,
           "brier_hard_n": 300,
-          "nll_hard": 1.1128104653939794,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.3099922839506173
         }
       }
     },
     "3": {
       "seed": 3,
-      "selected_epoch": 24,
-      "elapsed_s": 1226.4128511130111,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
@@ -564,14 +564,14 @@
         "decisions": 600
       },
       "selected_validation": {
-        "soft_cross_entropy": 0.9105557099978129,
-        "accuracy": 0.715,
-        "brier_soft": 0.1013037375236551,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
-        "epochs_completed": 27,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
@@ -585,159 +585,159 @@
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7145,
-          "accuracy_valid": 0.7145,
-          "ece_top_label": 0.18500231281217855,
-          "mean_confidence": 0.5300184130804106,
-          "brier_hard": 0.45082257101028994,
           "brier_hard_n": 2000,
-          "nll_hard": 0.7891005479512444,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
-          "soft_accuracy": 0.4447570518885741,
           "soft_accuracy_n": 2000,
-          "brier_soft": 0.09325671264191153,
           "brier_soft_n": 2000,
-          "kl_gold_to_prediction": 0.17157487793178008,
           "kl_gold_to_prediction_n": 2000,
-          "total_variation": 0.21097575386758916,
           "total_variation_n": 2000,
-          "score_mae": 0.32394138130337413,
           "score_mae_n": 800,
-          "within_one_level": 0.96375,
           "within_one_level_n": 800,
-          "macro_f1": 0.5630278027844289
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.285,
-          "accuracy_valid": 0.285,
-          "ece_top_label": 0.03904648151058483,
-          "mean_confidence": 0.3240464815105848,
-          "brier_hard": 0.7625373960165811,
           "brier_hard_n": 600,
-          "nll_hard": 1.4140716749662658,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.25772655273033623
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.135,
-          "accuracy_valid": 0.135,
-          "ece_top_label": 0.31863940813484304,
-          "mean_confidence": 0.45363940813484305,
-          "brier_hard": 0.9777635802768799,
           "brier_hard_n": 600,
-          "nll_hard": 2.137352366621794,
           "nll_hard_n": 600,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.06590661903472231
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.48833333333333334,
-          "accuracy_valid": 0.48833333333333334,
-          "ece_top_label": 0.10787050000000001,
-          "mean_confidence": 0.5912141666666667,
-          "brier_hard": 0.5250116893,
           "brier_hard_n": 600,
-          "nll_hard": 0.719837552287629,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.48696381173075903
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.25666666666666665,
-          "accuracy_valid": 0.25666666666666665,
-          "ece_top_label": 0.05066152500332892,
-          "mean_confidence": 0.3073281916699956,
-          "brier_hard": 0.8215936069738192,
           "brier_hard_n": 600,
-          "nll_hard": 1.6775697808435297,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "score_mae": 1.245226132804566,
           "score_mae_n": 600,
-          "within_one_level": 0.44333333333333336,
           "within_one_level_n": 600,
-          "macro_f1": 0.13260195835186606
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.5689655172413793,
-          "accuracy_valid": 0.5689655172413793,
-          "ece_top_label": 0.07609396551724136,
-          "mean_confidence": 0.6450594827586207,
-          "brier_hard": 0.4881634129310345,
           "brier_hard_n": 116,
-          "nll_hard": 0.6835747994579267,
           "nll_hard_n": 116,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 116,
-          "macro_f1": 0.5461658841940532
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.023333333333333334,
-          "accuracy_valid": 0.023333333333333334,
-          "ece_top_label": 0.14827139054632427,
-          "mean_confidence": 0.1716047238796576,
-          "brier_hard": 1.012268483556247,
           "brier_hard_n": 300,
-          "nll_hard": 3.540188562309456,
           "nll_hard_n": 300,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.014601824457593688
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.33666666666666667,
-          "accuracy_valid": 0.33666666666666667,
-          "ece_top_label": 0.21980800490067007,
-          "mean_confidence": 0.5564746715673368,
-          "brier_hard": 0.7709342547858443,
           "brier_hard_n": 300,
-          "nll_hard": 1.2650663651832452,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.20238216957239646
         }
       }
     },
     "4": {
       "seed": 4,
-      "selected_epoch": 7,
-      "elapsed_s": 452.7137592760264,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
@@ -745,14 +745,14 @@
         "decisions": 600
       },
       "selected_validation": {
-        "soft_cross_entropy": 0.8448993345101674,
-        "accuracy": 0.7533333333333333,
-        "brier_soft": 0.0668264798882107,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
-        "epochs_completed": 10,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
@@ -766,152 +766,152 @@
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7565,
-          "accuracy_valid": 0.7565,
-          "ece_top_label": 0.20311433485058888,
-          "mean_confidence": 0.5533856651494111,
-          "brier_hard": 0.4063906772103004,
           "brier_hard_n": 2000,
-          "nll_hard": 0.7152837956058272,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
-          "soft_accuracy": 0.46913207673904656,
           "soft_accuracy_n": 2000,
-          "brier_soft": 0.06497074467877281,
           "brier_soft_n": 2000,
-          "kl_gold_to_prediction": 0.12038107186141371,
           "kl_gold_to_prediction_n": 2000,
-          "total_variation": 0.17619801386776057,
           "total_variation_n": 2000,
-          "score_mae": 0.2447599606823941,
           "score_mae_n": 800,
-          "within_one_level": 0.99375,
           "within_one_level_n": 800,
-          "macro_f1": 0.6349242079343602
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.935,
-          "accuracy_valid": 0.935,
-          "ece_top_label": 0.0277188639855619,
-          "mean_confidence": 0.9106097216911906,
-          "brier_hard": 0.10223891825148304,
           "brier_hard_n": 600,
-          "nll_hard": 0.2046134786862912,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.9319475717225341
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.565,
-          "accuracy_valid": 0.565,
-          "ece_top_label": 0.3127789762695219,
-          "mean_confidence": 0.8750361088494466,
-          "brier_hard": 0.7278820816911535,
           "brier_hard_n": 600,
-          "nll_hard": 2.179428847187102,
           "nll_hard_n": 600,
-          "zero_probability_gold": 0.01,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.47063728412574746
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.8116666666666666,
-          "accuracy_valid": 0.8116666666666666,
-          "ece_top_label": 0.08128733333333335,
-          "mean_confidence": 0.8882866666666667,
-          "brier_hard": 0.2800035268666667,
           "brier_hard_n": 600,
-          "nll_hard": 0.4526075086973918,
           "nll_hard_n": 600,
-          "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "macro_f1": 0.7989317880539385
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.47333333333333333,
-          "accuracy_valid": 0.47333333333333333,
-          "ece_top_label": 0.11648579870821457,
-          "mean_confidence": 0.5717307193165773,
-          "brier_hard": 0.6735168048802583,
           "brier_hard_n": 600,
-          "nll_hard": 1.2841517570209144,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
-          "score_mae": 0.6724125988264632,
           "score_mae_n": 600,
-          "within_one_level": 0.7883333333333333,
           "within_one_level_n": 600,
-          "macro_f1": 0.46002232794452175
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7241379310344828,
-          "accuracy_valid": 0.7241379310344828,
-          "ece_top_label": 0.22484482758620694,
-          "mean_confidence": 0.9346275862068966,
-          "brier_hard": 0.44889866172413795,
           "brier_hard_n": 116,
-          "nll_hard": 1.682957724359292,
           "nll_hard_n": 116,
-          "zero_probability_gold": 0.02586206896551724,
           "zero_probability_gold_n": 116,
-          "macro_f1": 0.7118012422360249
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.7733333333333333,
-          "accuracy_valid": 0.7733333333333333,
-          "ece_top_label": 0.1762748719939652,
-          "mean_confidence": 0.9448706790799233,
-          "brier_hard": 0.3914895786578462,
           "brier_hard_n": 300,
-          "nll_hard": 2.594218156353015,
           "nll_hard_n": 300,
-          "zero_probability_gold": 0.06333333333333334,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.4611733462473961
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
-          "accuracy_all": 0.8766666666666667,
-          "accuracy_valid": 0.8766666666666667,
-          "ece_top_label": 0.059115758529190925,
-          "mean_confidence": 0.9151327612640476,
-          "brier_hard": 0.1964252077913328,
           "brier_hard_n": 300,
-          "nll_hard": 0.36246740512671244,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
-          "macro_f1": 0.8760253913421515
         }
       }
     }
@@ -919,54 +919,54 @@
   "summary": {
     "typed-decisions": {
       "n": 5,
-      "mean_percent": 70.23,
-      "sample_variance_pp_squared": 44.56824999999997,
-      "sample_std_pp": 6.675945625901995
     },
     "ag-news": {
       "n": 5,
-      "mean_percent": 54.56666666666666,
-      "sample_variance_pp_squared": 1255.536111111111,
-      "sample_std_pp": 35.433544997799906
     },
     "boolq": {
       "n": 5,
-      "mean_percent": 63.266666666666666,
-      "sample_variance_pp_squared": 256.79999999999995,
-      "sample_std_pp": 16.0249804992081
     },
     "emotion": {
       "n": 5,
-      "mean_percent": 30.333333333333332,
-      "sample_variance_pp_squared": 616.1666666666666,
-      "sample_std_pp": 24.82270466058577
     },
     "prompt-injections": {
       "n": 5,
-      "mean_percent": 57.758620689655174,
-      "sample_variance_pp_squared": 241.5279429250891,
-      "sample_std_pp": 15.541169290793055
     },
     "sst5": {
       "n": 5,
-      "mean_percent": 30.53333333333333,
-      "sample_variance_pp_squared": 185.14444444444447,
-      "sample_std_pp": 13.606779356057938
     },
     "massive-intent.en": {
       "n": 5,
-      "mean_percent": 32.53333333333333,
-      "sample_variance_pp_squared": 1566.1999999999998,
-      "sample_std_pp": 39.57524478761944
     },
     "xnli.en": {
       "n": 5,
-      "mean_percent": 55.53333333333334,
-      "sample_variance_pp_squared": 860.5333333333334,
-      "sample_std_pp": 29.33484844571953
     }
   },
-  "candidate_seed": 1,
   "variance_definition": "sample variance, n-1, percentage points squared",
   "benchmark_manifest": {
     "format_version": 1,

   "runs": {
     "0": {
       "seed": 0,
+      "selected_epoch": 11,
+      "elapsed_s": 612.5042744020466,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
         "decisions": 600
       },
       "selected_validation": {
+        "soft_cross_entropy": 0.8771124110619227,
+        "accuracy": 0.7116666666666667,
+        "brier_soft": 0.0828241604194045,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
+        "epochs_completed": 14,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7425,
+          "accuracy_valid": 0.7425,
+          "ece_top_label": 0.19222331993705236,
+          "mean_confidence": 0.5502766800629476,
+          "brier_hard": 0.41714854154295294,
           "brier_hard_n": 2000,
+          "nll_hard": 0.7346455747590726,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.4642362342348646,
           "soft_accuracy_n": 2000,
+          "brier_soft": 0.07282817778125876,
           "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.13635658195747785,
           "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.18626503447270873,
           "total_variation_n": 2000,
+          "score_mae": 0.2670681801272688,
           "score_mae_n": 800,
+          "within_one_level": 0.98125,
           "within_one_level_n": 800,
+          "macro_f1": 0.6211545025911077
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.9416666666666667,
+          "accuracy_valid": 0.9416666666666667,
+          "ece_top_label": 0.032780915974200804,
+          "mean_confidence": 0.9120336837609119,
+          "brier_hard": 0.0911117485859852,
           "brier_hard_n": 600,
+          "nll_hard": 0.18384446182116462,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.9391643390047506
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.585,
+          "accuracy_valid": 0.585,
+          "ece_top_label": 0.3290210797041948,
+          "mean_confidence": 0.9044507912979094,
+          "brier_hard": 0.7352614358764159,
           "brier_hard_n": 600,
+          "nll_hard": 2.325969372091789,
           "nll_hard_n": 600,
+          "zero_probability_gold": 0.01,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.4846329953255675
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8083333333333333,
+          "accuracy_valid": 0.8083333333333333,
+          "ece_top_label": 0.09874549999999996,
+          "mean_confidence": 0.9070788333333333,
+          "brier_hard": 0.2864656637,
           "brier_hard_n": 600,
+          "nll_hard": 0.45967158478397707,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.7946275764565816
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.38166666666666665,
+          "accuracy_valid": 0.38166666666666665,
+          "ece_top_label": 0.23314279850468647,
+          "mean_confidence": 0.614809465171353,
+          "brier_hard": 0.7997842513426261,
           "brier_hard_n": 600,
+          "nll_hard": 1.7175170777887119,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "score_mae": 0.9144952235712186,
           "score_mae_n": 600,
+          "within_one_level": 0.6466666666666666,
           "within_one_level_n": 600,
+          "macro_f1": 0.3319436236284081
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.6982758620689655,
+          "accuracy_valid": 0.6982758620689655,
+          "ece_top_label": 0.25387413793103447,
+          "mean_confidence": 0.95215,
+          "brier_hard": 0.5115509806896552,
           "brier_hard_n": 116,
+          "nll_hard": 2.710472238032788,
           "nll_hard_n": 116,
+          "zero_probability_gold": 0.06896551724137931,
           "zero_probability_gold_n": 116,
+          "macro_f1": 0.6809931641392315
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.79,
+          "accuracy_valid": 0.79,
+          "ece_top_label": 0.16259349215790048,
+          "mean_confidence": 0.9477817275562629,
+          "brier_hard": 0.3655905743529746,
           "brier_hard_n": 300,
+          "nll_hard": 2.4460407722040176,
           "nll_hard_n": 300,
+          "zero_probability_gold": 0.06,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.5695724937908466
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.89,
+          "accuracy_valid": 0.89,
+          "ece_top_label": 0.05910627327283609,
+          "mean_confidence": 0.9123050519493905,
+          "brier_hard": 0.18251872229879984,
           "brier_hard_n": 300,
+          "nll_hard": 0.33404125735595414,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.8905240397272149
         }
       }
     },
     "1": {
       "seed": 1,
+      "selected_epoch": 9,
+      "elapsed_s": 541.4585086829611,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
         "decisions": 600
       },
       "selected_validation": {
+        "soft_cross_entropy": 0.8583925066391627,
+        "accuracy": 0.7466666666666667,
+        "brier_soft": 0.07437195796364297,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
+        "epochs_completed": 12,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7565,
+          "accuracy_valid": 0.7565,
+          "ece_top_label": 0.19864268503011012,
+          "mean_confidence": 0.5578573149698899,
+          "brier_hard": 0.40482086772099385,
           "brier_hard_n": 2000,
+          "nll_hard": 0.7162594531200565,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.469737413364262,
           "soft_accuracy_n": 2000,
+          "brier_soft": 0.0659469654317038,
           "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.12376910578634269,
           "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.17750424596946363,
           "total_variation_n": 2000,
+          "score_mae": 0.24687533223920302,
           "score_mae_n": 800,
+          "within_one_level": 0.98625,
           "within_one_level_n": 800,
+          "macro_f1": 0.6300073343403317
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.9483333333333334,
+          "accuracy_valid": 0.9483333333333334,
+          "ece_top_label": 0.03462398162033311,
+          "mean_confidence": 0.9188607247165506,
+          "brier_hard": 0.08433672632610516,
           "brier_hard_n": 600,
+          "nll_hard": 0.17417200848974151,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.9456145576151846
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.5716666666666667,
+          "accuracy_valid": 0.5716666666666667,
+          "ece_top_label": 0.33449032496578573,
+          "mean_confidence": 0.90098409027152,
+          "brier_hard": 0.7328256742449804,
           "brier_hard_n": 600,
+          "nll_hard": 2.388523752936007,
           "nll_hard_n": 600,
+          "zero_probability_gold": 0.013333333333333334,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.4746076611364094
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8266666666666667,
+          "accuracy_valid": 0.8266666666666667,
+          "ece_top_label": 0.08269850000000001,
+          "mean_confidence": 0.9093651666666667,
+          "brier_hard": 0.26852242536666665,
           "brier_hard_n": 600,
+          "nll_hard": 0.44571515698724307,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.8140998140998141
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.4483333333333333,
+          "accuracy_valid": 0.4483333333333333,
+          "ece_top_label": 0.1552169576555402,
+          "mean_confidence": 0.5963372951766569,
+          "brier_hard": 0.715830700927659,
           "brier_hard_n": 600,
+          "nll_hard": 1.4312799777899081,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "score_mae": 0.763809024216731,
           "score_mae_n": 600,
+          "within_one_level": 0.7216666666666667,
           "within_one_level_n": 600,
+          "macro_f1": 0.4241661418376961
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7844827586206896,
+          "accuracy_valid": 0.7844827586206896,
+          "ece_top_label": 0.17151379310344822,
+          "mean_confidence": 0.9531103448275862,
+          "brier_hard": 0.36615182827586207,
           "brier_hard_n": 116,
+          "nll_hard": 1.9254142815070245,
           "nll_hard_n": 116,
+          "zero_probability_gold": 0.05172413793103448,
           "zero_probability_gold_n": 116,
+          "macro_f1": 0.7785414280259642
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7566666666666667,
+          "accuracy_valid": 0.7566666666666667,
+          "ece_top_label": 0.19351909840865464,
+          "mean_confidence": 0.9501857650753213,
+          "brier_hard": 0.4290677996735415,
           "brier_hard_n": 300,
+          "nll_hard": 3.2153508464750185,
           "nll_hard_n": 300,
+          "zero_probability_gold": 0.08666666666666667,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.4943833232448793
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.87,
+          "accuracy_valid": 0.87,
+          "ece_top_label": 0.07153730747712309,
+          "mean_confidence": 0.9147377091848204,
+          "brier_hard": 0.22017864577608134,
           "brier_hard_n": 300,
+          "nll_hard": 0.4059236764304257,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.8710695955086495
         }
       }
     },
     "2": {
       "seed": 2,
+      "selected_epoch": 13,
+      "elapsed_s": 699.7859417049913,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
         "decisions": 600
       },
       "selected_validation": {
+        "soft_cross_entropy": 0.8543428432941437,
+        "accuracy": 0.7583333333333333,
+        "brier_soft": 0.07187043125430743,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
+        "epochs_completed": 16,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.745,
+          "accuracy_valid": 0.745,
+          "ece_top_label": 0.18594273050266366,
+          "mean_confidence": 0.5591527012331047,
+          "brier_hard": 0.40870440390133495,
           "brier_hard_n": 2000,
+          "nll_hard": 0.7205417712302097,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.46875735649306255,
           "soft_accuracy_n": 2000,
+          "brier_soft": 0.06782387317668824,
           "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.12687181845569698,
           "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.17935579359201664,
           "total_variation_n": 2000,
+          "score_mae": 0.25587944352517583,
           "score_mae_n": 800,
+          "within_one_level": 0.99,
           "within_one_level_n": 800,
+          "macro_f1": 0.608344198168488
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.945,
+          "accuracy_valid": 0.945,
+          "ece_top_label": 0.03201179295141958,
+          "mean_confidence": 0.9180146184255912,
+          "brier_hard": 0.08699114920264774,
           "brier_hard_n": 600,
+          "nll_hard": 0.17458965637122142,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.9421288699146882
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.56,
+          "accuracy_valid": 0.56,
+          "ece_top_label": 0.3501647005433899,
+          "mean_confidence": 0.9077857005433899,
+          "brier_hard": 0.7581475668473461,
           "brier_hard_n": 600,
+          "nll_hard": 2.427279492608694,
           "nll_hard_n": 600,
+          "zero_probability_gold": 0.013333333333333334,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.4667304743466239
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8183333333333334,
+          "accuracy_valid": 0.8183333333333334,
+          "ece_top_label": 0.08817149999999999,
+          "mean_confidence": 0.9065048333333333,
+          "brier_hard": 0.2762812369666667,
           "brier_hard_n": 600,
+          "nll_hard": 0.453869636876017,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.8046122269724754
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.3983333333333333,
+          "accuracy_valid": 0.3983333333333333,
+          "ece_top_label": 0.21926419101950614,
+          "mean_confidence": 0.6138282869290971,
+          "brier_hard": 0.7663539453668811,
           "brier_hard_n": 600,
+          "nll_hard": 1.5341982352412973,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "score_mae": 0.7954480085445932,
           "score_mae_n": 600,
+          "within_one_level": 0.695,
           "within_one_level_n": 600,
+          "macro_f1": 0.3618869575527743
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7155172413793104,
+          "accuracy_valid": 0.7155172413793104,
+          "ece_top_label": 0.2621448275862069,
+          "mean_confidence": 0.9488327586206896,
+          "brier_hard": 0.4988274203448276,
           "brier_hard_n": 116,
+          "nll_hard": 2.61023697496577,
           "nll_hard_n": 116,
+          "zero_probability_gold": 0.06896551724137931,
           "zero_probability_gold_n": 116,
+          "macro_f1": 0.6992221261884184
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.75,
+          "accuracy_valid": 0.75,
+          "ece_top_label": 0.20336797564979028,
+          "mean_confidence": 0.9533679756497903,
+          "brier_hard": 0.4358421604675745,
           "brier_hard_n": 300,
+          "nll_hard": 3.1322756175288426,
           "nll_hard_n": 300,
+          "zero_probability_gold": 0.08,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.5034639202079357
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8733333333333333,
+          "accuracy_valid": 0.8733333333333333,
+          "ece_top_label": 0.05108680216338136,
+          "mean_confidence": 0.914837261771426,
+          "brier_hard": 0.20561652153986584,
           "brier_hard_n": 300,
+          "nll_hard": 0.3742216981370551,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.8742255731346232
         }
       }
     },
     "3": {
       "seed": 3,
+      "selected_epoch": 8,
+      "elapsed_s": 495.011001034989,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
         "decisions": 600
       },
       "selected_validation": {
+        "soft_cross_entropy": 0.8545633804798126,
+        "accuracy": 0.77,
+        "brier_soft": 0.06962224062221746,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
+        "epochs_completed": 11,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7655,
+          "accuracy_valid": 0.7655,
+          "ece_top_label": 0.2096452515650509,
+          "mean_confidence": 0.5558547484349491,
+          "brier_hard": 0.4011842616732685,
           "brier_hard_n": 2000,
+          "nll_hard": 0.7113399953872087,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.4683026193430771,
           "soft_accuracy_n": 2000,
+          "brier_soft": 0.06585425455571205,
           "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.12507067261764912,
           "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.17848467876854548,
           "total_variation_n": 2000,
+          "score_mae": 0.24395106365305969,
           "score_mae_n": 800,
+          "within_one_level": 0.98375,
           "within_one_level_n": 800,
+          "macro_f1": 0.6399493527359326
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.9366666666666666,
+          "accuracy_valid": 0.9366666666666666,
+          "ece_top_label": 0.0312034834128748,
+          "mean_confidence": 0.914535932360556,
+          "brier_hard": 0.09381807669545053,
           "brier_hard_n": 600,
+          "nll_hard": 0.1866731647377733,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.9336084571946038
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.565,
+          "accuracy_valid": 0.565,
+          "ece_top_label": 0.3370553794856075,
+          "mean_confidence": 0.8987647229086076,
+          "brier_hard": 0.7489678137408704,
           "brier_hard_n": 600,
+          "nll_hard": 2.3318777367931305,
           "nll_hard_n": 600,
+          "zero_probability_gold": 0.011666666666666667,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.4760859469034663
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.82,
+          "accuracy_valid": 0.82,
+          "ece_top_label": 0.09564866666666666,
+          "mean_confidence": 0.9069513333333333,
+          "brier_hard": 0.2824305872,
           "brier_hard_n": 600,
+          "nll_hard": 0.4619444556248629,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.8069498069498069
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.4116666666666667,
+          "accuracy_valid": 0.4116666666666667,
+          "ece_top_label": 0.19022023187751014,
+          "mean_confidence": 0.6018868985441768,
+          "brier_hard": 0.7571439809814903,
           "brier_hard_n": 600,
+          "nll_hard": 1.5683213482322946,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "score_mae": 0.8321864596740475,
           "score_mae_n": 600,
+          "within_one_level": 0.6783333333333333,
           "within_one_level_n": 600,
+          "macro_f1": 0.3816445513227996
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7844827586206896,
+          "accuracy_valid": 0.7844827586206896,
+          "ece_top_label": 0.17475344827586206,
+          "mean_confidence": 0.9523568965517242,
+          "brier_hard": 0.3675763103448276,
           "brier_hard_n": 116,
+          "nll_hard": 1.8418113090371786,
           "nll_hard_n": 116,
+          "zero_probability_gold": 0.04310344827586207,
           "zero_probability_gold_n": 116,
+          "macro_f1": 0.7797524113313588
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7733333333333333,
+          "accuracy_valid": 0.7733333333333333,
+          "ece_top_label": 0.17472328670404882,
+          "mean_confidence": 0.9480566200373822,
+          "brier_hard": 0.38383190196466177,
           "brier_hard_n": 300,
+          "nll_hard": 2.4090375945333213,
           "nll_hard_n": 300,
+          "zero_probability_gold": 0.05,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.502438924453748
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8666666666666667,
+          "accuracy_valid": 0.8666666666666667,
+          "ece_top_label": 0.054425126999604626,
+          "mean_confidence": 0.9150511806609052,
+          "brier_hard": 0.2024701804692983,
           "brier_hard_n": 300,
+          "nll_hard": 0.3723722500840691,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.8674417053037536
         }
       }
     },
     "4": {
       "seed": 4,
+      "selected_epoch": 8,
+      "elapsed_s": 496.1691781419795,
       "initial_validation": {
         "soft_cross_entropy": 1.544869564374288,
         "accuracy": 0.36833333333333335,
         "decisions": 600
       },
       "selected_validation": {
+        "soft_cross_entropy": 0.8540202794472377,
+        "accuracy": 0.76,
+        "brier_soft": 0.07019127607345581,
         "decisions": 600
       },
       "stopping": {
         "reason": "early_stopping",
+        "epochs_completed": 11,
         "patience": 3,
         "min_epochs": 10,
         "epochs_without_improvement": 3,
           "valid": 2000,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7545,
+          "accuracy_valid": 0.7545,
+          "ece_top_label": 0.2042592508573846,
+          "mean_confidence": 0.5504920240151281,
+          "brier_hard": 0.4113014025244486,
           "brier_hard_n": 2000,
+          "nll_hard": 0.7247566141198507,
           "nll_hard_n": 2000,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 2000,
+          "soft_accuracy": 0.46590212671846004,
           "soft_accuracy_n": 2000,
+          "brier_soft": 0.06812301484891332,
           "brier_soft_n": 2000,
+          "kl_gold_to_prediction": 0.12790407031921358,
           "kl_gold_to_prediction_n": 2000,
+          "total_variation": 0.18054231568172208,
           "total_variation_n": 2000,
+          "score_mae": 0.25377940453843617,
           "score_mae_n": 800,
+          "within_one_level": 0.98375,
           "within_one_level_n": 800,
+          "macro_f1": 0.6337422972118567
         },
         "ag-news": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.9516666666666667,
+          "accuracy_valid": 0.9516666666666667,
+          "ece_top_label": 0.03233758800344756,
+          "mean_confidence": 0.9199325508493339,
+          "brier_hard": 0.08178925980745573,
           "brier_hard_n": 600,
+          "nll_hard": 0.16974235809511878,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.9489828512364638
         },
         "emotion": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.5783333333333334,
+          "accuracy_valid": 0.5783333333333334,
+          "ece_top_label": 0.3295382068772924,
+          "mean_confidence": 0.9035819218093762,
+          "brier_hard": 0.7425345488860924,
           "brier_hard_n": 600,
+          "nll_hard": 2.3910245621280453,
           "nll_hard_n": 600,
+          "zero_probability_gold": 0.013333333333333334,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.48527808236443093
         },
         "boolq": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8316666666666667,
+          "accuracy_valid": 0.8316666666666667,
+          "ece_top_label": 0.09521133333333336,
+          "mean_confidence": 0.9097233333333333,
+          "brier_hard": 0.27490432886666666,
           "brier_hard_n": 600,
+          "nll_hard": 0.49707214589381826,
           "nll_hard_n": 600,
+          "zero_probability_gold": 0.0016666666666666668,
           "zero_probability_gold_n": 600,
+          "macro_f1": 0.8196294367140413
         },
         "sst5": {
           "attempted": 600,
           "valid": 600,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.41,
+          "accuracy_valid": 0.41,
+          "ece_top_label": 0.20145470242332547,
+          "mean_confidence": 0.6098530357566588,
+          "brier_hard": 0.7512849781669847,
           "brier_hard_n": 600,
+          "nll_hard": 1.5195838995022448,
           "nll_hard_n": 600,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 600,
+          "score_mae": 0.8012926973404637,
           "score_mae_n": 600,
+          "within_one_level": 0.7016666666666667,
           "within_one_level_n": 600,
+          "macro_f1": 0.37608916605464326
         },
         "prompt-injections": {
           "attempted": 116,
           "valid": 116,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.7413793103448276,
+          "accuracy_valid": 0.7413793103448276,
+          "ece_top_label": 0.21426120689655168,
+          "mean_confidence": 0.9556405172413793,
+          "brier_hard": 0.43549763362068966,
           "brier_hard_n": 116,
+          "nll_hard": 2.2934410633534186,
           "nll_hard_n": 116,
+          "zero_probability_gold": 0.0603448275862069,
           "zero_probability_gold_n": 116,
+          "macro_f1": 0.7276995305164319
         },
         "massive-intent.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.77,
+          "accuracy_valid": 0.77,
+          "ece_top_label": 0.17971198952007902,
+          "mean_confidence": 0.9447345124535086,
+          "brier_hard": 0.4058348152163361,
           "brier_hard_n": 300,
+          "nll_hard": 2.9106532352305194,
           "nll_hard_n": 300,
+          "zero_probability_gold": 0.07333333333333333,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.5030980723143226
         },
         "xnli.en": {
           "attempted": 300,
           "valid": 300,
           "failed": 0,
           "coverage": 1.0,
+          "accuracy_all": 0.8633333333333333,
+          "accuracy_valid": 0.8633333333333333,
+          "ece_top_label": 0.0751956003124493,
+          "mean_confidence": 0.9162548569668819,
+          "brier_hard": 0.22384958324541668,
           "brier_hard_n": 300,
+          "nll_hard": 0.4127381776210696,
           "nll_hard_n": 300,
           "zero_probability_gold": 0.0,
           "zero_probability_gold_n": 300,
+          "macro_f1": 0.864378848443158
         }
       }
     }
   "summary": {
     "typed-decisions": {
       "n": 5,
+      "mean_percent": 75.28,
+      "sample_variance_pp_squared": 0.8619999999999957,
+      "sample_std_pp": 0.9284395510748105
     },
     "ag-news": {
       "n": 5,
+      "mean_percent": 94.46666666666667,
+      "sample_variance_pp_squared": 0.3388888888888897,
+      "sample_std_pp": 0.5821416398857667
     },
     "boolq": {
       "n": 5,
+      "mean_percent": 82.10000000000001,
+      "sample_variance_pp_squared": 0.7861111111111168,
+      "sample_std_pp": 0.8866290718846956
     },
     "emotion": {
       "n": 5,
+      "mean_percent": 57.2,
+      "sample_variance_pp_squared": 1.0055555555555546,
+      "sample_std_pp": 1.0027739304327543
     },
     "prompt-injections": {
       "n": 5,
+      "mean_percent": 74.48275862068965,
+      "sample_variance_pp_squared": 15.457788347205712,
+      "sample_std_pp": 3.93163939689358
     },
     "sst5": {
       "n": 5,
+      "mean_percent": 41.0,
+      "sample_variance_pp_squared": 6.027777777777775,
+      "sample_std_pp": 2.4551533104427055
     },
     "massive-intent.en": {
       "n": 5,
+      "mean_percent": 76.8,
+      "sample_variance_pp_squared": 2.422222222222218,
+      "sample_std_pp": 1.556349003990499
     },
     "xnli.en": {
       "n": 5,
+      "mean_percent": 87.26666666666667,
+      "sample_variance_pp_squared": 1.0777777777777784,
+      "sample_std_pp": 1.038160766826496
     }
   },
+  "candidate_seed": 4,
   "variance_definition": "sample variance, n-1, percentage points squared",
   "benchmark_manifest": {
     "format_version": 1,

release_manifest.json ADDED Viewed

	@@ -0,0 +1,54 @@

+{
+  "published": false,
+  "review_status": "export verified; package prepared for user review and upload",
+  "files_sha256": {
+    "LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
+    "README.md": "db6038098fae3817f20fdabc66afb13d7e7cd1306d0b611eea7f70ef84ad08bc",
+    "THIRD_PARTY.md": "3dd2683344d45ce64c15f45c723f4b0db720deee84446365fdf4084a73ebeceb",
+    "USAGE.md": "038e1d21dca7d2e2da5b086a34e5b446edb8901406609c8c8b0e3407f31ab7d9",
+    "adapter.safetensors": "f23ad342a969773bbbcb18619ce8a8843bbb105d94fe64c5a837d68f26829081",
+    "adapter_config.json": "b26f0cb6ea6c6f1e66f254b832d5000b245d309fc05880f234d0f7489cadb54a",
+    "ariadne_bench/__init__.py": "a73c9fdf21413382cb93b523bfb3925aa3fd354e18faeead4650e86a30645411",
+    "ariadne_bench/__main__.py": "935a1c1166b0c1ea35a82256345000bf2c73ded718d77773bc27a71ecce28f7d",
+    "ariadne_bench/adapters.py": "3bd4820165be077b3b98fe83ac23e649e1ad69186ad5f2a4f5ee7addf1427cf5",
+    "ariadne_bench/cli.py": "de267c0502489eb4bb37e4b8f6faa8da8be37a56ced270f754424a40404d1af6",
+    "ariadne_bench/datasets.py": "31c722c64895519a4635b501da88862b446384682e3e8ee884495941d6990912",
+    "ariadne_bench/experiments/__init__.py": "95ad0a7f30c4e0a194d0a6e2ae8905abae5d622edbb3d4e5d791b4bc460aee5d",
+    "ariadne_bench/experiments/align.py": "31e2733dcc223a1e1a6250d41c4158f84091f1cb4067d61388e06b5b01ae34c4",
+    "ariadne_bench/experiments/prepare_alignment.py": "c4f39c10ce46ee13c0d89128d18a85ec4a86729e81e070f5834ed01fc1c405a9",
+    "ariadne_bench/experiments/spectrum.py": "22d4801b585c539d0d579a9d2a00076e1002fd337bfe31d1906d8ae3fb8d4320",
+    "ariadne_bench/frozen_input_interface.py": "253f0186dac6e4a7f4b0fcf3c33449c3b3882cf8be538dca6eeca8d9e4ce9b83",
+    "ariadne_bench/full_finetune.py": "6067be8cf837d254b919c841c9a9851e1546a413dbf6324ff242e5d2f56c9734",
+    "ariadne_bench/full_input_interface.py": "a8f440a2185085c64896300a243ac52a9619c4d05a1b4e43c50c2c32aa1da996",
+    "ariadne_bench/hybrid.py": "90fbbdab9b8157a32854d27c2d93089be6dc88dfe9320426f102eae4e03d9ffd",
+    "ariadne_bench/hybrid_control.py": "1b53678bea73fe7608570b78d7215d1dff2e38d6ad3627615f26c06ae74d8b4c",
+    "ariadne_bench/interfaces.py": "19700fb8170b2460401900cb96ebdff22bd7bf778ca5132e98d56ba68f344187",
+    "ariadne_bench/metrics.py": "fc6fde20c95d5052de52d05d6fb9b6eb69b780c46966f5bf1e517738b276a4b4",
+    "ariadne_bench/portable.py": "259f633a8bb6663d0594d9639ea8d80ef493f5a63d4b11444a4003f2f9a318e8",
+    "ariadne_bench/reproducibility.py": "5b0c161c596f25277b3326c645df53d5d1f2dc0272da9e77bc4a4c0d56188b6e",
+    "ariadne_bench/runner.py": "b20e2af4ba254e2d48a07563356f95728c4807018c3c989b429e9922674c64cd",
+    "ariadne_bench/schema.py": "67794f1172c1ef9d9625b84c8096fd48b8fa7cd10e3f88fc13ba3a98ef7c7a76",
+    "benchmarks/sources.lock.json": "6e6eaab2778092c8127257fce202c9fa8d4625dd0d029490dd9f137da91d134c",
+    "environment.json": "c5b9c796d97f3dec6d242caf79aa5d2ac69c91717f2be6ed5a51627bf6d58c17",
+    "evidence/adapter-equivalence.json": "7ffdc6216702ab5de02a1745217b369de9f2f8934e16e5c80dd090a3343e9e84",
+    "evidence/audit.json": "7a2954a470b3f94626ca678aef7f9bf1f13551be20e642c1a16a0a90b751bd4d",
+    "evidence/candidate-uncertainty.json": "9e97410249531de151864285412ab3bd337b117b3573f6439cf74ae17926cae3",
+    "evidence/identity-check.json": "0cc8904dc58e1e0ed49496c7ae01980c52e3a1395d61ef91286cec2c9dcdfa25",
+    "evidence/lr-comparison.json": "724a08155ff3f39a335132cffb0248fdb08f06874e84ce6a582c8af9199ac69d",
+    "evidence/replay-check.json": "d362a58603077feddc4ad2513c19f8caea25da1a9ab03eec25c5be5c14c3eec0",
+    "evidence/seed-0-history.json": "1090a62e8f1256e9510f8c2c913de14954a5e54e14396a962578ebef83b4a6f3",
+    "evidence/seed-1-history.json": "6aee05fe087ec5d970329743a06c1cbff3e2a0e8209cdcb2db41649b66b75432",
+    "evidence/seed-2-history.json": "71afe93da2e39ef81417074bc1a4559b9d6f070f677e76f74818129633cf3c33",
+    "evidence/seed-3-history.json": "4bf723c1659ffc8adeb57d06d51132f4f5af74c811c98feef71de4f6fa3bd66f",
+    "evidence/seed-4-history.json": "2609c807813f55ffe5aef7e6c96aafefe1647de1ac589568d7fdc56e46f5c9fe",
+    "export-verification.json": "65e46623219d80979a7108e69c0ed89d0b1d5c23a82c4255fee64647694aac3f",
+    "licenses/jev-benchmarks-LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
+    "licenses/laya-LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
+    "load_adapter.py": "2f305ef95fef2b2482e9214836fc0071a59cfd6c6d3aaf42cebdbfb131272081",
+    "make_training_config.py": "23e9d4acb90141a3d690bc607c861df92137b9457304ae19b8a925398b55536d",
+    "metrics.json": "e3cd35993afc309f7ccd99b1a3d96316944e1ab42fd489c7f15465f43ec4a78b",
+    "requirements.txt": "285a8ae6529abeefd058f2e9679e0e314b19fda38169e685ea16aec59b053bd9",
+    "training_protocol.json": "00b2db7abbd07ba3220a130fe36ed64958f1dfac15773d183f1ae8dc9d8334b3",
+    "upload-manifest.json": "613b92d535a5c8fca8dfe1d6535600f6da68ba0bf110b53829d13638614830b6"
+  }
+}

training_protocol.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "experiment": "E0i",
   "seeds": [
     0,
     1,
@@ -7,7 +7,7 @@
     3,
     4
   ],
-  "created_at": "2026-09-27T13:49:50.006241+00:00",
   "data": {
     "train": {
       "format_version": 1,
@@ -67,7 +67,7 @@
   "microbatch": 8,
   "accumulation": 4,
   "effective_batch": 32,
-  "interface_lr": 0.0003,
   "weight_decay": 0.01,
   "gradient_clip": 1,
   "training_scope": "bridge",
@@ -77,8 +77,23 @@
   "device": "cuda:1",
   "deterministic": true,
   "native_sdk_calibration": true,
-  "publish_policy": "draft locally; upload only after user and assistant agree",
   "baseline_predictions_sha256": "37dff666a00a340cd6e37f775dc60df33a3c6eb78f76bfee7f094443c225b6e8",
   "base_model": "convaiinnovations/laya",
   "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
   "training_sources_sha256": {

 {
+  "experiment": "E0j five-seed confirmation",
   "seeds": [
     0,
     1,
     3,
     4
   ],
+  "created_at": "2026-09-27T15:29:06.749690+00:00",
   "data": {
     "train": {
       "format_version": 1,
   "microbatch": 8,
   "accumulation": 4,
   "effective_batch": 32,
+  "interface_lr": 0.0001,
   "weight_decay": 0.01,
   "gradient_clip": 1,
   "training_scope": "bridge",
   "device": "cuda:1",
   "deterministic": true,
   "native_sdk_calibration": true,
+  "publish_policy": "provide user-requested local bundle; preserve existing card; no assistant upload",
   "baseline_predictions_sha256": "37dff666a00a340cd6e37f775dc60df33a3c6eb78f76bfee7f094443c225b6e8",
+  "run_order": [
+    0,
+    2,
+    3,
+    1,
+    4
+  ],
+  "seed_0_reference": "completed controlled pilot, unchanged",
+  "comparison": "LR is the only training change from E0i; min10/patience3 and all other settings fixed",
+  "scope": "adaptive experimental follow-up selected after the seed-0 pilot; not a fresh blind test",
+  "replay_evidence": "Two fresh seed-0 LR 1e-4 processes matched losses, metrics and tensors for two epochs; reported seed-0 metric prefix matched. Verified after the sweep.",
+  "driver_sha256": {
+    "/mnt/storage/ariadne/scripts/run_frozen_linear_lr_confirmation.py": "f7c3841277682cbae43f4eb51fa3cb5fc87fd53068607206240ce137b7e94a9b"
+  },
+  "release_replay_sha256": "dd3479114e8af11d06708febc1e86b887cc6e3cc5ce1523a249e54b9432412ec",
   "base_model": "convaiinnovations/laya",
   "base_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
   "training_sources_sha256": {

upload-manifest.json CHANGED Viewed

@@ -1,17 +1,17 @@
 {
   "published_by_assistant": false,
-  "candidate_seed": 1,
-  "selected_epoch": 14,
   "format": "compact affine adapter requiring pinned base Laya",
-  "existing_model_card_preserved": true,
-  "inference_code_and_weights_match_verified_export": true,
-  "removed_only_other_seed_loading_entries": true,
   "files_sha256": {
     "LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
     "THIRD_PARTY.md": "3dd2683344d45ce64c15f45c723f4b0db720deee84446365fdf4084a73ebeceb",
-    "USAGE.md": "e6d76a3a450fd589e52cab5190a065cbd783082a98d3524071bcf60a7015ebef",
-    "adapter.safetensors": "e5ec5e1d494509f63960b6949af15be8390442271805ba45d796a1911b5c239a",
-    "adapter_config.json": "8d7450b812f696e1dbeb3d7abfa13fbee0cde95aae0349a94e5f22c9b458ed1f",
     "ariadne_bench/__init__.py": "a73c9fdf21413382cb93b523bfb3925aa3fd354e18faeead4650e86a30645411",
     "ariadne_bench/__main__.py": "935a1c1166b0c1ea35a82256345000bf2c73ded718d77773bc27a71ecce28f7d",
     "ariadne_bench/adapters.py": "3bd4820165be077b3b98fe83ac23e649e1ad69186ad5f2a4f5ee7addf1427cf5",
@@ -34,22 +34,24 @@
     "ariadne_bench/schema.py": "67794f1172c1ef9d9625b84c8096fd48b8fa7cd10e3f88fc13ba3a98ef7c7a76",
     "benchmarks/sources.lock.json": "6e6eaab2778092c8127257fce202c9fa8d4625dd0d029490dd9f137da91d134c",
     "environment.json": "c5b9c796d97f3dec6d242caf79aa5d2ac69c91717f2be6ed5a51627bf6d58c17",
-    "evidence/adapter-equivalence.json": "c77faa691bd273d363dbb196155545b4f72697b9d0f4c80451ffb06b0090d200",
-    "evidence/audit.json": "1fb474a47923e22374a47e06f545e747420c0a7bd269bbbeaabb5869e89a8d70",
-    "evidence/candidate-uncertainty.json": "39eba4a205eaea5f13384da24ca8563de364c9edb6858c1db38eb4111960ce40",
     "evidence/identity-check.json": "0cc8904dc58e1e0ed49496c7ae01980c52e3a1395d61ef91286cec2c9dcdfa25",
-    "evidence/replay-check.json": "4f246eb3d6afc0c0b1d306a20b82060ae564d476ec22451322b065edb946d15e",
-    "evidence/seed-0-history.json": "bc3de468970bf350426d9c4281456e258984838feb4d8dbbfe54c8752ebb24b7",
-    "evidence/seed-1-history.json": "8ee78b5685138712512b39035ef48a491ed52813c2131a8e9b80715871ab9405",
-    "evidence/seed-2-history.json": "d9bf7ab3991b85b6e3877b9ebd4ca2fc113cba1716f56da5aa16f4018ac4b197",
-    "evidence/seed-3-history.json": "7c0f19638e31f28e438af879ceaa68bbe06480e6f7fb72e8e749b4aabb304341",
-    "evidence/seed-4-history.json": "601c5bd91dcbb004ff8a07621f3f125ba578cc4107a9884e8fe69ac17751a741",
-    "export-verification.json": "b9f8667626e2facd03f310c0c67c100ddde9481218c51cbead7972fd048d4afb",
     "licenses/jev-benchmarks-LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
     "licenses/laya-LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
     "load_adapter.py": "2f305ef95fef2b2482e9214836fc0071a59cfd6c6d3aaf42cebdbfb131272081",
-    "metrics.json": "dd45292f04dc5b81cc958a39f618aa8748b5198e39b5effee6c95520c92d94b6",
     "requirements.txt": "285a8ae6529abeefd058f2e9679e0e314b19fda38169e685ea16aec59b053bd9",
-    "training_protocol.json": "dccf81e3d8a70c756a95a88280cbec139eea43bc391d67fa7b40e384a8824f57"
   }
 }

 {
   "published_by_assistant": false,
+  "candidate_seed": 4,
+  "selected_epoch": 8,
   "format": "compact affine adapter requiring pinned base Laya",
+  "includes_model_card": true,
+  "all_5116_export_answers_exact": true,
   "files_sha256": {
     "LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
+    "README.md": "db6038098fae3817f20fdabc66afb13d7e7cd1306d0b611eea7f70ef84ad08bc",
     "THIRD_PARTY.md": "3dd2683344d45ce64c15f45c723f4b0db720deee84446365fdf4084a73ebeceb",
+    "USAGE.md": "038e1d21dca7d2e2da5b086a34e5b446edb8901406609c8c8b0e3407f31ab7d9",
+    "adapter.safetensors": "f23ad342a969773bbbcb18619ce8a8843bbb105d94fe64c5a837d68f26829081",
+    "adapter_config.json": "b26f0cb6ea6c6f1e66f254b832d5000b245d309fc05880f234d0f7489cadb54a",
     "ariadne_bench/__init__.py": "a73c9fdf21413382cb93b523bfb3925aa3fd354e18faeead4650e86a30645411",
     "ariadne_bench/__main__.py": "935a1c1166b0c1ea35a82256345000bf2c73ded718d77773bc27a71ecce28f7d",
     "ariadne_bench/adapters.py": "3bd4820165be077b3b98fe83ac23e649e1ad69186ad5f2a4f5ee7addf1427cf5",
     "ariadne_bench/schema.py": "67794f1172c1ef9d9625b84c8096fd48b8fa7cd10e3f88fc13ba3a98ef7c7a76",
     "benchmarks/sources.lock.json": "6e6eaab2778092c8127257fce202c9fa8d4625dd0d029490dd9f137da91d134c",
     "environment.json": "c5b9c796d97f3dec6d242caf79aa5d2ac69c91717f2be6ed5a51627bf6d58c17",
+    "evidence/adapter-equivalence.json": "7ffdc6216702ab5de02a1745217b369de9f2f8934e16e5c80dd090a3343e9e84",
+    "evidence/audit.json": "7a2954a470b3f94626ca678aef7f9bf1f13551be20e642c1a16a0a90b751bd4d",
+    "evidence/candidate-uncertainty.json": "9e97410249531de151864285412ab3bd337b117b3573f6439cf74ae17926cae3",
     "evidence/identity-check.json": "0cc8904dc58e1e0ed49496c7ae01980c52e3a1395d61ef91286cec2c9dcdfa25",
+    "evidence/lr-comparison.json": "724a08155ff3f39a335132cffb0248fdb08f06874e84ce6a582c8af9199ac69d",
+    "evidence/replay-check.json": "d362a58603077feddc4ad2513c19f8caea25da1a9ab03eec25c5be5c14c3eec0",
+    "evidence/seed-0-history.json": "1090a62e8f1256e9510f8c2c913de14954a5e54e14396a962578ebef83b4a6f3",
+    "evidence/seed-1-history.json": "6aee05fe087ec5d970329743a06c1cbff3e2a0e8209cdcb2db41649b66b75432",
+    "evidence/seed-2-history.json": "71afe93da2e39ef81417074bc1a4559b9d6f070f677e76f74818129633cf3c33",
+    "evidence/seed-3-history.json": "4bf723c1659ffc8adeb57d06d51132f4f5af74c811c98feef71de4f6fa3bd66f",
+    "evidence/seed-4-history.json": "2609c807813f55ffe5aef7e6c96aafefe1647de1ac589568d7fdc56e46f5c9fe",
+    "export-verification.json": "65e46623219d80979a7108e69c0ed89d0b1d5c23a82c4255fee64647694aac3f",
     "licenses/jev-benchmarks-LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
     "licenses/laya-LICENSE": "a6cba85bc92e0cff7a450b1d873c0eaa2e9fc96bf472df0247a26bec77bf3ff9",
     "load_adapter.py": "2f305ef95fef2b2482e9214836fc0071a59cfd6c6d3aaf42cebdbfb131272081",
+    "make_training_config.py": "23e9d4acb90141a3d690bc607c861df92137b9457304ae19b8a925398b55536d",
+    "metrics.json": "e3cd35993afc309f7ccd99b1a3d96316944e1ab42fd489c7f15465f43ec4a78b",
     "requirements.txt": "285a8ae6529abeefd058f2e9679e0e314b19fda38169e685ea16aec59b053bd9",
+    "training_protocol.json": "00b2db7abbd07ba3220a130fe36ed64958f1dfac15773d183f1ae8dc9d8334b3"
   }
 }