v0.2.0 reconciliation: banner, v3 result, shallow-baseline framing, checksums, model-index, provenance

- Add 'research benchmark, not for deployment' banner.
- Headline AP 0.9926 placed alongside all-frames LR AP 0.8762.
- Add v3 section: xor_order_color defeats v2 (LR 0.5486, Transformer flatlines AP 0.5068).
- Fix pipeline_tag tabular-classification -> time-series-classification.
- Add last_updated, model-index (verified v2 metrics only).
- Add SHA256 CHECKSUMS for all 10 shipped non-README artifacts.
- Add PROVENANCE.md mapping each model-index value to its source JSON.
- Add license clarification (synthetic data, no third-party dataset license).

CHECKSUMS.sha256

@@ -0,0 +1,10 @@
+7fb1523a96a0a7f23c4cf343fc6f5a27d7bd5fa4a844f92304b19afcca4d396e  artifact_index.json
+9e20b26b5cec972be2882f4cb1333bbc18d15335c0199c78ba682113704bfb03  claims_ledger.md
+8ac6b51e1449f19eac5edf1d4e5ce5d74c0b0f3a259c3ef8629c0d141999bbc3  assets/release_visual.svg
+29979a6d89071cf05e1b7ffa5ec27ae967430f24ef47b5e8d7601aef54625a93  baselines.json
+051c5426158bf0957cac905844222f38ef18ea3634ba78b47e7a6e9036adf1f8  baselines.md
+428c87622bcb6191c4ce2081d0543da0b9a3e4c9425c10168d9e83ba4a2cbb22  benchmark_v2_summary.json
+85c704597307aa2484bda87fd5215f35cc3217cc05d1dc8a6a978d8f1792a5a9  data_diagnostics.json
+361f8077142e9704fb6d45edf14c8d5c0b0c35e926a99487d78f6c2722b2ed67  data_diagnostics.md
+016dd558e2e08fa70cbb9b6f78bdd332e69d2416be78b13dad38affa7aa1574c  transformer_v2_35e_panel.json
+a85a02fe164e1493ccddc55d4275951a10422957e6ec62c15587be5a781d1a90  transformer_v2_minrecall080_fpr020.json

PROVENANCE.md

@@ -0,0 +1,70 @@
+# PROVENANCE — UI-PreFail-v2 reported metrics
+
+Every numeric value in the `model-index` block of `README.md` traces to a JSON
+file shipped in this repo or in the local PreFailureNet repo. The mapping is
+recorded here rather than inline in `model-index` so the YAML stays
+schema-compliant for Hugging Face's parser.
+
+Tag: `v0.2.0-reconciliation` (2026-05-25).
+
+## v2 Transformer (held-out test, n=256, 128 failure / 128 stable)
+
+Source file: `transformer_v2_35e_panel.json` (shipped in this repo)
+
+| Metric name on card | Field in JSON | Raw value |
+| --- | --- | --- |
+| AP (failure vs stable) | `ap_failure_vs_stable` | `0.9926430180293419` |
+| Macro F1 (used classes) | `macro_f1_used_only` | `0.9910648756727424` |
+| Pair-rank accuracy | `pair_rank_acc` | `1.0` |
+| Pair intervention success | `pair_intervention_success` | `0.984375` |
+| FPR at predicted class | `fpr_at_predicted_class` | `0.015625` |
+| Recall at predicted class | `recall_at_predicted_class` | `1.0` |
+
+## Gate calibration (risk_only, min_recall=0.80, max_FPR=0.20)
+
+Source file: `transformer_v2_minrecall080_fpr020.json` (shipped in this repo)
+
+| Metric name on card | Field in JSON | Raw value |
+| --- | --- | --- |
+| Gate test recall | `test_metrics.recall` | `1.0` |
+| Gate test FPR (false intervention rate) | `test_metrics.false_intervention_rate` | `0.0078125` |
+| Gate test precision | `test_metrics.precision` | `0.9922480620155039` |
+
+For reference (validation split, not on card):
+
+| Validation metric | Field | Raw value |
+| --- | --- | --- |
+| Recall | `validation_metrics.recall` | `0.9921875` |
+| FPR | `validation_metrics.false_intervention_rate` | `0.0` |
+| Precision | `validation_metrics.precision` | `1.0` |
+| Threshold | `validation_metrics.threshold` | `0.9435477256774902` |
+
+## Shallow baseline (all-frames LR) on v2
+
+Source file: `baselines.json` (shipped in this repo), entry `all_frames`.
+
+| Field | Raw value |
+| --- | --- |
+| `ap_failure_vs_stable` | `0.8761761121534359` |
+| `pair_rank_acc` | `0.8828125` |
+| `pair_intervention_success` | `0.6015625` |
+
+## v3 results (not shipped in this repo; cited in README v3 section)
+
+Sources in the local PreFailureNet working tree at
+`~/Projects/PreFailureNet/runs/benchmark_v3/`:
+
+| Card claim | Source file | Field | Raw value |
+| --- | --- | --- | --- |
+| v3 all-frames LR AP (rounded 0.5486 / 0.549 in docs) | `baselines_seed2920/baselines.json` (entry `all_frames`) | `ap_failure_vs_stable` | regenerable from local repo |
+| v3 Transformer precursor_detection_ap | `transformer_v3/metrics.json` | `precursor_detection_ap` | `0.5068114175400058` |
+| v3 Transformer intervention_auroc | `transformer_v3/metrics.json` | `intervention_auroc` | `0.500152587890625` |
+| v3 generator commit / implementation | local commit `3a323b1`, `src/prefailurenet/data/synthetic.py`, function `_temporal_order_v3_pair` | n/a | n/a |
+| v3 cross-reference doc | `docs/v3_split_evaluation.md` (rounds AP to `0.549`) | n/a | n/a |
+
+## Phase 1 audit reference
+
+Full ground-truth audit: `/tmp/hf_audit/AUDIT.md` (claim-by-claim verification
+against shipped JSONs). Phase 1 verified the numbers above as artifact-match,
+not as re-execution; no metric was re-trained or re-fit during the
+reconciliation.

README.md

@@ -2,7 +2,8 @@
 language: en
 license: apache-2.0
 library_name: pytorch
-pipeline_tag: tabular-classification
+pipeline_tag: time-series-classification
+last_updated: 2026-05-25
 tags:
   - benchmark
   - temporal-modeling
@@ -13,19 +14,88 @@ tags:
   - evaluation
   - browser-agents
   - ai-agents
+model-index:
+  - name: PreFailureNet UI-PreFail-v2 (Transformer, 35 epochs)
+    results:
+      - task:
+          type: time-series-classification
+          name: Pre-failure detection (synthetic ordered counterfactual, v2)
+        dataset:
+          type: synthetic
+          name: UI-PreFail-v2 temporal_order_v2 (seed 2910 test split, n=256)
+        metrics:
+          - type: average_precision
+            name: AP (failure vs stable)
+            value: 0.9926
+          - type: f1
+            name: Macro F1 (used classes)
+            value: 0.9911
+          - type: accuracy
+            name: Pair-rank accuracy
+            value: 1.0
+          - type: accuracy
+            name: Pair intervention success
+            value: 0.9844
+          - type: false_positive_rate
+            name: FPR at predicted class
+            value: 0.0156
+          - type: recall
+            name: Recall at predicted class
+            value: 1.0
+          - type: recall
+            name: Gate test recall
+            value: 1.0
+          - type: false_positive_rate
+            name: Gate test FPR (false intervention rate)
+            value: 0.0078
+          - type: precision
+            name: Gate test precision
+            value: 0.9922
 ---
 
+> **RESEARCH BENCHMARK, NOT FOR DEPLOYMENT.** This package is a synthetic counterfactual benchmark for studying ordered-temporal evidence in pre-failure detection. It is not a browser-agent guardrail and has no real-world validation.
+
 # PreFailureNet / UI-PreFail-v2
 
 ![PreFailureNet UI-PreFail-v2 release visual](assets/release_visual.svg)
 
 PreFailureNet / UI-PreFail-v2 is a synthetic ordered temporal counterfactual benchmark for pre-failure detection in browser-agent-like workflows. It includes shortcut probes, baseline ladders, a reference Transformer, and calibrated gate diagnostics. It is not a production browser-agent guardrail.
 
+## Headline Result vs Shallow Baseline
+
+Sources: `transformer_v2_35e_panel.json`, `baselines.json` (entry `all_frames`).
+
+| Method | AP (failure vs stable) | n_test |
+| --- | ---: | ---: |
+| Transformer v2, 35 epochs | `0.9926` | `256` |
+| All-frames logistic regression (flattened frames) | `0.8762` | `256` |
+
+A shallow logistic regression over flattened frames already reaches AP 0.8762 on this split, so the v2 headline reflects a benchmark that does not require deep temporal modeling. See v3 below for a split where this gap closes.
+
+## v3: xor_order_color defeats v2
+
+v3 introduces the `xor_order_color` family (local commit `3a323b1`, file `src/prefailurenet/data/synthetic.py`, function `_temporal_order_v3_pair`). The label is the XOR of an event-order bit and a per-pair color bit painted identically into every frame, so ordered-frame linear probes lose their handle while the pair structure stays intact.
+
+On v3 (test seed 2920, n_test=256, 128/128 pairs), the all-frames ordered LR drops from AP `0.8762` on v2 to AP `0.5486` on v3 (rounded to `0.549` in `docs/v3_split_evaluation.md`), which is below the project's 0.60 hard gate and within ~0.05 of chance. Source: `~/Projects/PreFailureNet/runs/benchmark_v3/baselines_seed2920/baselines.json` (entry `all_frames`, field `ap_failure_vs_stable`), cross-referenced in `docs/v3_split_evaluation.md`.
+
+The v2 Transformer recipe retrained on v3 flatlines:
+
+| Field | Value |
+| --- | ---: |
+| `precursor_detection_ap` | `0.5068` |
+| `intervention_auroc` | `0.5002` |
+
+Source: `~/Projects/PreFailureNet/runs/benchmark_v3/transformer_v3/metrics.json` (raw values `0.5068114175400058` and `0.500152587890625`).
+
+v3 inverts the v2 narrative. v2 results should be read as a defeated benchmark, not as evidence that the v2 Transformer architecture learns temporal order.
+
+v3 will be published as a separate HF repo or addendum after a v3-targeted model exists. Currently no model beats v3 baseline.
+
 ## Exact Positioning
 
 This package is a benchmark and research artifact. The correct claim is:
 
-> Ordered temporal evidence is required under synthetic counterfactual controls.
+> Ordered temporal evidence is required under synthetic counterfactual controls (v2). v3 shows this benchmark is itself defeated by an XOR-color counterfactual; the v2 headline is therefore a benchmark, not a capability claim.
 
 Do not read this release as evidence of production browser-agent safety, real-world website generalization, or a deployable intervention gate.
 
@@ -39,6 +109,7 @@ Do not read this release as evidence of production browser-agent safety, real-wo
 - Reference Transformer v2 held-out test metrics.
 - Risk-only gate calibration diagnostics.
 - Reproduction commands.
+- SHA256 checksums for all shipped artifacts (`CHECKSUMS.sha256`).
 
 Raw NPZ arrays and model checkpoint weights are not included in this compact HF package. They are reproducible from the commands below.
 
@@ -82,7 +153,7 @@ UI-PreFail-v2 / `temporal_order_v2` changes the label rule:
 - Safe and unsafe samples have the same unordered frame bag.
 - The label depends on the order of two event-bearing frames.
 
-This makes static, final-frame, non-vision, single mid-frame, and unordered-bag probes weak by construction. Ordered frame sequences and temporal deltas still contain useful signal.
+This makes static, final-frame, non-vision, single mid-frame, and unordered-bag probes weak by construction. Ordered frame sequences and temporal deltas still contain useful signal. (v3 closes the remaining handle. See above.)
 
 ## Held-Out v2 Pair Controls
 
@@ -197,6 +268,24 @@ PYTHONPATH=src python3 scripts/check_hf_package.py
 | `data_diagnostics.md` | Human-readable diagnostics. |
 | `transformer_v2_35e_panel.json` | Reference Transformer held-out test metrics. |
 | `transformer_v2_minrecall080_fpr020.json` | Gate calibration diagnostics. |
+| `CHECKSUMS.sha256` | SHA256 checksums for all shipped non-README artifacts. |
+
+## Checksums
+
+Verify with `sha256sum -c CHECKSUMS.sha256` from the repo root.
+
+| File | SHA256 |
+| --- | --- |
+| `artifact_index.json` | `7fb1523a96a0a7f23c4cf343fc6f5a27d7bd5fa4a844f92304b19afcca4d396e` |
+| `claims_ledger.md` | `9e20b26b5cec972be2882f4cb1333bbc18d15335c0199c78ba682113704bfb03` |
+| `assets/release_visual.svg` | `8ac6b51e1449f19eac5edf1d4e5ce5d74c0b0f3a259c3ef8629c0d141999bbc3` |
+| `baselines.json` | `29979a6d89071cf05e1b7ffa5ec27ae967430f24ef47b5e8d7601aef54625a93` |
+| `baselines.md` | `051c5426158bf0957cac905844222f38ef18ea3634ba78b47e7a6e9036adf1f8` |
+| `benchmark_v2_summary.json` | `428c87622bcb6191c4ce2081d0543da0b9a3e4c9425c10168d9e83ba4a2cbb22` |
+| `data_diagnostics.json` | `85c704597307aa2484bda87fd5215f35cc3217cc05d1dc8a6a978d8f1792a5a9` |
+| `data_diagnostics.md` | `361f8077142e9704fb6d45edf14c8d5c0b0c35e926a99487d78f6c2722b2ed67` |
+| `transformer_v2_35e_panel.json` | `016dd558e2e08fa70cbb9b6f78bdd332e69d2416be78b13dad38affa7aa1574c` |
+| `transformer_v2_minrecall080_fpr020.json` | `a85a02fe164e1493ccddc55d4275951a10422957e6ec62c15587be5a781d1a90` |
 
 ## Verified
 
@@ -207,6 +296,7 @@ PYTHONPATH=src python3 scripts/check_hf_package.py
 - Transformer v2 result exists.
 - Gate calibration result exists.
 - v1 flaw is documented.
+- v3 XOR counterfactual split exists locally and defeats both the ordered-LR baseline and the v2 Transformer recipe (see v3 section above).
 
 ## Unverified / Not Claimed
 
@@ -222,14 +312,19 @@ PYTHONPATH=src python3 scripts/check_hf_package.py
 - The benchmark is synthetic.
 - The UI events are rendered primitives, not production browser traces.
 - The reference result is on generated data only.
-- A flattened all-frames LR is strong because it sees ordered frames; v2 tests ordered temporal evidence, not uniquely deep temporal reasoning.
+- A flattened all-frames LR is strong because it sees ordered frames; v2 tests ordered temporal evidence, not uniquely deep temporal reasoning. The v3 XOR split removes even this handle and currently has no model that beats baseline.
 - Gate calibration is a benchmark diagnostic, not a safety policy.
 - The class schema contains unused classes in this split.
 
+## License
+
+Apache-2.0 for the package code, artifacts, and model card. Training and test data are synthetic and generated by the reproduction commands in this repo. No third-party dataset license applies.
+
 ## Next Research Steps
 
 - Add external browser-agent traces with consent and clear provenance.
-- Add stronger adversarial temporal controls where full-frame linear probes degrade.
+- Add stronger adversarial temporal controls where full-frame linear probes degrade (v3 is the first such step).
 - Add held-out UI event grammars and templates.
 - Evaluate under real browser screenshots and DOM/action logs.
 - Separate benchmark scoring from intervention-policy deployment work.
+- Train a v3-targeted model that actually beats v3 baseline before publishing a v3 HF repo.