README.md

---
language: en
license: apache-2.0
library_name: pytorch
pipeline_tag: time-series-classification
last_updated: 2026-05-25
tags:
  - benchmark
  - temporal-modeling
  - sequence-learning
  - synthetic-data
  - counterfactuals
  - transformers
  - evaluation
  - browser-agents
  - ai-agents
model-index:
  - name: PreFailureNet UI-PreFail-v2 (Transformer, 35 epochs)
    results:
      - task:
          type: time-series-classification
          name: Pre-failure detection (synthetic ordered counterfactual, v2)
        dataset:
          type: synthetic
          name: UI-PreFail-v2 temporal_order_v2 (seed 2910 test split, n=256)
        metrics:
          - type: average_precision
            name: AP (failure vs stable)
            value: 0.9926
          - type: f1
            name: Macro F1 (used classes)
            value: 0.9911
          - type: accuracy
            name: Pair-rank accuracy
            value: 1.0
          - type: accuracy
            name: Pair intervention success
            value: 0.9844
          - type: false_positive_rate
            name: FPR at predicted class
            value: 0.0156
          - type: recall
            name: Recall at predicted class
            value: 1.0
          - type: recall
            name: Gate test recall
            value: 1.0
          - type: false_positive_rate
            name: Gate test FPR (false intervention rate)
            value: 0.0078
          - type: precision
            name: Gate test precision
            value: 0.9922
---

> **RESEARCH BENCHMARK, NOT FOR DEPLOYMENT.** This package is a synthetic counterfactual benchmark for studying ordered-temporal evidence in pre-failure detection. It is not a browser-agent guardrail and has no real-world validation.

# PreFailureNet / UI-PreFail-v2

![PreFailureNet UI-PreFail-v2 release visual](assets/release_visual.svg)

PreFailureNet / UI-PreFail-v2 is a synthetic ordered temporal counterfactual benchmark for pre-failure detection in browser-agent-like workflows. It includes shortcut probes, baseline ladders, a reference Transformer, and calibrated gate diagnostics. It is not a production browser-agent guardrail.

## Headline Result vs Shallow Baseline

Sources: `transformer_v2_35e_panel.json`, `baselines.json` (entry `all_frames`).

| Method | AP (failure vs stable) | n_test |
| --- | ---: | ---: |
| Transformer v2, 35 epochs | `0.9926` | `256` |
| All-frames logistic regression (flattened frames) | `0.8762` | `256` |

A shallow logistic regression over flattened frames already reaches AP 0.8762 on this split, so the v2 headline reflects a benchmark that does not require deep temporal modeling. See v3 below for a split where this gap closes.

## v3: xor_order_color defeats v2

v3 introduces the `xor_order_color` family (local commit `3a323b1`, file `src/prefailurenet/data/synthetic.py`, function `_temporal_order_v3_pair`). The label is the XOR of an event-order bit and a per-pair color bit painted identically into every frame, so ordered-frame linear probes lose their handle while the pair structure stays intact.

On v3 (test seed 2920, n_test=256, 128/128 pairs), the all-frames ordered LR drops from AP `0.8762` on v2 to AP `0.5486` on v3 (rounded to `0.549` in `docs/v3_split_evaluation.md`), which is below the project's 0.60 hard gate and within ~0.05 of chance. Source: `~/Projects/PreFailureNet/runs/benchmark_v3/baselines_seed2920/baselines.json` (entry `all_frames`, field `ap_failure_vs_stable`), cross-referenced in `docs/v3_split_evaluation.md`.

The v2 Transformer recipe retrained on v3 flatlines:

| Field | Value |
| --- | ---: |
| `precursor_detection_ap` | `0.5068` |
| `intervention_auroc` | `0.5002` |

Source: `~/Projects/PreFailureNet/runs/benchmark_v3/transformer_v3/metrics.json` (raw values `0.5068114175400058` and `0.500152587890625`).

v3 inverts the v2 narrative. v2 results should be read as a defeated benchmark, not as evidence that the v2 Transformer architecture learns temporal order.

v3 will be published as a separate HF repo or addendum after a v3-targeted model exists. Currently no model beats v3 baseline.

## Exact Positioning

This package is a benchmark and research artifact. The correct claim is:

> Ordered temporal evidence is required under synthetic counterfactual controls (v2). v3 shows this benchmark is itself defeated by an XOR-color counterfactual; the v2 headline is therefore a benchmark, not a capability claim.

Do not read this release as evidence of production browser-agent safety, real-world website generalization, or a deployable intervention gate.

## What This Package Contains

- A public claims ledger.
- A compact artifact index.
- Benchmark v2 summary metrics.
- Shortcut probes and baseline ladder.
- Pair-matching and temporal-signal diagnostics.
- Reference Transformer v2 held-out test metrics.
- Risk-only gate calibration diagnostics.
- Reproduction commands.
- SHA256 checksums for all shipped artifacts (`CHECKSUMS.sha256`).

Raw NPZ arrays and model checkpoint weights are not included in this compact HF package. They are reproducible from the commands below.

## Tested Environment And Disk Requirements

This milestone was validated on Ubuntu 22.04 / Linux 6.8, x86_64, Python 3.10.12. Use Python
3.10 for reproduction; the project metadata allows Python `>=3.10`, but the release validation
environment is Python 3.10.

CUDA is optional for the published benchmark checks. The generator, diagnostics, shortcut
baselines, HF package checker, and tests run on CPU. Transformer training can use CPU or CUDA
depending on the local Torch install; no CUDA, Jetson, TensorRT, browser-extension, or live
agent deployment result is claimed.

Disk planning:

- Compact HF package: under 1 MB.
- Reproduced v2 NPZ splits: about 40 MB for train/calibration/test.
- Existing `runs/benchmark_v2` artifact directory: about 50 MB.
- Python environment with Torch, OpenCV, scikit-learn, and Pillow: budget 2-5 GB.

## Why v1 Was Flawed

The original temporal counterfactual split was useful but flawed:

- It matched state, context, action tokens, and final frame.
- The label signal lived in 1-2 mid-sequence visual patches.
- Patch position and active steps were family-structured.
- A logistic regression on mid-frame pixels reached AP `0.9249` and pair-rank `0.9875`.
- The strongest v1 Transformer reached AP `0.8746`, so it still lost to the mid-frame LR probe.

Conclusion: v1 was shortcut-controlled, but it rewarded localized mid-frame patch detection more than temporal reasoning. v1 remains documented as a negative result and shortcut-control scaffold.

## What v2 Fixes

UI-PreFail-v2 / `temporal_order_v2` changes the label rule:

- Safe and unsafe samples have the same UI template/style/distractors.
- Safe and unsafe samples have identical state, context, and action tokens.
- Safe and unsafe samples have the same final frame.
- Safe and unsafe samples have the same unordered frame bag.
- The label depends on the order of two event-bearing frames.

This makes static, final-frame, non-vision, single mid-frame, and unordered-bag probes weak by construction. Ordered frame sequences and temporal deltas still contain useful signal. (v3 closes the remaining handle. See above.)

## Held-Out v2 Pair Controls

Source: `data_diagnostics.json`.

| Control | Held-out test result |
| --- | ---: |
| pair count | `128` |
| state-identical pairs | `128 / 128` |
| context-identical pairs | `128 / 128` |
| action-identical pairs | `128 / 128` |
| final-frame-identical pairs | `128 / 128` |
| unordered-frame-bag-identical pairs | `128 / 128` |

## Shortcut Probe Table

Source: `baselines.json`.

| Probe | AP | Pair-rank | Interpretation |
| --- | ---: | ---: | --- |
| last-state LR | `0.5000` | `0.0000` | non-vision shortcut controlled |
| all-actions LR | `0.5000` | `0.0000` | action shortcut controlled |
| last-frame LR | `0.5005` | `0.0859` | final-frame shortcut controlled |
| single mid-frame LR | `0.5524` | `0.2734` | weak, not dominant |
| all-frame bag LR | `0.5048` | `0.5781` | unordered frame bag controlled |
| mid-frames LR | `0.7746` | `0.7891` | partially order-aware |
| all-frames LR | `0.8762` | `0.8828` | order-aware linear pixels work |
| frame-deltas LR | `0.8525` | `0.8516` | temporal deltas are useful |

## Reference Results

Source: `transformer_v2_35e_panel.json` and `transformer_v2_minrecall080_fpr020.json`.

| Method | AP | Macro F1 used | Pair-rank | Pair intervention success | FPR @ predicted class | Recall @ predicted class |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Transformer v2, 35 epochs | `0.9926` | `0.9911` | `1.0000` | `0.9844` | `0.0156` | `1.0000` |

Gate calibration selected on the calibration split with `min_recall=0.80`, `max_false_intervention_rate=0.20`:

| Split | Recall | FPR | Precision |
| --- | ---: | ---: | ---: |
| calibration | `0.9922` | `0.0000` | `1.0000` |
| held-out test | `1.0000` | `0.0078` | `0.9922` |

This is a synthetic benchmark calibration result, not a deployment threshold.

## Reproduction Commands

Run from the repository root:

```bash
PYTHONPATH=src python3 scripts/generate_synthetic.py \
  --domain software --split temporal_order_v2 --split-role train \
  --num-samples 512 --seed 910 --output-dir runs/benchmark_v2/data

PYTHONPATH=src python3 scripts/generate_synthetic.py \
  --domain software --split temporal_order_v2 --split-role calibration \
  --num-samples 256 --seed 1910 --output-dir runs/benchmark_v2/data

PYTHONPATH=src python3 scripts/generate_synthetic.py \
  --domain software --split temporal_order_v2 --split-role test \
  --num-samples 256 --seed 2910 --output-dir runs/benchmark_v2/data

PYTHONPATH=src python3 scripts/temporal_data_diagnostics.py \
  --train runs/benchmark_v2/data/software_synthetic_n512_seed910.npz \
  --test runs/benchmark_v2/data/software_synthetic_n256_seed2910.npz \
  --output-dir runs/benchmark_v2/diagnostics

PYTHONPATH=src python3 scripts/temporal_baselines.py \
  --train runs/benchmark_v2/data/software_synthetic_n512_seed910.npz \
  --test runs/benchmark_v2/data/software_synthetic_n256_seed2910.npz \
  --output-dir runs/benchmark_v2/baselines

PYTHONPATH=src python3 scripts/train.py --config configs/benchmark_v2_transformer.yaml

PYTHONPATH=src python3 scripts/temporal_model_eval.py \
  --config configs/benchmark_v2_transformer.yaml \
  --checkpoint runs/benchmark_v2/transformer_v2/best.pt \
  --name transformer_v2_35e \
  --split test \
  --output-dir runs/benchmark_v2/model_panels

PYTHONPATH=src python3 scripts/calibrate_gate.py \
  --config configs/benchmark_v2_transformer.yaml \
  --checkpoint runs/benchmark_v2/transformer_v2/best.pt \
  --output runs/benchmark_v2/gate_calibration/transformer_v2_minrecall080_fpr020.json \
  --min-recall 0.80 \
  --max-false-intervention-rate 0.20

PYTHONPATH=src python3 scripts/build_benchmark_v2_summary.py \
  --root runs/benchmark_v2 \
  --output runs/benchmark_v2/benchmark_v2_summary.json
```

Package validation:

```bash
PYTHONPATH=src python3 scripts/check_hf_package.py
```

## Artifact Map

| File | Purpose |
| --- | --- |
| `claims_ledger.md` | Public claim boundary. |
| `assets/release_visual.svg` | Compact v1 flaw to v2 ordered-event benchmark visual. |
| `artifact_index.json` | Machine-readable package contents. |
| `benchmark_v2_summary.json` | Consolidated v2 metrics and decision. |
| `baselines.json` | Shortcut probes and baseline ladder. |
| `baselines.md` | Human-readable baseline table. |
| `data_diagnostics.json` | Pair controls and temporal signal diagnostics. |
| `data_diagnostics.md` | Human-readable diagnostics. |
| `transformer_v2_35e_panel.json` | Reference Transformer held-out test metrics. |
| `transformer_v2_minrecall080_fpr020.json` | Gate calibration diagnostics. |
| `CHECKSUMS.sha256` | SHA256 checksums for all shipped non-README artifacts. |

## Checksums

Verify with `sha256sum -c CHECKSUMS.sha256` from the repo root.

| File | SHA256 |
| --- | --- |
| `artifact_index.json` | `7fb1523a96a0a7f23c4cf343fc6f5a27d7bd5fa4a844f92304b19afcca4d396e` |
| `claims_ledger.md` | `9e20b26b5cec972be2882f4cb1333bbc18d15335c0199c78ba682113704bfb03` |
| `assets/release_visual.svg` | `8ac6b51e1449f19eac5edf1d4e5ce5d74c0b0f3a259c3ef8629c0d141999bbc3` |
| `baselines.json` | `29979a6d89071cf05e1b7ffa5ec27ae967430f24ef47b5e8d7601aef54625a93` |
| `baselines.md` | `051c5426158bf0957cac905844222f38ef18ea3634ba78b47e7a6e9036adf1f8` |
| `benchmark_v2_summary.json` | `428c87622bcb6191c4ce2081d0543da0b9a3e4c9425c10168d9e83ba4a2cbb22` |
| `data_diagnostics.json` | `85c704597307aa2484bda87fd5215f35cc3217cc05d1dc8a6a978d8f1792a5a9` |
| `data_diagnostics.md` | `361f8077142e9704fb6d45edf14c8d5c0b0c35e926a99487d78f6c2722b2ed67` |
| `transformer_v2_35e_panel.json` | `016dd558e2e08fa70cbb9b6f78bdd332e69d2416be78b13dad38affa7aa1574c` |
| `transformer_v2_minrecall080_fpr020.json` | `a85a02fe164e1493ccddc55d4275951a10422957e6ec62c15587be5a781d1a90` |

## Verified

- v2 generator exists.
- v2 matched ordered-event pairs exist.
- v2 shortcut probes exist.
- v2 baseline ladder exists.
- Transformer v2 result exists.
- Gate calibration result exists.
- v1 flaw is documented.
- v3 XOR counterfactual split exists locally and defeats both the ordered-LR baseline and the v2 Transformer recipe (see v3 section above).

## Unverified / Not Claimed

- No production browser traffic.
- No live browser-agent deployment.
- No deployable browser-agent gate.
- No real-world website generalization.
- No claim that deep temporal reasoning is required.
- No claim that this benchmark proves browser-agent safety.

## Limitations

- The benchmark is synthetic.
- The UI events are rendered primitives, not production browser traces.
- The reference result is on generated data only.
- A flattened all-frames LR is strong because it sees ordered frames; v2 tests ordered temporal evidence, not uniquely deep temporal reasoning. The v3 XOR split removes even this handle and currently has no model that beats baseline.
- Gate calibration is a benchmark diagnostic, not a safety policy.
- The class schema contains unused classes in this split.

## License

Apache-2.0 for the package code, artifacts, and model card. Training and test data are synthetic and generated by the reproduction commands in this repo. No third-party dataset license applies.

## Next Research Steps

- Add external browser-agent traces with consent and clear provenance.
- Add stronger adversarial temporal controls where full-frame linear probes degrade (v3 is the first such step).
- Add held-out UI event grammars and templates.
- Evaluate under real browser screenshots and DOM/action logs.
- Separate benchmark scoring from intervention-policy deployment work.
- Train a v3-targeted model that actually beats v3 baseline before publishing a v3 HF repo.