Duplicate from cua-ai/cua-s1-forms

Co-authored-by: Dillon DuPont <ddupont@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,125 @@
+---
+license: mit
+tags:
+  - jev
+  - system-one
+  - computer-use
+  - form-filling
+  - option-attention
+base_model: []
+pipeline_tag: other
+---
+
+# cua-s1-forms
+
+A small, jev-like ("System One") one-pass option scorer for GUI form filling, trained to
+work as the decision layer behind [cua-driver](https://github.com/trycua/cua/tree/main/libs/cua-driver).
+
+Unlike an autoregressive LLM, this model does not generate text. Given a UI element and a
+list of typed options (one option per document entity, plus `check` / `click` / `skip`), it
+returns one probability per option in a single forward pass — the same input/output
+contract as TypeSafe's [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev).
+Every actionable element on a form is scored independently and in parallel in one batch;
+execution order (fills, then checkboxes, then the one submit click) is decided by
+downstream code, not the model.
+
+Full writeup, training code, synthetic data generator and live Cua Driver integration:
+https://github.com/trycua/cua/tree/main/libs/cua-s1.
+
+## Architecture
+
+- Byte-level embedding + 2-layer Transformer encoder (width 128, 4 heads) over the context
+  and, separately, over each option's text
+- jevlike's `AttentionHead`: each option becomes a query against the context tokens,
+  producing an attended context vector, then a shared dot product turns each
+  (option, attended-context) pair into one logit; softmax over the live option count
+- 706,048 trainable parameters, 2.8 MB checkpoint (`state_dict` + `config` + training
+  history + best validation metrics)
+
+## Input / output
+
+Context (one per element, byte-truncated to 224 bytes):
+```
+TASK fill the form from the document, then submit
+FORM Northwind Clinic - New Patient Registration
+ELEMENT Edit "Phone number" value=""
+```
+Options (one per document entity, plus the three fixed actions, byte-truncated to 96 bytes
+each): `fill Tel: (503) 555-0142`, `fill DOB: 03/14/1987`, ..., `check`, `click`, `skip`.
+
+Output: one probability per option. The executor picks the argmax, looks up the entity by
+index if the action is `fill`, and orders the resulting actions before sending them to
+cua-driver (`set_value` / `click`).
+
+## Training
+
+- 10,000 synthetic episodes (`cua_s1/synth.py`): random form (2–16 fields from a 55-concept
+  catalogue with form-label/document-label synonyms), random person, random document with
+  distractor entities and forced look-alike confuser pairs (e.g. `email` vs `street`,
+  `phone` vs `emergency contact phone`, `state` vs `university`), random window-title
+  suffixes and 20% title dropout
+- Splits are disjoint by exact form field signature — a test form's field set never appears
+  in training
+- AdamW, cosine schedule with warmup, 6 epochs, batch size 128, cross-entropy over the live
+  option count
+
+## Results (see the repo's `docs/RESULTS.md` for the full ladder)
+
+| split | top-1 | notes |
+| --- | ---: | --- |
+| synthetic test (form-disjoint, ~15k decisions) | 99.95% | hard confuser pairs forced in |
+| real demo eval (3 real forms + 3 real PDFs, 196 decisions, nothing synthetic) | 100% | |
+| shuffled-context control | 37% | confirms the model reads the element, not option statistics |
+
+Head-to-head against the real hosted Jev API (`jev-latest`, zero fine-tuning, same task):
+99.7% for this model vs 83.6% for hosted Jev overall; 96% for hosted Jev on decisions that
+require real judgment (fill vs check vs click) and 74% on recognizing an already-filled
+field as a no-op — a convention this model was trained on and hosted Jev was not. Full
+numbers in the repo.
+
+## Files
+
+- `cua-s1-forms.safetensors` + `cua-s1-forms.json` — the checkpoint in the format
+  `cua_s1.checkpoint` expects: tensors only in safetensors, everything else (architecture
+  config, a SHA-256 signature over the tensors, free-form metadata) in a plain JSON
+  sidecar. This is the format to use; `cua_s1`'s own loader rejects pickled `.pt`/`.pth`
+  files by design (arbitrary pickle is a code-execution risk for a public checkpoint).
+- `cua-s1-forms.pt` — the original PyTorch pickle checkpoint, kept only for anyone still
+  loading it directly with `torch.load(..., weights_only=False)` outside `cua_s1`. New code
+  should use the safetensors pair above.
+
+Both encode the exact same weights; converted with a script that reimplements
+`cua_s1.checkpoint.save_checkpoint_files`'s exact document/signature format, and verified to
+produce bit-for-bit identical model output against the original `.pt`.
+
+## Usage
+
+```python
+from pathlib import Path
+from huggingface_hub import hf_hub_download
+from cua_s1.model import load_checkpoint, select_device
+
+repo = "cua-ai/cua-s1-forms"
+weights = Path(hf_hub_download(repo, "cua-s1-forms.safetensors"))
+hf_hub_download(repo, "cua-s1-forms.json", local_dir=weights.parent)  # sits next to the weights
+
+# validates format, version and SHA-256 tensor signature before returning
+model, collator, config = load_checkpoint(weights, select_device("auto"))
+```
+
+See [`cua_s1/planner.py`](https://github.com/trycua/cua/tree/main/libs/cua-s1/python/src/cua_s1/planner.py)
+for the full snapshot → score → order → execute loop against a live Cua Driver session.
+
+## Limitations
+
+- Only ever chooses among entities a PDF/document extractor already found as `Label: value`
+  pairs; it cannot invent a value.
+- Trained entirely on synthetic forms plus a small (196-decision) real eval; not validated
+  on arbitrary real-world forms outside the demo set.
+- Byte-level encoder, English-centric label vocabulary.
+- Not calibrated with TypeSafe's RLCD method — this is an independent research checkpoint,
+  not a reproduction of Jev.
+
+## License
+
+MIT.

cua-s1-forms.json

@@ -0,0 +1,44 @@
+{
+  "config": {
+    "context_tokens": 224,
+    "encoder": "tinyx",
+    "heads": 4,
+    "hf_model": "Qwen/Qwen2.5-0.5B",
+    "layers": 2,
+    "option_tokens": 96,
+    "rank": 128,
+    "width": 128
+  },
+  "format": "cua-s1",
+  "format_version": 1,
+  "metadata": {
+    "best_validation": {
+      "ece": 0.00014795374386267213,
+      "examples": 22054,
+      "nll": 0.0019850827802381913,
+      "per_action": {
+        "check": {
+          "acc": 1.0,
+          "n": 835
+        },
+        "click": {
+          "acc": 1.0,
+          "n": 939
+        },
+        "fill": {
+          "acc": 0.9998858968507531,
+          "n": 8764
+        },
+        "skip": {
+          "acc": 0.9989579715178881,
+          "n": 11516
+        }
+      },
+      "rows_per_second": 2589.071654630599,
+      "top1": 0.9994105100631714
+    },
+    "converted_by": "scripts/convert_to_safetensors.py",
+    "source_checkpoint": "jevform-best.pt"
+  },
+  "state_signature": "0c661c669bc22189c81637553dcd105fe6178c8a8de85f6f6307c3c635403bfa"
+}

cua-s1-forms.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5077f0c9baf6b5fc10f21512e1aa15207a395598416a6ffdd95f0d3dd5ab8df
+size 2840436

cua-s1-forms.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05954c1caf51c2fb6c13ea4acbfc88a2e7653dea192252bb51dc89e76a356ddc
+size 2828784