Add experimental INT8 weights and full matched benchmark

INT8 weights with FP16 compute: 813,422 bytes versus 1,511,163 bytes. All 24,370 choices match PyTorch/FP16, with 24,359 correct (99.9549%); matched median 0.990 ms for both. Numerical parity fails: 64 INT8 probability errors above 0.005, maximum 0.067738, no normalization failures. Default and ANE-gather artifacts unchanged. Publish exact settings, complete per-row trace, demo checks and compute-plan/fallback reports.

Files changed (11) hide show

README.md +32 -0
checksums.json +10 -1
int8-weights/conversion.json +85 -0
int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel +3 -0
int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin +3 -0
int8-weights/cua_s1_forms_int8_options32.mlpackage/Manifest.json +18 -0
reports/int8-demo-verification.json +1758 -0
reports/int8-fallback.json +112 -0
reports/int8-profile.json +1105 -0
reports/int8-synthetic-test-decisions.jsonl.gz +3 -0
reports/int8-synthetic-test.json +1473 -0

README.md CHANGED Viewed

@@ -276,6 +276,38 @@ Reports: [parity](reports/ane-gather-verification.json),
 `uv run --frozen python convert-coreml.py --optimization ane-gather --output-dir build/ane-gather`
 in the [Mobius conversion directory](https://github.com/FluidInference/mobius/tree/codex/cua-s1-forms/models/computer-use/cua-s1-forms/coreml#optional-higher-ane-variant).
 ## Application responsibilities
 The application must extract document entities, describe UI elements, build

 `uv run --frozen python convert-coreml.py --optimization ane-gather --output-dir build/ane-gather`
 in the [Mobius conversion directory](https://github.com/FluidInference/mobius/tree/codex/cua-s1-forms/models/computer-use/cua-s1-forms/coreml#optional-higher-ane-variant).
+## INT8 weight trial
+A matched run over all **24,370 synthetic decisions** on the same M5 Pro:
+| Export | Package size | Accuracy | Median | p95 |
+| --- | ---: | ---: | ---: | ---: |
+| Original FP16 | 1.51 MB | 99.9549% | 0.990 ms | 1.102 ms |
+| INT8 weights, FP16 compute | 0.81 MB | 99.9549% | 0.990 ms | 1.104 ms |
+**46.2% smaller**, with every selected option unchanged and effectively identical
+latency. Numerical parity still fails: 64 rows exceed the 0.005 probability-error
+limit (maximum 0.067738), versus 11 for FP16. No INT8 probability-sum violations
+were observed. The original remains the default.
+The experimental `int8-weights/cua_s1_forms_int8_options32.mlpackage` uses INT8
+weights with FP16 computation. Compile it locally or load it with the existing
+Swift `load(from:)` API. Inputs and outputs match the original model. This
+variant was measured on M5 Pro; iPhone behavior has not been measured.
+[Full report](reports/int8-synthetic-test.json) ·
+[Complete per-row trace](reports/int8-synthetic-test-decisions.jsonl.gz) ·
+[Quantization manifest](int8-weights/conversion.json) ·
+[Reproduction](https://github.com/FluidInference/mobius/tree/codex/cua-s1-forms/models/computer-use/cua-s1-forms/coreml#int8-weight-trial)
+The matched benchmark uses CPU+ANE, batch 1, three warmups/model, and one complete
+pass with alternating Core ML order; encoding/loading/UI are excluded. All models
+get the same 24,359 decisions correct. Public compute plans assign 149 ANE and
+32 CPU operations, with 19 unassigned constant-dequantization operations. This
+compression does not establish an INT8 activation path or a speed/energy gain.
+No calibration, training, or test-based tuning was performed. Numerical parity
+remains failed; the default FP16 artifacts are unchanged.
 ## Application responsibilities
 The application must extract document entities, describe UI elements, build

checksums.json CHANGED Viewed

@@ -1,7 +1,7 @@
 {
   "LICENSE": "c0779290c1d4783169aa3dbfb55feb505e563ef8a004bbf55298ceffcfbda8d9",
   "NOTICES.md": "027c72741eaa695e60d7b6cebd3666372c81d443a96b673b72a897cf432ddfd0",
-  "README.md": "5739754098ca5a480397693115f079e3e500c2a06abb822f152cf7cbdc9943ac",
   "UPSTREAM-THIRD-PARTY-NOTICES.md": "4091e69b45c8cc97e30a066fbbd56148dbef66ab716432c048d2333c9c464213",
   "ane-gather/conversion.json": "4d569a4020f1e9711da3f7fff0d11acef1812c4652414cf64728c77bc2bae4c4",
   "ane-gather/cua_s1_forms_fp16_options32.mlmodelc/analytics/coremldata.bin": "526dccb87bdc036df1e8add0b50dd6540db534808fc2f8caa97cd897b4fc1fa3",
@@ -23,6 +23,10 @@
   "demo/browser-demo.gif": "6cf597927e7bcb5617719453eecb86c7c06ab9a91fcaf46de272b42abe12d4fc",
   "demo/browser-demo.mp4": "5fbb3925c4824d15e6c24c504db91a96d318f4e751a20a070fdedf64e587d1a0",
   "demo/browser-preview.png": "e624ec2a797d0b3b050e66055d6a9fb09a1f6a9ace67b8eb071ce5f2f9e282b1",
   "preprocessing.py": "18c74ede43f95f91cd638c7e9631923a353e3a24d3b522ddb8b18b480f9c2846",
   "reports/ane-comparison.json": "14b29f87704c6e8bd5f9bd866ea825be3972fcea6cf3c4760e38dc8774f937b6",
   "reports/ane-fallback.json": "2f52b0e11f7cfbea6a6be6942de8816d8fed0630ac384a28551d405648fcb4e0",
@@ -31,6 +35,11 @@
   "reports/ane-gather-verification.json": "fa51fe0537dcd9cc8d1d75fc0be4747d887aa2e3af199ec08edd8f9400ce3318",
   "reports/ane-profile.json": "8f654717397a294acc793861676a1480c160bb3f7ab5be6ff67ebb6a4473b832",
   "reports/browser-validation.json": "299641c7b9f6cc2464cfcd984b4ef4445385e4710bc15abae7cee8ba262331b6",
   "reports/swift-ane-validation.json": "ed323e6d6619fb7698d3546599d6450b745bc8cc6100c02e13f7ad4d59859a3a",
   "reports/swift-variant-comparison.json": "593053d7011a1c6d6881b130c3408099d0fcc09c714c4064d4f853f0dc63b651",
   "reports/synthetic-test-decisions.jsonl.gz": "b6f6edc9de34fb37b10847a1f9568b530ce2b8e1bd241f51bf2e637f489d6554",

 {
   "LICENSE": "c0779290c1d4783169aa3dbfb55feb505e563ef8a004bbf55298ceffcfbda8d9",
   "NOTICES.md": "027c72741eaa695e60d7b6cebd3666372c81d443a96b673b72a897cf432ddfd0",
+  "README.md": "ba32d1ab1fc14a855bd30bc07a8e8474f6b095f17c8126f6a9fc6a810b6ab31f",
   "UPSTREAM-THIRD-PARTY-NOTICES.md": "4091e69b45c8cc97e30a066fbbd56148dbef66ab716432c048d2333c9c464213",
   "ane-gather/conversion.json": "4d569a4020f1e9711da3f7fff0d11acef1812c4652414cf64728c77bc2bae4c4",
   "ane-gather/cua_s1_forms_fp16_options32.mlmodelc/analytics/coremldata.bin": "526dccb87bdc036df1e8add0b50dd6540db534808fc2f8caa97cd897b4fc1fa3",
   "demo/browser-demo.gif": "6cf597927e7bcb5617719453eecb86c7c06ab9a91fcaf46de272b42abe12d4fc",
   "demo/browser-demo.mp4": "5fbb3925c4824d15e6c24c504db91a96d318f4e751a20a070fdedf64e587d1a0",
   "demo/browser-preview.png": "e624ec2a797d0b3b050e66055d6a9fb09a1f6a9ace67b8eb071ce5f2f9e282b1",
+  "int8-weights/conversion.json": "b8a51ceeb65fe8020b2a186fae2ebf586443956111acfbac1a3f208a062c1c80",
+  "int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
+  "int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
+  "int8-weights/cua_s1_forms_int8_options32.mlpackage/Manifest.json": "2206ec01787a85c12f1376fa5a832c5ac912f06172c6ff06b26ae8415470d246",
   "preprocessing.py": "18c74ede43f95f91cd638c7e9631923a353e3a24d3b522ddb8b18b480f9c2846",
   "reports/ane-comparison.json": "14b29f87704c6e8bd5f9bd866ea825be3972fcea6cf3c4760e38dc8774f937b6",
   "reports/ane-fallback.json": "2f52b0e11f7cfbea6a6be6942de8816d8fed0630ac384a28551d405648fcb4e0",
   "reports/ane-gather-verification.json": "fa51fe0537dcd9cc8d1d75fc0be4747d887aa2e3af199ec08edd8f9400ce3318",
   "reports/ane-profile.json": "8f654717397a294acc793861676a1480c160bb3f7ab5be6ff67ebb6a4473b832",
   "reports/browser-validation.json": "299641c7b9f6cc2464cfcd984b4ef4445385e4710bc15abae7cee8ba262331b6",
+  "reports/int8-demo-verification.json": "5602c8c254c0ae4c696e771291b25d1196ccc0d63f48eaaccdde0245803a5368",
+  "reports/int8-fallback.json": "9bb86a529a8e7e353efa12449a5ea930817dc7b1afdacec6ae24f3c318bc5835",
+  "reports/int8-profile.json": "0b962528dc7e5bb1d53abeb4b549f93b93f549a1e1935c3d77ef2606e83bf6a9",
+  "reports/int8-synthetic-test-decisions.jsonl.gz": "69720b9e11430ecff7ea0718f14bf16792de6f39f114faba250cfe855e4e4156",
+  "reports/int8-synthetic-test.json": "952ebe281d0aef0c05667d3c0dc5f9f175be709639a542028940059393078b5d",
   "reports/swift-ane-validation.json": "ed323e6d6619fb7698d3546599d6450b745bc8cc6100c02e13f7ad4d59859a3a",
   "reports/swift-variant-comparison.json": "593053d7011a1c6d6881b130c3408099d0fcc09c714c4064d4f853f0dc63b651",
   "reports/synthetic-test-decisions.jsonl.gz": "b6f6edc9de34fb37b10847a1f9568b530ce2b8e1bd241f51bf2e637f489d6554",

int8-weights/conversion.json ADDED Viewed

	@@ -0,0 +1,85 @@

+{
+  "model": "cua_s1_forms_int8_options32.mlpackage",
+  "precision": "int8_weights_float16_compute",
+  "minimum_target": "iOS17/macOS14",
+  "limits": {
+    "context_bytes": 224,
+    "option_bytes": 96,
+    "max_options": 32
+  },
+  "model_config": {
+    "context_tokens": 224,
+    "encoder": "tinyx",
+    "heads": 4,
+    "hf_model": "Qwen/Qwen2.5-0.5B",
+    "layers": 2,
+    "option_tokens": 96,
+    "rank": 128,
+    "width": 128
+  },
+  "parameters": 706048,
+  "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "trace_row": 0,
+  "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "export_seconds": 0.5374167499830946,
+  "python": "3.11.11",
+  "torch": "2.7.0",
+  "coremltools": "9.0",
+  "package_files": {
+    "Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
+    "Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
+    "Manifest.json": "2206ec01787a85c12f1376fa5a832c5ac912f06172c6ff06b26ae8415470d246"
+  },
+  "quantization": {
+    "name": "int8-weights",
+    "settings": {
+      "mode": "linear_symmetric",
+      "dtype": "int8",
+      "granularity": "per_channel",
+      "weight_threshold": 2048
+    },
+    "activations": "float16",
+    "calibration": "None; weight-only, data-free quantization",
+    "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+    "source_package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+      "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+      "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+    },
+    "source_package_bytes": 1511163,
+    "package_bytes": 813422,
+    "compression_seconds": 0.3180606249952689,
+    "operation_counts": {
+      "add": 15,
+      "cast": 13,
+      "concat": 2,
+      "const": 226,
+      "constexpr_affine_dequantize": 19,
+      "equal": 2,
+      "expand_dims": 5,
+      "fill_like": 3,
+      "gather": 2,
+      "greater_equal": 4,
+      "layer_norm": 8,
+      "linear": 15,
+      "logical_not": 2,
+      "matmul": 8,
+      "maximum": 1,
+      "mul": 7,
+      "not_equal": 3,
+      "real_div": 1,
+      "reduce_sum": 3,
+      "relu": 3,
+      "reshape": 31,
+      "select": 9,
+      "slice_by_index": 13,
+      "softmax": 5,
+      "squeeze": 3,
+      "tile": 2,
+      "transpose": 21
+    },
+    "script_sha256": "22071babfcb02ef77225203a479d9bb14b9fa35462ba38bcc4bf07d29c125a97"
+  }
+}

int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a
+size 69573

int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f
+size 743232

int8-weights/cua_s1_forms_int8_options32.mlpackage/Manifest.json ADDED Viewed

	@@ -0,0 +1,18 @@

+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "6D5496F1-54BB-4AAA-8375-1DFA1FFE6768": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "B0C97583-069E-42B8-897B-30A05F77B14A": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "6D5496F1-54BB-4AAA-8375-1DFA1FFE6768"
+}

reports/int8-demo-verification.json ADDED Viewed

	@@ -0,0 +1,1758 @@

+{
+  "purpose": "Local conversion parity on the upstream demo; not a generalization or live GUI benchmark",
+  "created_utc": "2026-09-19T22:03:05.949263+00:00",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "dataset_file": "demo.jsonl",
+  "dataset_sha256": "4f43b442e79ba2e2ce731e27e9b8e340c2b5dfcaffc92d8ff564c34f115ff1ca",
+  "conversion": {
+    "model": "cua_s1_forms_int8_options32.mlpackage",
+    "precision": "int8_weights_float16_compute",
+    "minimum_target": "iOS17/macOS14",
+    "limits": {
+      "context_bytes": 224,
+      "option_bytes": 96,
+      "max_options": 32
+    },
+    "model_config": {
+      "context_tokens": 224,
+      "encoder": "tinyx",
+      "heads": 4,
+      "hf_model": "Qwen/Qwen2.5-0.5B",
+      "layers": 2,
+      "option_tokens": 96,
+      "rank": 128,
+      "width": 128
+    },
+    "parameters": 706048,
+    "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+    "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+    "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+    "trace_row": 0,
+    "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+    "export_seconds": 0.5374167499830946,
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
+      "Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
+      "Manifest.json": "2206ec01787a85c12f1376fa5a832c5ac912f06172c6ff06b26ae8415470d246"
+    },
+    "quantization": {
+      "name": "int8-weights",
+      "settings": {
+        "mode": "linear_symmetric",
+        "dtype": "int8",
+        "granularity": "per_channel",
+        "weight_threshold": 2048
+      },
+      "activations": "float16",
+      "calibration": "None; weight-only, data-free quantization",
+      "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+      "source_package_files": {
+        "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+        "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+        "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+      },
+      "source_package_bytes": 1511163,
+      "package_bytes": 813422,
+      "compression_seconds": 0.3180606249952689,
+      "operation_counts": {
+        "add": 15,
+        "cast": 13,
+        "concat": 2,
+        "const": 226,
+        "constexpr_affine_dequantize": 19,
+        "equal": 2,
+        "expand_dims": 5,
+        "fill_like": 3,
+        "gather": 2,
+        "greater_equal": 4,
+        "layer_norm": 8,
+        "linear": 15,
+        "logical_not": 2,
+        "matmul": 8,
+        "maximum": 1,
+        "mul": 7,
+        "not_equal": 3,
+        "real_div": 1,
+        "reduce_sum": 3,
+        "relu": 3,
+        "reshape": 31,
+        "select": 9,
+        "slice_by_index": 13,
+        "softmax": 5,
+        "squeeze": 3,
+        "tile": 2,
+        "transpose": 21
+      },
+      "script_sha256": "22071babfcb02ef77225203a479d9bb14b9fa35462ba38bcc4bf07d29c125a97"
+    }
+  },
+  "environment": {
+    "machine": "Apple M5 Pro",
+    "os": "27.0",
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "torch_cpu_threads": 2
+  },
+  "rows": 196,
+  "upstream": {
+    "correct": 196,
+    "accuracy": 1.0,
+    "inference_latency_ms": {
+      "count": 196,
+      "median_ms": 2.3908544972073287,
+      "p95_ms": 3.0149687590892427,
+      "min_ms": 1.7770830017980188,
+      "max_ms": 9.844208980211988
+    }
+  },
+  "export_adapter_fp32": {
+    "argmax_agreement": 196,
+    "max_abs_probability_error": 1.1324882507324219e-06
+  },
+  "host_preprocessing_latency_ms": {
+    "count": 196,
+    "median_ms": 0.03708299482241273,
+    "p95_ms": 0.04522973904386163,
+    "min_ms": 0.024834007490426302,
+    "max_ms": 0.08750002598389983
+  },
+  "thresholds": {
+    "argmax_agreement": 1.0,
+    "max_abs_probability_error": 0.005,
+    "export_adapter_max_abs_probability_error": 0.0001,
+    "allow_accuracy_loss": false
+  },
+  "backends": {
+    "CPU_AND_NE": {
+      "passed": false,
+      "correct": 196,
+      "accuracy": 1.0,
+      "argmax_agreement": 196,
+      "max_abs_probability_error": 0.0053362250328063965,
+      "mean_row_max_abs_probability_error": 4.465129200521148e-05,
+      "max_abs_logit_error": 0.41709184646606445,
+      "load_seconds": 0.6060812089999672,
+      "warm_inference_latency_ms": {
+        "count": 196,
+        "median_ms": 0.9286874992540106,
+        "p95_ms": 1.1076562368543819,
+        "min_ms": 0.8750410052016377,
+        "max_ms": 1.3015410222578794
+      },
+      "reversed_option_order": {
+        "rows": 6,
+        "max_abs_probability_error": 3.1828880310058594e-05
+      },
+      "per_action": {
+        "fill": {
+          "rows": 36,
+          "correct": 36
+        },
+        "skip": {
+          "rows": 150,
+          "correct": 150
+        },
+        "check": {
+          "rows": 4,
+          "correct": 4
+        },
+        "click": {
+          "rows": 6,
+          "correct": 6
+        }
+      },
+      "decisions": [
+        {
+          "row": 0,
+          "label": 22,
+          "upstream": 22,
+          "coreml": 22,
+          "max_abs_probability_error": 3.1828880310058594e-05,
+          "max_abs_logit_error": 0.17777347564697266
+        },
+        {
+          "row": 1,
+          "label": 23,
+          "upstream": 23,
+          "coreml": 23,
+          "max_abs_probability_error": 3.814697265625e-05,
+          "max_abs_logit_error": 0.13521480560302734
+        },
+        {
+          "row": 2,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.21187782287597656
+        },
+        {
+          "row": 3,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 3.218650817871094e-05,
+          "max_abs_logit_error": 0.11447498202323914
+        },
+        {
+          "row": 4,
+          "label": 4,
+          "upstream": 4,
+          "coreml": 4,
+          "max_abs_probability_error": 2.944469451904297e-05,
+          "max_abs_logit_error": 0.20640206336975098
+        },
+        {
+          "row": 5,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 1.2636184692382812e-05,
+          "max_abs_logit_error": 0.26122236251831055
+        },
+        {
+          "row": 6,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 2.3365020751953125e-05,
+          "max_abs_logit_error": 0.15306663513183594
+        },
+        {
+          "row": 7,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 0.00014638900756835938,
+          "max_abs_logit_error": 0.14249515533447266
+        },
+        {
+          "row": 8,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 1.1920928955078125e-06,
+          "max_abs_logit_error": 0.15889453887939453
+        },
+        {
+          "row": 9,
+          "label": 10,
+          "upstream": 10,
+          "coreml": 10,
+          "max_abs_probability_error": 2.1457672119140625e-05,
+          "max_abs_logit_error": 0.1523761749267578
+        },
+        {
+          "row": 10,
+          "label": 11,
+          "upstream": 11,
+          "coreml": 11,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 0.14359092712402344
+        },
+        {
+          "row": 11,
+          "label": 13,
+          "upstream": 13,
+          "coreml": 13,
+          "max_abs_probability_error": 3.2186508178710938e-06,
+          "max_abs_logit_error": 0.3163447380065918
+        },
+        {
+          "row": 12,
+          "label": 15,
+          "upstream": 15,
+          "coreml": 15,
+          "max_abs_probability_error": 2.384185791015625e-06,
+          "max_abs_logit_error": 0.19838738441467285
+        },
+        {
+          "row": 13,
+          "label": 16,
+          "upstream": 16,
+          "coreml": 16,
+          "max_abs_probability_error": 0.0005360841751098633,
+          "max_abs_logit_error": 0.15249061584472656
+        },
+        {
+          "row": 14,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8596649169921875e-05,
+          "max_abs_logit_error": 0.08178329467773438
+        },
+        {
+          "row": 15,
+          "label": 24,
+          "upstream": 24,
+          "coreml": 24,
+          "max_abs_probability_error": 1.6689300537109375e-05,
+          "max_abs_logit_error": 0.19499272108078003
+        },
+        {
+          "row": 16,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.440092489701783e-08,
+          "max_abs_logit_error": 0.10968685150146484
+        },
+        {
+          "row": 17,
+          "label": 25,
+          "upstream": 25,
+          "coreml": 25,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.22722625732421875
+        },
+        {
+          "row": 18,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 8.430758313693332e-09,
+          "max_abs_logit_error": 0.16490459442138672
+        },
+        {
+          "row": 19,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.0266471551243512e-09,
+          "max_abs_logit_error": 0.09899425506591797
+        },
+        {
+          "row": 20,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4500966588902884e-08,
+          "max_abs_logit_error": 0.16230183839797974
+        },
+        {
+          "row": 21,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.647379281599637e-11,
+          "max_abs_logit_error": 0.11515188217163086
+        },
+        {
+          "row": 22,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.14793038368225098
+        },
+        {
+          "row": 23,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 0.23419427871704102
+        },
+        {
+          "row": 24,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.5055844551924444e-11,
+          "max_abs_logit_error": 0.16013479232788086
+        },
+        {
+          "row": 25,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.995705792434137e-09,
+          "max_abs_logit_error": 0.2223043441772461
+        },
+        {
+          "row": 26,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.470584068551943e-10,
+          "max_abs_logit_error": 0.15115368366241455
+        },
+        {
+          "row": 27,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.973024935241767e-11,
+          "max_abs_logit_error": 0.1729874610900879
+        },
+        {
+          "row": 28,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.994321927715873e-09,
+          "max_abs_logit_error": 0.24140071868896484
+        },
+        {
+          "row": 29,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4426039316183257e-11,
+          "max_abs_logit_error": 0.09259410947561264
+        },
+        {
+          "row": 30,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.1546640396118164
+        },
+        {
+          "row": 31,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.78299757975276e-10,
+          "max_abs_logit_error": 0.11323285102844238
+        },
+        {
+          "row": 32,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.185003176284454e-09,
+          "max_abs_logit_error": 0.17779135704040527
+        },
+        {
+          "row": 33,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.4556297823229443e-09,
+          "max_abs_logit_error": 0.19167208671569824
+        },
+        {
+          "row": 34,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 0.0004820823669433594,
+          "max_abs_logit_error": 0.18633151054382324
+        },
+        {
+          "row": 35,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.462750809366753e-09,
+          "max_abs_logit_error": 0.1198587417602539
+        },
+        {
+          "row": 36,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.7510341227277024e-10,
+          "max_abs_logit_error": 0.04924964904785156
+        },
+        {
+          "row": 37,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.657592190757896e-08,
+          "max_abs_logit_error": 0.0999135971069336
+        },
+        {
+          "row": 38,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 9.552543478452691e-11,
+          "max_abs_logit_error": 0.07785224914550781
+        },
+        {
+          "row": 39,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 4.04716482549361e-10,
+          "max_abs_logit_error": 0.11640000343322754
+        },
+        {
+          "row": 40,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 0.00021719932556152344,
+          "max_abs_logit_error": 0.18055367469787598
+        },
+        {
+          "row": 41,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.146233744175333e-09,
+          "max_abs_logit_error": 0.2431175708770752
+        },
+        {
+          "row": 42,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8819494851385343e-09,
+          "max_abs_logit_error": 0.14107990264892578
+        },
+        {
+          "row": 43,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.15815162658691406
+        },
+        {
+          "row": 44,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.780525865635596e-10,
+          "max_abs_logit_error": 0.0665130615234375
+        },
+        {
+          "row": 45,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.414331317674879e-11,
+          "max_abs_logit_error": 0.11009097099304199
+        },
+        {
+          "row": 46,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.599257724142603e-10,
+          "max_abs_logit_error": 0.10155582427978516
+        },
+        {
+          "row": 47,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.212314607705437e-11,
+          "max_abs_logit_error": 0.09491920471191406
+        },
+        {
+          "row": 48,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8596649169921875e-05,
+          "max_abs_logit_error": 0.08178329467773438
+        },
+        {
+          "row": 49,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.932842080831776e-10,
+          "max_abs_logit_error": 0.1341402530670166
+        },
+        {
+          "row": 50,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.440092489701783e-08,
+          "max_abs_logit_error": 0.10968685150146484
+        },
+        {
+          "row": 51,
+          "label": 25,
+          "upstream": 25,
+          "coreml": 25,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.22722625732421875
+        },
+        {
+          "row": 52,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 8.430758313693332e-09,
+          "max_abs_logit_error": 0.16490459442138672
+        },
+        {
+          "row": 53,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.0266471551243512e-09,
+          "max_abs_logit_error": 0.09899425506591797
+        },
+        {
+          "row": 54,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4500966588902884e-08,
+          "max_abs_logit_error": 0.16230183839797974
+        },
+        {
+          "row": 55,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.647379281599637e-11,
+          "max_abs_logit_error": 0.11515188217163086
+        },
+        {
+          "row": 56,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.14793038368225098
+        },
+        {
+          "row": 57,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 0.23419427871704102
+        },
+        {
+          "row": 58,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.5055844551924444e-11,
+          "max_abs_logit_error": 0.16013479232788086
+        },
+        {
+          "row": 59,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.995705792434137e-09,
+          "max_abs_logit_error": 0.2223043441772461
+        },
+        {
+          "row": 60,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.470584068551943e-10,
+          "max_abs_logit_error": 0.15115368366241455
+        },
+        {
+          "row": 61,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.973024935241767e-11,
+          "max_abs_logit_error": 0.1729874610900879
+        },
+        {
+          "row": 62,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.994321927715873e-09,
+          "max_abs_logit_error": 0.24140071868896484
+        },
+        {
+          "row": 63,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4426039316183257e-11,
+          "max_abs_logit_error": 0.09259410947561264
+        },
+        {
+          "row": 64,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.1546640396118164
+        },
+        {
+          "row": 65,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.78299757975276e-10,
+          "max_abs_logit_error": 0.11323285102844238
+        },
+        {
+          "row": 66,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.185003176284454e-09,
+          "max_abs_logit_error": 0.17779135704040527
+        },
+        {
+          "row": 67,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.4556297823229443e-09,
+          "max_abs_logit_error": 0.19167208671569824
+        },
+        {
+          "row": 68,
+          "label": 0,
+          "upstream": 0,
+          "coreml": 0,
+          "max_abs_probability_error": 3.147125244140625e-05,
+          "max_abs_logit_error": 0.16318607330322266
+        },
+        {
+          "row": 69,
+          "label": 1,
+          "upstream": 1,
+          "coreml": 1,
+          "max_abs_probability_error": 5.030632019042969e-05,
+          "max_abs_logit_error": 0.16801834106445312
+        },
+        {
+          "row": 70,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.12366819381713867
+        },
+        {
+          "row": 71,
+          "label": 3,
+          "upstream": 3,
+          "coreml": 3,
+          "max_abs_probability_error": 5.2928924560546875e-05,
+          "max_abs_logit_error": 0.23097705841064453
+        },
+        {
+          "row": 72,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 4.649162292480469e-06,
+          "max_abs_logit_error": 0.28166770935058594
+        },
+        {
+          "row": 73,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 2.1576881408691406e-05,
+          "max_abs_logit_error": 0.14033889770507812
+        },
+        {
+          "row": 74,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 4.470348358154297e-05,
+          "max_abs_logit_error": 0.1834554672241211
+        },
+        {
+          "row": 75,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 1.1563301086425781e-05,
+          "max_abs_logit_error": 0.16341686248779297
+        },
+        {
+          "row": 76,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 2.86102294921875e-06,
+          "max_abs_logit_error": 0.16710281372070312
+        },
+        {
+          "row": 77,
+          "label": 13,
+          "upstream": 13,
+          "coreml": 13,
+          "max_abs_probability_error": 0.0003635883331298828,
+          "max_abs_logit_error": 0.22506141662597656
+        },
+        {
+          "row": 78,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.10105061531066895
+        },
+        {
+          "row": 79,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 0.0003609657287597656,
+          "max_abs_logit_error": 0.19726498425006866
+        },
+        {
+          "row": 80,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.225250244140625
+        },
+        {
+          "row": 81,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.2853279113769531
+        },
+        {
+          "row": 82,
+          "label": 19,
+          "upstream": 19,
+          "coreml": 19,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.2046489715576172
+        },
+        {
+          "row": 83,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.0942090545902374e-09,
+          "max_abs_logit_error": 0.17853069305419922
+        },
+        {
+          "row": 84,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.1555185317993164
+        },
+        {
+          "row": 85,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.5405053888659666e-10,
+          "max_abs_logit_error": 0.24918603897094727
+        },
+        {
+          "row": 86,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.76837158203125e-07,
+          "max_abs_logit_error": 0.1990499496459961
+        },
+        {
+          "row": 87,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.2444934844970703
+        },
+        {
+          "row": 88,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3188037328859537e-09,
+          "max_abs_logit_error": 0.23791265487670898
+        },
+        {
+          "row": 89,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9970605169561395e-09,
+          "max_abs_logit_error": 0.13373339176177979
+        },
+        {
+          "row": 90,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7617375336342889e-09,
+          "max_abs_logit_error": 0.22733306884765625
+        },
+        {
+          "row": 91,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.612549260787688e-10,
+          "max_abs_logit_error": 0.1239767074584961
+        },
+        {
+          "row": 92,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.506330630512425e-12,
+          "max_abs_logit_error": 0.19014644622802734
+        },
+        {
+          "row": 93,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.5150824594911683e-08,
+          "max_abs_logit_error": 0.2004070281982422
+        },
+        {
+          "row": 94,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3666674458789885e-09,
+          "max_abs_logit_error": 0.13628053665161133
+        },
+        {
+          "row": 95,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.969476136678196e-10,
+          "max_abs_logit_error": 0.27038145065307617
+        },
+        {
+          "row": 96,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.657726538846106e-10,
+          "max_abs_logit_error": 0.16453242301940918
+        },
+        {
+          "row": 97,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.1947021484375
+        },
+        {
+          "row": 98,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.41709184646606445
+        },
+        {
+          "row": 99,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.1457672119140625e-06,
+          "max_abs_logit_error": 0.16466069221496582
+        },
+        {
+          "row": 100,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.16838455200195312
+        },
+        {
+          "row": 101,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.789079107084035e-08,
+          "max_abs_logit_error": 0.12038791179656982
+        },
+        {
+          "row": 102,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.5862225694339713e-09,
+          "max_abs_logit_error": 0.16624808311462402
+        },
+        {
+          "row": 103,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 0.00024586915969848633,
+          "max_abs_logit_error": 0.19487690925598145
+        },
+        {
+          "row": 104,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.841255187988281e-06,
+          "max_abs_logit_error": 0.13808655738830566
+        },
+        {
+          "row": 105,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.928799035575082e-10,
+          "max_abs_logit_error": 0.13288813829421997
+        },
+        {
+          "row": 106,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.4136979082101107e-09,
+          "max_abs_logit_error": 0.0866403579711914
+        },
+        {
+          "row": 107,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.302839361538304e-10,
+          "max_abs_logit_error": 0.11413812637329102
+        },
+        {
+          "row": 108,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.18335914611816406
+        },
+        {
+          "row": 109,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.10105061531066895
+        },
+        {
+          "row": 110,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.1961498260498047
+        },
+        {
+          "row": 111,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.811981201171875e-05,
+          "max_abs_logit_error": 0.30625247955322266
+        },
+        {
+          "row": 112,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.2853279113769531
+        },
+        {
+          "row": 113,
+          "label": 19,
+          "upstream": 19,
+          "coreml": 19,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.2046489715576172
+        },
+        {
+          "row": 114,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.0942090545902374e-09,
+          "max_abs_logit_error": 0.17853069305419922
+        },
+        {
+          "row": 115,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.1555185317993164
+        },
+        {
+          "row": 116,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.5405053888659666e-10,
+          "max_abs_logit_error": 0.24918603897094727
+        },
+        {
+          "row": 117,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.76837158203125e-07,
+          "max_abs_logit_error": 0.1990499496459961
+        },
+        {
+          "row": 118,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.2444934844970703
+        },
+        {
+          "row": 119,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3188037328859537e-09,
+          "max_abs_logit_error": 0.23791265487670898
+        },
+        {
+          "row": 120,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9970605169561395e-09,
+          "max_abs_logit_error": 0.13373339176177979
+        },
+        {
+          "row": 121,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7617375336342889e-09,
+          "max_abs_logit_error": 0.22733306884765625
+        },
+        {
+          "row": 122,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.612549260787688e-10,
+          "max_abs_logit_error": 0.1239767074584961
+        },
+        {
+          "row": 123,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.506330630512425e-12,
+          "max_abs_logit_error": 0.19014644622802734
+        },
+        {
+          "row": 124,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.5150824594911683e-08,
+          "max_abs_logit_error": 0.2004070281982422
+        },
+        {
+          "row": 125,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3666674458789885e-09,
+          "max_abs_logit_error": 0.13628053665161133
+        },
+        {
+          "row": 126,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.969476136678196e-10,
+          "max_abs_logit_error": 0.27038145065307617
+        },
+        {
+          "row": 127,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.657726538846106e-10,
+          "max_abs_logit_error": 0.16453242301940918
+        },
+        {
+          "row": 128,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.1947021484375
+        },
+        {
+          "row": 129,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.41709184646606445
+        },
+        {
+          "row": 130,
+          "label": 1,
+          "upstream": 1,
+          "coreml": 1,
+          "max_abs_probability_error": 3.170967102050781e-05,
+          "max_abs_logit_error": 0.12571239471435547
+        },
+        {
+          "row": 131,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.0871438980102539
+        },
+        {
+          "row": 132,
+          "label": 3,
+          "upstream": 3,
+          "coreml": 3,
+          "max_abs_probability_error": 2.7894973754882812e-05,
+          "max_abs_logit_error": 0.16924285888671875
+        },
+        {
+          "row": 133,
+          "label": 4,
+          "upstream": 4,
+          "coreml": 4,
+          "max_abs_probability_error": 1.0967254638671875e-05,
+          "max_abs_logit_error": 0.21336686611175537
+        },
+        {
+          "row": 134,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 6.258487701416016e-05,
+          "max_abs_logit_error": 0.16839587688446045
+        },
+        {
+          "row": 135,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 1.2993812561035156e-05,
+          "max_abs_logit_error": 0.12725162506103516
+        },
+        {
+          "row": 136,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 6.198883056640625e-06,
+          "max_abs_logit_error": 0.1457538604736328
+        },
+        {
+          "row": 137,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 0.0053362250328063965,
+          "max_abs_logit_error": 0.15258073806762695
+        },
+        {
+          "row": 138,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 0.00022917985916137695,
+          "max_abs_logit_error": 0.12661266326904297
+        },
+        {
+          "row": 139,
+          "label": 10,
+          "upstream": 10,
+          "coreml": 10,
+          "max_abs_probability_error": 4.851818084716797e-05,
+          "max_abs_logit_error": 0.2998933792114258
+        },
+        {
+          "row": 140,
+          "label": 11,
+          "upstream": 11,
+          "coreml": 11,
+          "max_abs_probability_error": 3.0994415283203125e-06,
+          "max_abs_logit_error": 0.19615697860717773
+        },
+        {
+          "row": 141,
+          "label": 12,
+          "upstream": 12,
+          "coreml": 12,
+          "max_abs_probability_error": 9.34600830078125e-05,
+          "max_abs_logit_error": 0.11831855773925781
+        },
+        {
+          "row": 142,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.14543867111206055
+        },
+        {
+          "row": 143,
+          "label": 16,
+          "upstream": 16,
+          "coreml": 16,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.13959884643554688
+        },
+        {
+          "row": 144,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.1920928955078125e-06,
+          "max_abs_logit_error": 0.20041000843048096
+        },
+        {
+          "row": 145,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.6954879760742188e-06,
+          "max_abs_logit_error": 0.14644718170166016
+        },
+        {
+          "row": 146,
+          "label": 17,
+          "upstream": 17,
+          "coreml": 17,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.3961820602416992
+        },
+        {
+          "row": 147,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.429776876539563e-09,
+          "max_abs_logit_error": 0.22239947319030762
+        },
+        {
+          "row": 148,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 9.215698426601193e-09,
+          "max_abs_logit_error": 0.13992595672607422
+        },
+        {
+          "row": 149,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.84208304060607e-10,
+          "max_abs_logit_error": 0.16092300415039062
+        },
+        {
+          "row": 150,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.947568818333917e-12,
+          "max_abs_logit_error": 0.15441608428955078
+        },
+        {
+          "row": 151,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.291168614363073e-10,
+          "max_abs_logit_error": 0.15936005115509033
+        },
+        {
+          "row": 152,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.924417788629512e-10,
+          "max_abs_logit_error": 0.173073410987854
+        },
+        {
+          "row": 153,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.837478791344438e-08,
+          "max_abs_logit_error": 0.11217308044433594
+        },
+        {
+          "row": 154,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.3051329034750054e-11,
+          "max_abs_logit_error": 0.14353275299072266
+        },
+        {
+          "row": 155,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.650468511860126e-10,
+          "max_abs_logit_error": 0.1971302032470703
+        },
+        {
+          "row": 156,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.181078218703284e-12,
+          "max_abs_logit_error": 0.14509010314941406
+        },
+        {
+          "row": 157,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.6455586893115992e-10,
+          "max_abs_logit_error": 0.2037668228149414
+        },
+        {
+          "row": 158,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.109900363426846e-10,
+          "max_abs_logit_error": 0.12031292915344238
+        },
+        {
+          "row": 159,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.693580800742893e-09,
+          "max_abs_logit_error": 0.18417954444885254
+        },
+        {
+          "row": 160,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.3718365998727222e-08,
+          "max_abs_logit_error": 0.17492496967315674
+        },
+        {
+          "row": 161,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.09895849227905273
+        },
+        {
+          "row": 162,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.042114572413766e-08,
+          "max_abs_logit_error": 0.22567105293273926
+        },
+        {
+          "row": 163,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9131388140358752e-10,
+          "max_abs_logit_error": 0.08278036117553711
+        },
+        {
+          "row": 164,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0254025184508464e-08,
+          "max_abs_logit_error": 0.0756688117980957
+        },
+        {
+          "row": 165,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.15840287151903e-11,
+          "max_abs_logit_error": 0.18943119049072266
+        },
+        {
+          "row": 166,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.6797479790729994e-09,
+          "max_abs_logit_error": 0.08643817901611328
+        },
+        {
+          "row": 167,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.54509399403247e-10,
+          "max_abs_logit_error": 0.2648029327392578
+        },
+        {
+          "row": 168,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.2841672897338867
+        },
+        {
+          "row": 169,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.4169340140378637e-11,
+          "max_abs_logit_error": 0.10731315612792969
+        },
+        {
+          "row": 170,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.0329603412093675e-11,
+          "max_abs_logit_error": 0.14229869842529297
+        },
+        {
+          "row": 171,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.10416412353515625
+        },
+        {
+          "row": 172,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 4.66337235494052e-09,
+          "max_abs_logit_error": 0.08718490600585938
+        },
+        {
+          "row": 173,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.2885730260592254e-09,
+          "max_abs_logit_error": 0.07872724533081055
+        },
+        {
+          "row": 174,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.1279766010119374e-09,
+          "max_abs_logit_error": 0.17765092849731445
+        },
+        {
+          "row": 175,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.14543867111206055
+        },
+        {
+          "row": 176,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0728836059570312e-06,
+          "max_abs_logit_error": 0.14330339431762695
+        },
+        {
+          "row": 177,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.2799782567185503e-08,
+          "max_abs_logit_error": 0.151580810546875
+        },
+        {
+          "row": 178,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.6954879760742188e-06,
+          "max_abs_logit_error": 0.14644718170166016
+        },
+        {
+          "row": 179,
+          "label": 17,
+          "upstream": 17,
+          "coreml": 17,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.3961820602416992
+        },
+        {
+          "row": 180,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.429776876539563e-09,
+          "max_abs_logit_error": 0.22239947319030762
+        },
+        {
+          "row": 181,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 9.215698426601193e-09,
+          "max_abs_logit_error": 0.13992595672607422
+        },
+        {
+          "row": 182,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.84208304060607e-10,
+          "max_abs_logit_error": 0.16092300415039062
+        },
+        {
+          "row": 183,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.947568818333917e-12,
+          "max_abs_logit_error": 0.15441608428955078
+        },
+        {
+          "row": 184,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.291168614363073e-10,
+          "max_abs_logit_error": 0.15936005115509033
+        },
+        {
+          "row": 185,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.924417788629512e-10,
+          "max_abs_logit_error": 0.173073410987854
+        },
+        {
+          "row": 186,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.837478791344438e-08,
+          "max_abs_logit_error": 0.11217308044433594
+        },
+        {
+          "row": 187,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.3051329034750054e-11,
+          "max_abs_logit_error": 0.14353275299072266
+        },
+        {
+          "row": 188,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.650468511860126e-10,
+          "max_abs_logit_error": 0.1971302032470703
+        },
+        {
+          "row": 189,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.181078218703284e-12,
+          "max_abs_logit_error": 0.14509010314941406
+        },
+        {
+          "row": 190,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.6455586893115992e-10,
+          "max_abs_logit_error": 0.2037668228149414
+        },
+        {
+          "row": 191,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.109900363426846e-10,
+          "max_abs_logit_error": 0.12031292915344238
+        },
+        {
+          "row": 192,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.693580800742893e-09,
+          "max_abs_logit_error": 0.18417954444885254
+        },
+        {
+          "row": 193,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.3718365998727222e-08,
+          "max_abs_logit_error": 0.17492496967315674
+        },
+        {
+          "row": 194,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.09895849227905273
+        },
+        {
+          "row": 195,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.042114572413766e-08,
+          "max_abs_logit_error": 0.22567105293273926
+        }
+      ]
+    }
+  },
+  "passed": false,
+  "code_sha256": {
+    "ane_gather.py": "c3379a3797041dcfed82eaaea3990771ec91f1dff7ec146b025988ad7a8c9aae",
+    "assets.py": "f7a5bc931075237e00c73bf21fee3575be35531f934f23b5e22627d50a79f92f",
+    "benchmark-synthetic.py": "891da8408156e2b80849b4d799c77c9a31c091dfd999fcd044639deefac3e888",
+    "compare-ane.py": "4633df7ad5f90d6aed37d5ef4f7a2d2231fdadd272ef1aa3057c237b2f937df8",
+    "convert-coreml.py": "46fe0194df924033126286e601214b6789f5d667aae38c153cf4dbc1b0e629b0",
+    "export-reference.py": "ff9eab4d6e4e72ffaf074d82610f41fbea1e802da7c0e987ce562a89eb561b99",
+    "export_model.py": "c61da5d678c77030d1e9a78359626063ea31da46fb6434e49517fdeaa7800949",
+    "preprocessing.py": "18c74ede43f95f91cd638c7e9631923a353e3a24d3b522ddb8b18b480f9c2846",
+    "profile-coreml.py": "d5e1386f54fc6c482647120e593e7ffbbe4bbb277c2cc6d4949154660fe239fd",
+    "quantize-int8.py": "22071babfcb02ef77225203a479d9bb14b9fa35462ba38bcc4bf07d29c125a97",
+    "score-report.py": "c78e45caa86fb59f9a56ef18cc6a35020ea6e281a6a23d23618f56da85a0d95e",
+    "synthetic_test.py": "e2b68725236e6827eb422480c8cc8aa189b2490c1594650d8f2928771d1bc626",
+    "verify.py": "bc20ec9df47708e9b48e411076abe4eab7c18313f67e11334de2d9e3ecb3efa7"
+  }
+}

reports/int8-fallback.json ADDED Viewed

	@@ -0,0 +1,112 @@

+{
+  "hardware": {
+    "device": "arm64",
+    "chip": "Apple M5 Pro",
+    "ram": "24GB",
+    "os_version": "macOS 27.0",
+    "timestamp": "2026-09-19T22:05:35.064926+00:00"
+  },
+  "models": [
+    {
+      "model_path": "/Users/hanweng/Documents/FluidAudio/.mobius/cua-s1-prs/mobius/models/computer-use/cua-s1-forms/coreml/build/int8-weights/cua_s1_forms_int8_options32.mlmodelc",
+      "model_name": "cua_s1_forms_int8_options32",
+      "fallback": {
+        "compute_units": "cpu_and_neural_engine",
+        "total_ops": 200,
+        "ane_ops": 149,
+        "gpu_ops": 0,
+        "cpu_ops": 51,
+        "ane_percent": 74.5,
+        "reasons": [
+          {
+            "reason": "Unsupported tensor data type: int32",
+            "count": 25,
+            "estimated_cpu_runtime_ms": 0.0375,
+            "op_types": {
+              "ios17.cast": 10,
+              "ios17.slice_by_index": 4,
+              "ios17.add": 4,
+              "select": 4,
+              "ios17.gather": 2,
+              "ios17.reshape": 1
+            },
+            "ops": [
+              "var_27",
+              "var_43",
+              "var_43_to_fp16",
+              "context_ids_to_int16",
+              "cast_54",
+              "add_3",
+              "select_0",
+              "select_0_to_int16",
+              "cast_0",
+              "add_0",
+              "select_0_1",
+              "op_59_cast_fp16_cast_uint16_cast_uint16",
+              "flat_ids",
+              "var_333",
+              "var_349",
+              "var_349_to_fp16",
+              "flat_ids_to_int16",
+              "cast_55",
+              "add_4",
+              "select_1",
+              "select_1_to_int16",
+              "cast_1",
+              "add_1",
+              "select_1_1",
+              "op_365_cast_fp16_cast_uint16_cast_uint16"
+            ]
+          },
+          {
+            "reason": "ANE not available for this op",
+            "count": 19,
+            "estimated_cpu_runtime_ms": 0,
+            "op_types": {
+              "ios16.constexpr_affine_dequantize": 19
+            },
+            "ops": [
+              "model_embedding_weight_to_fp16_quantized",
+              "model_position_weight_to_fp16_quantized",
+              "model_encoder_layers_0_self_attn_in_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_0_self_attn_out_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_0_linear1_weight_to_fp16_quantized",
+              "model_encoder_layers_0_linear2_weight_to_fp16_quantized",
+              "model_encoder_layers_1_self_attn_in_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_1_self_attn_out_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_1_linear1_weight_to_fp16_quantized",
+              "model_encoder_layers_1_linear2_weight_to_fp16_quantized",
+              "op_369_to_fp16_quantized",
+              "op_394_to_fp16_quantized",
+              "model_option_encoder_layers_0_self_attn_in_proj_weight_to_fp16_quantized",
+              "model_option_encoder_layers_0_self_attn_out_proj_weight_to_fp16_quantized",
+              "model_option_encoder_layers_0_linear1_weight_to_fp16_quantized",
+              "model_option_encoder_layers_0_linear2_weight_to_fp16_quantized",
+              "model_head_query_weight_to_fp16_quantized",
+              "model_head_key_weight_to_fp16_quantized",
+              "model_head_value_weight_to_fp16_quantized"
+            ]
+          },
+          {
+            "reason": "Unable to resolve operation input \"y\".",
+            "count": 7,
+            "estimated_cpu_runtime_ms": 0.007,
+            "op_types": {
+              "ios17.greater_equal": 4,
+              "ios17.not_equal": 3
+            },
+            "ops": [
+              "option_mask_1",
+              "context_mask",
+              "greater_equal_0",
+              "greater_equal_0_1",
+              "flat_mask",
+              "greater_equal_1",
+              "greater_equal_1_1"
+            ]
+          }
+        ]
+      }
+    }
+  ]
+}

reports/int8-profile.json ADDED Viewed

	@@ -0,0 +1,1105 @@

+{
+  "created_utc": "2026-09-19T22:05:06.621713+00:00",
+  "purpose": "Bounded real-input latency and scheduler placement; not a general accuracy benchmark",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "package_files": {
+    "Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
+    "Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
+    "Manifest.json": "2206ec01787a85c12f1376fa5a832c5ac912f06172c6ff06b26ae8415470d246"
+  },
+  "dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "dataset_sha256": "4f43b442e79ba2e2ce731e27e9b8e340c2b5dfcaffc92d8ff564c34f115ff1ca",
+  "environment": {
+    "chip": "Apple M5 Pro",
+    "memory_bytes": 25769803776,
+    "macos": "27.0",
+    "os_build": "26A428",
+    "python": "3.11.11",
+    "coremltools": "9.0"
+  },
+  "protocol": {
+    "rows": [
+      0,
+      68,
+      130
+    ],
+    "pages": [
+      "patient-registration",
+      "job-application",
+      "auto-claim"
+    ],
+    "option_counts": [
+      27,
+      21,
+      19
+    ],
+    "first_call_row": 0,
+    "warmup_passes": 2,
+    "timed_passes": 10,
+    "backend_order": [
+      "CPU_AND_NE"
+    ],
+    "timing_scope": "Synchronous Python MLModel.predict; pre-encoded inputs; output checks outside timer",
+    "load_scope": "MLModel construction; system caches retained, not first-install cold start",
+    "first_call_scope": "First predict after this load; may use existing system/ANE caches",
+    "plan_order": "Compute plan loaded after timing each backend"
+  },
+  "backends": {
+    "CPU_AND_NE": {
+      "load_ms": 634.8958330054302,
+      "first_prediction_ms": 3.1584999815095216,
+      "warm_prediction": {
+        "count": 30,
+        "median_ms": 1.0110830189660192,
+        "p95_ms": 1.124097959836945,
+        "min_ms": 0.9326659783255309,
+        "max_ms": 1.1762090143747628
+      },
+      "timed_predictions_correct": 30,
+      "calls": [
+        {
+          "row": 0,
+          "milliseconds": 1.089458994101733
+        },
+        {
+          "row": 68,
+          "milliseconds": 1.0625830036588013
+        },
+        {
+          "row": 130,
+          "milliseconds": 1.1127080069854856
+        },
+        {
+          "row": 0,
+          "milliseconds": 1.0287080076523125
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9326659783255309
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9901669982355088
+        },
+        {
+          "row": 0,
+          "milliseconds": 1.0140000085812062
+        },
+        {
+          "row": 68,
+          "milliseconds": 1.0877909953705966
+        },
+        {
+          "row": 130,
+          "milliseconds": 1.0328329808544368
+        },
+        {
+          "row": 0,
+          "milliseconds": 1.047459023538977
+        },
+        {
+          "row": 68,
+          "milliseconds": 1.0382909968029708
+        },
+        {
+          "row": 130,
+          "milliseconds": 1.1334170121699572
+        },
+        {
+          "row": 0,
+          "milliseconds": 1.1762090143747628
+        },
+        {
+          "row": 68,
+          "milliseconds": 1.0909580159932375
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9953750122804195
+        },
+        {
+          "row": 0,
+          "milliseconds": 1.0375829879194498
+        },
+        {
+          "row": 68,
+          "milliseconds": 1.0102080123033375
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9982500050682575
+        },
+        {
+          "row": 0,
+          "milliseconds": 1.0159579978790134
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9699999936856329
+        },
+        {
+          "row": 130,
+          "milliseconds": 1.0119580256287009
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9712500032037497
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.980541022727266
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9917500137817115
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9814579971134663
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.942542013945058
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.93891701544635
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9628330008126795
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9442089940421283
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9703329997137189
+        }
+      ],
+      "compute_plan": {
+        "kind": "MLComputePlan preferred device; not a runtime utilization or power trace",
+        "assigned_operation_count": 181,
+        "device_counts": {
+          "cpu": 32,
+          "gpu": 0,
+          "ane": 149,
+          "unknown": 0
+        },
+        "unassigned_operation_types": {
+          "const": 226,
+          "ios16.constexpr_affine_dequantize": 19
+        },
+        "operations": [
+          {
+            "name": "option_mask_1",
+            "type": "ios17.not_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "context_mask",
+            "type": "ios17.not_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "var_27",
+            "type": "ios17.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "fill_like_0_cast_fp16",
+            "type": "ios16.fill_like",
+            "device": "ane"
+          },
+          {
+            "name": "var_43",
+            "type": "ios17.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "var_43_to_fp16",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "safe_context_ids_cast_fp16",
+            "type": "ios17.concat",
+            "device": "ane"
+          },
+          {
+            "name": "context_ids_to_int16",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_54",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_0",
+            "type": "ios17.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_3",
+            "type": "ios17.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_0",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "select_0_to_int16",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_0",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_0_1",
+            "type": "ios17.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_0",
+            "type": "ios17.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_0_1",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "op_59_cast_fp16_cast_uint16_cast_uint16",
+            "type": "ios17.gather",
+            "device": "cpu"
+          },
+          {
+            "name": "src_1_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "mask_1_cast_fp16",
+            "type": "ios17.equal",
+            "device": "ane"
+          },
+          {
+            "name": "key_padding_mask_1_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "query_1_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "query_3_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "linear_0_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "var_124_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_125_cast_fp16",
+            "type": "ios17.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_126_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "var_127_cast_fp16",
+            "type": "ios17.squeeze",
+            "device": "ane"
+          },
+          {
+            "name": "q_1_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "k_1_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "v_1_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "var_136_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_143_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_150_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_154_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_156_cast_fp16",
+            "type": "tile",
+            "device": "ane"
+          },
+          {
+            "name": "q_3_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "q_5_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "k_3_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "k_5_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "v_3_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "v_5_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "mul_1_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_0_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "add_0_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "softmax_0_cast_fp16",
+            "type": "ios16.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_1_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_172_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_3_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "linear_1_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_7_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_3_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "input_5_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "input_7_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_2_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_11_cast_fp16",
+            "type": "ios16.relu",
+            "device": "ane"
+          },
+          {
+            "name": "linear_3_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_17_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "query_5_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "query_7_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "linear_4_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "var_231_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_232_cast_fp16",
+            "type": "ios17.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_233_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "var_234_cast_fp16",
+            "type": "ios17.squeeze",
+            "device": "ane"
+          },
+          {
+            "name": "q_7_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "k_7_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "v_7_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "var_243_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_250_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_257_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "q_9_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "q_11_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "k_9_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "k_11_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "v_9_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "v_11_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "mul_3_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_1_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "add_1_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "softmax_1_cast_fp16",
+            "type": "ios16.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_9_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_279_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_11_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "linear_5_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_15_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_19_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "input_21_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "input_23_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_6_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_27_cast_fp16",
+            "type": "ios16.relu",
+            "device": "ane"
+          },
+          {
+            "name": "linear_7_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "context_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "flat_ids",
+            "type": "ios17.reshape",
+            "device": "cpu"
+          },
+          {
+            "name": "flat_mask",
+            "type": "ios17.not_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "var_333",
+            "type": "ios17.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "fill_like_1_cast_fp16",
+            "type": "ios16.fill_like",
+            "device": "ane"
+          },
+          {
+            "name": "var_349",
+            "type": "ios17.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "var_349_to_fp16",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "safe_ids_cast_fp16",
+            "type": "ios17.concat",
+            "device": "ane"
+          },
+          {
+            "name": "flat_ids_to_int16",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_55",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_1",
+            "type": "ios17.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_4",
+            "type": "ios17.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_1",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "select_1_to_int16",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_1",
+            "type": "ios17.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_1_1",
+            "type": "ios17.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_1",
+            "type": "ios17.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_1_1",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "op_365_cast_fp16_cast_uint16_cast_uint16",
+            "type": "ios17.gather",
+            "device": "cpu"
+          },
+          {
+            "name": "src_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "mask_cast_fp16",
+            "type": "ios17.equal",
+            "device": "ane"
+          },
+          {
+            "name": "key_padding_mask_7_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "query_9_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "query_11_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "linear_8_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "var_428_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_429_cast_fp16",
+            "type": "ios17.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_430_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "var_431_cast_fp16",
+            "type": "ios17.squeeze",
+            "device": "ane"
+          },
+          {
+            "name": "q_13_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "k_13_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "v_13_cast_fp16",
+            "type": "ios17.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "var_440_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_447_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_454_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_458_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_460_cast_fp16",
+            "type": "tile",
+            "device": "ane"
+          },
+          {
+            "name": "q_15_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "q_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "k_15_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "k_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "v_15_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "v_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "mul_5_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_2_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "add_2_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "softmax_2_cast_fp16",
+            "type": "ios16.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_17_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_476_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_19_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "linear_9_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_35_cast_fp16",
+            "type": "ios17.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "input_37_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "input_39_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_10_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_43_cast_fp16",
+            "type": "ios16.relu",
+            "device": "ane"
+          },
+          {
+            "name": "linear_11_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "hidden_cast_fp16",
+            "type": "ios17.add",
+            "device": "ane"
+          },
+          {
+            "name": "var_504",
+            "type": "ios17.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_504_to_fp16",
+            "type": "ios17.cast",
+            "device": "ane"
+          },
+          {
+            "name": "var_510_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_515_cast_fp16",
+            "type": "ios16.reduce_sum",
+            "device": "ane"
+          },
+          {
+            "name": "var_520_cast_fp16",
+            "type": "ios16.reduce_sum",
+            "device": "ane"
+          },
+          {
+            "name": "var_522_cast_fp16",
+            "type": "ios17.maximum",
+            "device": "ane"
+          },
+          {
+            "name": "pooled_cast_fp16",
+            "type": "ios17.real_div",
+            "device": "ane"
+          },
+          {
+            "name": "options_cast_fp16",
+            "type": "ios17.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "input_53_cast_fp16",
+            "type": "ios17.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_12_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "linear_13_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "linear_14_cast_fp16",
+            "type": "ios17.linear",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_3_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "_inversed_scores_1_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_567",
+            "type": "ios17.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_569",
+            "type": "ios17.logical_not",
+            "device": "ane"
+          },
+          {
+            "name": "scores_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "var_571_cast_fp16",
+            "type": "ios16.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_4_cast_fp16",
+            "type": "ios17.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_574_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_576_cast_fp16",
+            "type": "ios16.reduce_sum",
+            "device": "ane"
+          },
+          {
+            "name": "_inversed_logits_1_cast_fp16",
+            "type": "ios17.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_579",
+            "type": "ios17.logical_not",
+            "device": "ane"
+          },
+          {
+            "name": "logits_3_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "var_587_cast_fp16",
+            "type": "ios16.fill_like",
+            "device": "ane"
+          },
+          {
+            "name": "logits_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "var_591_cast_fp16",
+            "type": "ios16.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "logits",
+            "type": "ios17.cast",
+            "device": "ane"
+          },
+          {
+            "name": "probabilities",
+            "type": "ios17.cast",
+            "device": "ane"
+          }
+        ]
+      }
+    }
+  },
+  "script_sha256": "d5e1386f54fc6c482647120e593e7ffbbe4bbb277c2cc6d4949154660fe239fd"
+}

reports/int8-synthetic-test-decisions.jsonl.gz ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69720b9e11430ecff7ea0718f14bf16792de6f39f114faba250cfe855e4e4156
+size 2307507

reports/int8-synthetic-test.json ADDED Viewed

	@@ -0,0 +1,1473 @@

+{
+  "purpose": "Full published synthetic test; no fitting, filtering, resampling or test-based tuning",
+  "created_utc": "2026-09-19T22:07:40.987091+00:00",
+  "dataset": {
+    "dataset_repository": "cua-ai/cua-s1-forms",
+    "dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+    "path": "artifacts/test.jsonl",
+    "url": "https://huggingface.co/datasets/cua-ai/cua-s1-forms/resolve/8273f34778b99ac2e12d9f6e7d57dad99ae20845/test.jsonl",
+    "sha256": "d63a7e0db195d4d20154a40b2f8dd09ce3bb65487a158c638da5c609d4475e7c",
+    "rows": 24370,
+    "bytes": 18710924,
+    "upstream_split_claim": "Synthetic test, form-signature-disjoint from train/validation. Train and validation signatures have not been independently re-audited here.",
+    "model_card_claim": {
+      "synthetic_test_top1": 0.9995,
+      "approximate_decisions": 15000,
+      "note": "The released file has 24370 rows. No matching full result ledger or exact 15000-row manifest is supplied by the linked model card."
+    }
+  },
+  "manifest_sha256": "e582b951ecc07809a0b06a2bc0790827b90a86ce57c70d5b6177ef04f0b4f1ca",
+  "input_census": {
+    "rows": 24370,
+    "unique_episode_seeds": 1040,
+    "maxima": {
+      "context_bytes": 217,
+      "option_bytes": 92,
+      "max_options": 32
+    },
+    "per_action": {
+      "check": 816,
+      "click": 1040,
+      "fill": 9802,
+      "skip": 12712
+    },
+    "option_count_histogram": {
+      "7": 17,
+      "8": 46,
+      "9": 80,
+      "10": 227,
+      "11": 361,
+      "12": 589,
+      "13": 888,
+      "14": 1254,
+      "15": 1513,
+      "16": 1761,
+      "17": 2373,
+      "18": 2014,
+      "19": 2277,
+      "20": 2506,
+      "21": 2396,
+      "22": 1818,
+      "23": 1422,
+      "24": 907,
+      "25": 688,
+      "26": 524,
+      "27": 483,
+      "28": 89,
+      "29": 33,
+      "30": 29,
+      "31": 35,
+      "32": 40
+    },
+    "excluded_rows": 0,
+    "truncated_contexts": 0,
+    "truncated_options": 0
+  },
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "conversions": {
+    "baseline": {
+      "model": "cua_s1_forms_fp16_options32.mlpackage",
+      "precision": "float16",
+      "minimum_target": "iOS17/macOS14",
+      "limits": {
+        "context_bytes": 224,
+        "option_bytes": 96,
+        "max_options": 32
+      },
+      "model_config": {
+        "context_tokens": 224,
+        "encoder": "tinyx",
+        "heads": 4,
+        "hf_model": "Qwen/Qwen2.5-0.5B",
+        "layers": 2,
+        "option_tokens": 96,
+        "rank": 128,
+        "width": 128
+      },
+      "parameters": 706048,
+      "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+      "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+      "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+      "trace_row": 0,
+      "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+      "export_seconds": 0.5374167499830946,
+      "python": "3.11.11",
+      "torch": "2.7.0",
+      "coremltools": "9.0",
+      "package_files": {
+        "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+        "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+        "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+      }
+    },
+    "int8-weights": {
+      "model": "cua_s1_forms_int8_options32.mlpackage",
+      "precision": "int8_weights_float16_compute",
+      "minimum_target": "iOS17/macOS14",
+      "limits": {
+        "context_bytes": 224,
+        "option_bytes": 96,
+        "max_options": 32
+      },
+      "model_config": {
+        "context_tokens": 224,
+        "encoder": "tinyx",
+        "heads": 4,
+        "hf_model": "Qwen/Qwen2.5-0.5B",
+        "layers": 2,
+        "option_tokens": 96,
+        "rank": 128,
+        "width": 128
+      },
+      "parameters": 706048,
+      "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+      "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+      "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+      "trace_row": 0,
+      "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+      "export_seconds": 0.5374167499830946,
+      "python": "3.11.11",
+      "torch": "2.7.0",
+      "coremltools": "9.0",
+      "package_files": {
+        "Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
+        "Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
+        "Manifest.json": "2206ec01787a85c12f1376fa5a832c5ac912f06172c6ff06b26ae8415470d246"
+      },
+      "quantization": {
+        "name": "int8-weights",
+        "settings": {
+          "mode": "linear_symmetric",
+          "dtype": "int8",
+          "granularity": "per_channel",
+          "weight_threshold": 2048
+        },
+        "activations": "float16",
+        "calibration": "None; weight-only, data-free quantization",
+        "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+        "source_package_files": {
+          "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+          "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+          "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+        },
+        "source_package_bytes": 1511163,
+        "package_bytes": 813422,
+        "compression_seconds": 0.3180606249952689,
+        "operation_counts": {
+          "add": 15,
+          "cast": 13,
+          "concat": 2,
+          "const": 226,
+          "constexpr_affine_dequantize": 19,
+          "equal": 2,
+          "expand_dims": 5,
+          "fill_like": 3,
+          "gather": 2,
+          "greater_equal": 4,
+          "layer_norm": 8,
+          "linear": 15,
+          "logical_not": 2,
+          "matmul": 8,
+          "maximum": 1,
+          "mul": 7,
+          "not_equal": 3,
+          "real_div": 1,
+          "reduce_sum": 3,
+          "relu": 3,
+          "reshape": 31,
+          "select": 9,
+          "slice_by_index": 13,
+          "softmax": 5,
+          "squeeze": 3,
+          "tile": 2,
+          "transpose": 21
+        },
+        "script_sha256": "22071babfcb02ef77225203a479d9bb14b9fa35462ba38bcc4bf07d29c125a97"
+      }
+    }
+  },
+  "environment": {
+    "machine": "Apple M5 Pro",
+    "memory_bytes": 25769803776,
+    "os": "27.0",
+    "os_build": "26A428",
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "torch_cpu_threads": 2,
+    "torch_interop_threads": 1,
+    "torch_mha_fastpath": false
+  },
+  "protocol": {
+    "batch_size": 1,
+    "warmups_per_model": 3,
+    "timed_passes": 1,
+    "coreml_compute_units": "CPU_AND_NE",
+    "coreml_order": "AB on even rows, BA on odd rows",
+    "timing": "PyTorch CPU forward+softmax / synchronous Core ML predict; excludes encoding, validation, GUI and network",
+    "preprocessing": "Upstream variable-length collator; independent fixed-shape Core ML encoder shared by both exports",
+    "load_note": "Process/model creation can use existing system caches; not a cold-start experiment",
+    "output_audit": "Imperfect finite probability sums are retained unchanged and fail the original normalization gate",
+    "calibration_note": "NLL/ECE use raw emitted scores; no renormalization or calibration was applied"
+  },
+  "thresholds": {
+    "argmax_agreement": 1.0,
+    "max_abs_probability_error": 0.005,
+    "allow_accuracy_loss": false,
+    "probability_sum_atol": 0.001,
+    "probability_sum_rtol": 1e-05
+  },
+  "host_preprocessing_ms": {
+    "upstream_collator_ms": {
+      "count": 24370,
+      "median_ms": 0.09233300806954503,
+      "p95_ms": 0.12889755889773366,
+      "min_ms": 0.04445898230187595,
+      "max_ms": 1.0104170069098473
+    },
+    "shared_coreml_encoding_ms": {
+      "count": 24370,
+      "median_ms": 0.027417001547291875,
+      "p95_ms": 0.03995798761025071,
+      "min_ms": 0.012666976545006037,
+      "max_ms": 0.49662499804981053
+    }
+  },
+  "results": {
+    "upstream_pytorch": {
+      "correct": 24359,
+      "rows": 24370,
+      "accuracy": 0.999548625359048,
+      "model_call_latency_ms": {
+        "count": 24370,
+        "median_ms": 1.7357920005451888,
+        "p95_ms": 2.984285591810476,
+        "min_ms": 1.0271249921061099,
+        "max_ms": 30.681957985507324
+      },
+      "load_ms": 10.63050000811927,
+      "first_call_ms": 5.652791995089501,
+      "calibration": {
+        "nll": 0.0021302032038990836,
+        "ece": 0.00035450826133425167,
+        "equal_width_bins": 15,
+        "bins": [
+          {
+            "bin": 7,
+            "count": 2,
+            "accuracy": 0.5,
+            "mean_confidence": 0.5216051042079926
+          },
+          {
+            "bin": 8,
+            "count": 5,
+            "accuracy": 0.8,
+            "mean_confidence": 0.575071656703949
+          },
+          {
+            "bin": 9,
+            "count": 7,
+            "accuracy": 0.7142857142857143,
+            "mean_confidence": 0.6404258779117039
+          },
+          {
+            "bin": 10,
+            "count": 4,
+            "accuracy": 1.0,
+            "mean_confidence": 0.6963506639003754
+          },
+          {
+            "bin": 11,
+            "count": 11,
+            "accuracy": 0.9090909090909091,
+            "mean_confidence": 0.7664791941642761
+          },
+          {
+            "bin": 12,
+            "count": 21,
+            "accuracy": 0.9523809523809523,
+            "mean_confidence": 0.8310647464933849
+          },
+          {
+            "bin": 13,
+            "count": 22,
+            "accuracy": 0.9545454545454546,
+            "mean_confidence": 0.9173552746122534
+          },
+          {
+            "bin": 14,
+            "count": 24298,
+            "accuracy": 0.9998353773973166,
+            "mean_confidence": 0.999802232897422
+          }
+        ],
+        "nll_probability_floor": 1.1754943508222875e-38
+      },
+      "upstream_action_metrics": {
+        "examples": 24370,
+        "accuracy": 0.999548625359048,
+        "coverage": 0.4788264259335248,
+        "abstention_rate": 0.5211735740664751,
+        "selective_accuracy": 0.9990573313908647,
+        "wrong_action_rate": 0.00045137464095199015,
+        "wrong_target_rate": 0.0,
+        "unsafe_action_rate": 0.00045137464095199015,
+        "counts": {
+          "total": 24370,
+          "correct": 24359,
+          "abstained": 12701,
+          "acted": 11669,
+          "acted_correct": 11658,
+          "wrong_action": 11,
+          "unsafe_action": 11
+        },
+        "per_action": {
+          "abstain": {
+            "accuracy": 0.9991346758967904,
+            "abstention_rate": 0.9991346758967904,
+            "examples": 12712
+          },
+          "check": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 816
+          },
+          "click": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 1040
+          },
+          "fill": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 9802
+          }
+        }
+      },
+      "macro_action_accuracy": 0.9997836689741976,
+      "wrong_rows": [
+        137,
+        3216,
+        6631,
+        6748,
+        7280,
+        10703,
+        12255,
+        13480,
+        15214,
+        21022,
+        23671
+      ]
+    },
+    "baseline": {
+      "correct": 24359,
+      "rows": 24370,
+      "accuracy": 0.999548625359048,
+      "model_call_latency_ms": {
+        "count": 24370,
+        "median_ms": 0.9897080017253757,
+        "p95_ms": 1.1015420022886246,
+        "min_ms": 0.884292006958276,
+        "max_ms": 3.03837499814108
+      },
+      "load_ms": 640.2017919754144,
+      "first_call_ms": 2.522958006011322,
+      "calibration": {
+        "nll": 0.002108235388127147,
+        "ece": 0.000331678613561794,
+        "equal_width_bins": 15,
+        "bins": [
+          {
+            "bin": 7,
+            "count": 2,
+            "accuracy": 0.5,
+            "mean_confidence": 0.5224609375
+          },
+          {
+            "bin": 8,
+            "count": 5,
+            "accuracy": 0.8,
+            "mean_confidence": 0.57607421875
+          },
+          {
+            "bin": 9,
+            "count": 6,
+            "accuracy": 0.8333333333333334,
+            "mean_confidence": 0.637451171875
+          },
+          {
+            "bin": 10,
+            "count": 5,
+            "accuracy": 0.8,
+            "mean_confidence": 0.69248046875
+          },
+          {
+            "bin": 11,
+            "count": 12,
+            "accuracy": 0.9166666666666666,
+            "mean_confidence": 0.7687581380208334
+          },
+          {
+            "bin": 12,
+            "count": 20,
+            "accuracy": 0.95,
+            "mean_confidence": 0.8322265625
+          },
+          {
+            "bin": 13,
+            "count": 22,
+            "accuracy": 0.9545454545454546,
+            "mean_confidence": 0.9176136363636364
+          },
+          {
+            "bin": 14,
+            "count": 24298,
+            "accuracy": 0.9998353773973166,
+            "mean_confidence": 0.9998245660008025
+          }
+        ],
+        "nll_probability_floor": 1.1754943508222875e-38
+      },
+      "upstream_action_metrics": {
+        "examples": 24370,
+        "accuracy": 0.999548625359048,
+        "coverage": 0.4788264259335248,
+        "abstention_rate": 0.5211735740664751,
+        "selective_accuracy": 0.9990573313908647,
+        "wrong_action_rate": 0.00045137464095199015,
+        "wrong_target_rate": 0.0,
+        "unsafe_action_rate": 0.00045137464095199015,
+        "counts": {
+          "total": 24370,
+          "correct": 24359,
+          "abstained": 12701,
+          "acted": 11669,
+          "acted_correct": 11658,
+          "wrong_action": 11,
+          "unsafe_action": 11
+        },
+        "per_action": {
+          "abstain": {
+            "accuracy": 0.9991346758967904,
+            "abstention_rate": 0.9991346758967904,
+            "examples": 12712
+          },
+          "check": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 816
+          },
+          "click": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 1040
+          },
+          "fill": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 9802
+          }
+        }
+      },
+      "macro_action_accuracy": 0.9997836689741976,
+      "wrong_rows": [
+        137,
+        3216,
+        6631,
+        6748,
+        7280,
+        10703,
+        12255,
+        13480,
+        15214,
+        21022,
+        23671
+      ],
+      "paired_with_upstream": {
+        "rows": 24370,
+        "argmax_agreement": 24370,
+        "agreement_rate": 1.0,
+        "reference_correct_candidate_wrong": 0,
+        "reference_wrong_candidate_correct": 0,
+        "both_wrong": 11,
+        "disagreement_rows": []
+      },
+      "max_abs_probability_error": 0.020487427711486816,
+      "mean_row_max_abs_probability_error": 4.673297645567538e-05,
+      "probability_tolerance_violation_rows": [
+        137,
+        884,
+        2945,
+        3216,
+        8706,
+        9193,
+        12725,
+        13480,
+        13768,
+        15698,
+        23992
+      ],
+      "conversion_parity_passed": false,
+      "output_validation_failures": {
+        "19270": [
+          "Live option probabilities do not sum to one"
+        ]
+      }
+    },
+    "int8-weights": {
+      "correct": 24359,
+      "rows": 24370,
+      "accuracy": 0.999548625359048,
+      "model_call_latency_ms": {
+        "count": 24370,
+        "median_ms": 0.9902919991873205,
+        "p95_ms": 1.103792006324511,
+        "min_ms": 0.8826660050544888,
+        "max_ms": 49.30691700428724
+      },
+      "load_ms": 615.115666994825,
+      "first_call_ms": 1.328666985500604,
+      "calibration": {
+        "nll": 0.002145651629876076,
+        "ece": 0.00048729947809817673,
+        "equal_width_bins": 15,
+        "bins": [
+          {
+            "bin": 7,
+            "count": 4,
+            "accuracy": 0.75,
+            "mean_confidence": 0.520751953125
+          },
+          {
+            "bin": 8,
+            "count": 4,
+            "accuracy": 1.0,
+            "mean_confidence": 0.576904296875
+          },
+          {
+            "bin": 9,
+            "count": 6,
+            "accuracy": 0.8333333333333334,
+            "mean_confidence": 0.6383463541666666
+          },
+          {
+            "bin": 10,
+            "count": 3,
+            "accuracy": 0.3333333333333333,
+            "mean_confidence": 0.7058919270833334
+          },
+          {
+            "bin": 11,
+            "count": 15,
+            "accuracy": 0.9333333333333333,
+            "mean_confidence": 0.7711263020833333
+          },
+          {
+            "bin": 12,
+            "count": 15,
+            "accuracy": 0.9333333333333333,
+            "mean_confidence": 0.8310546875
+          },
+          {
+            "bin": 13,
+            "count": 26,
+            "accuracy": 1.0,
+            "mean_confidence": 0.9116023137019231
+          },
+          {
+            "bin": 14,
+            "count": 24297,
+            "accuracy": 0.9997942132773593,
+            "mean_confidence": 0.9998235539624234
+          }
+        ],
+        "nll_probability_floor": 1.1754943508222875e-38
+      },
+      "upstream_action_metrics": {
+        "examples": 24370,
+        "accuracy": 0.999548625359048,
+        "coverage": 0.4788264259335248,
+        "abstention_rate": 0.5211735740664751,
+        "selective_accuracy": 0.9990573313908647,
+        "wrong_action_rate": 0.00045137464095199015,
+        "wrong_target_rate": 0.0,
+        "unsafe_action_rate": 0.00045137464095199015,
+        "counts": {
+          "total": 24370,
+          "correct": 24359,
+          "abstained": 12701,
+          "acted": 11669,
+          "acted_correct": 11658,
+          "wrong_action": 11,
+          "unsafe_action": 11
+        },
+        "per_action": {
+          "abstain": {
+            "accuracy": 0.9991346758967904,
+            "abstention_rate": 0.9991346758967904,
+            "examples": 12712
+          },
+          "check": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 816
+          },
+          "click": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 1040
+          },
+          "fill": {
+            "accuracy": 1.0,
+            "abstention_rate": 0.0,
+            "examples": 9802
+          }
+        }
+      },
+      "macro_action_accuracy": 0.9997836689741976,
+      "wrong_rows": [
+        137,
+        3216,
+        6631,
+        6748,
+        7280,
+        10703,
+        12255,
+        13480,
+        15214,
+        21022,
+        23671
+      ],
+      "paired_with_upstream": {
+        "rows": 24370,
+        "argmax_agreement": 24370,
+        "agreement_rate": 1.0,
+        "reference_correct_candidate_wrong": 0,
+        "reference_wrong_candidate_correct": 0,
+        "both_wrong": 11,
+        "disagreement_rows": []
+      },
+      "max_abs_probability_error": 0.0677376389503479,
+      "mean_row_max_abs_probability_error": 8.935863356371478e-05,
+      "probability_tolerance_violation_rows": [
+        137,
+        410,
+        1225,
+        2076,
+        2093,
+        2818,
+        2945,
+        3064,
+        3151,
+        3216,
+        3298,
+        3479,
+        3756,
+        3978,
+        4469,
+        5294,
+        5759,
+        6426,
+        6466,
+        7177,
+        7515,
+        8243,
+        8706,
+        8714,
+        8891,
+        9193,
+        9253,
+        9977,
+        9981,
+        10703,
+        11122,
+        11257,
+        11880,
+        12725,
+        12754,
+        13370,
+        13423,
+        13693,
+        13768,
+        14656,
+        15132,
+        15214,
+        15252,
+        15698,
+        16253,
+        16920,
+        16951,
+        17304,
+        18339,
+        18549,
+        18802,
+        19308,
+        19669,
+        19957,
+        20462,
+        20476,
+        21844,
+        21871,
+        22336,
+        22714,
+        22791,
+        23276,
+        23670,
+        23992
+      ],
+      "conversion_parity_passed": false,
+      "output_validation_failures": {}
+    }
+  },
+  "failures": [
+    {
+      "row": 137,
+      "context": "TASK fill the form from the document, then submit\nFORM Lakeside Vet - Pet Owner Registration - Microsoft Edge\nELEMENT Edit \"Full name\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Last: Novak",
+        "baseline": "fill Last: Novak",
+        "int8-weights": "fill Last: Novak"
+      }
+    },
+    {
+      "row": 410,
+      "context": "TASK fill the form from the document, then submit\nFORM Form\nELEMENT Edit \"Date of loss\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 884,
+      "context": "TASK fill the form from the document, then submit\nFORM Acme Robotics - Job Application\nELEMENT Edit \"Health plan\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 1225,
+      "context": "TASK fill the form from the document, then submit\nFORM City Library - Membership Sign-up\nELEMENT Edit \"Vehicle identification number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 2076,
+      "context": "TASK fill the form from the document, then submit\nFORM Page 2 of 3\nELEMENT Edit \"Graduation year\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 2093,
+      "context": "TASK fill the form from the document, then submit\nFORM Untitled\nELEMENT Edit \"Title\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 2818,
+      "context": "TASK fill the form from the document, then submit\nFORM Acme Robotics - Job Application\nELEMENT Edit \"Your email\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 2945,
+      "context": "TASK fill the form from the document, then submit\nFORM Metro Utilities - New Service Request\nELEMENT Edit \"Organization\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 3064,
+      "context": "TASK fill the form from the document, then submit\nFORM Riverside Dental - Patient Intake\nELEMENT Edit \"Make\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 3151,
+      "context": "TASK fill the form from the document, then submit\nFORM Evergreen Rentals - Tenant Application - Google Chrome\nELEMENT Edit \"Date of loss\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 3216,
+      "context": "TASK fill the form from the document, then submit\nFORM Northwind Clinic - New Patient Registration - Google Chrome\nELEMENT Edit \"Email\" value=\"oivanova66@yahoo.com\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Email address: oivanova66@yahoo.com",
+        "baseline": "fill Email address: oivanova66@yahoo.com",
+        "int8-weights": "fill Email address: oivanova66@yahoo.com"
+      }
+    },
+    {
+      "row": 3298,
+      "context": "TASK fill the form from the document, then submit\nFORM City Library - Membership Sign-up - Mozilla Firefox\nELEMENT Edit \"DL number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 3479,
+      "context": "TASK fill the form from the document, then submit\nFORM Form\nELEMENT Edit \"Insurance company\" value=\"Centene\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 3756,
+      "context": "TASK fill the form from the document, then submit\nFORM Page 2 of 3\nELEMENT Edit \"Claim #\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 3978,
+      "context": "TASK fill the form from the document, then submit\nFORM Cedar Pharmacy - Prescription Transfer - Microsoft Edge\nELEMENT Edit \"Family doctor\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 4469,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment\nELEMENT Edit \"City\" value=\"Portland\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 5294,
+      "context": "TASK fill the form from the document, then submit\nFORM Untitled\nELEMENT Edit \"Email\" value=\"nadia.fischer@fastmail.com\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 5759,
+      "context": "TASK fill the form from the document, then submit\nFORM Pinewood Camp - Parent Consent Form\nELEMENT Edit \"Relationship to patient\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 6426,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment - Microsoft Edge\nELEMENT CheckBox \"Consent to electronic communication (required)\" unchecked",
+      "gold_option": "check",
+      "selected_options": {
+        "upstream_pytorch": "check",
+        "baseline": "check",
+        "int8-weights": "check"
+      }
+    },
+    {
+      "row": 6466,
+      "context": "TASK fill the form from the document, then submit\nFORM Lakeside Vet - Pet Owner Registration - JevBrowser\nELEMENT Edit \"Details of loss\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 6631,
+      "context": "TASK fill the form from the document, then submit\nFORM Blue Sun Freight - Vendor Onboarding - Google Chrome\nELEMENT Edit \"Given name\" value=\"Priya\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Physician: Dr. Fischer",
+        "baseline": "fill Physician: Dr. Fischer",
+        "int8-weights": "fill Physician: Dr. Fischer"
+      }
+    },
+    {
+      "row": 6748,
+      "context": "TASK fill the form from the document, then submit\nFORM JevBrowser\nELEMENT Edit \"State / Province\" value=\"NE\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill ST: NE",
+        "baseline": "fill ST: NE",
+        "int8-weights": "fill ST: NE"
+      }
+    },
+    {
+      "row": 7177,
+      "context": "TASK fill the form from the document, then submit\nFORM Form\nELEMENT Edit \"Claim #\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 7280,
+      "context": "TASK fill the form from the document, then submit\nFORM \nELEMENT Edit \"Given name\" value=\"Tomas\" hint=\"First\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill First name: Tomas",
+        "baseline": "fill First name: Tomas",
+        "int8-weights": "fill First name: Tomas"
+      }
+    },
+    {
+      "row": 7515,
+      "context": "TASK fill the form from the document, then submit\nFORM Acme Robotics - Job Application\nELEMENT Edit \"Province\" value=\"GA\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 8243,
+      "context": "TASK fill the form from the document, then submit\nFORM \nELEMENT Edit \"Passport number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 8706,
+      "context": "TASK fill the form from the document, then submit\nFORM Northwind Clinic - New Patient Registration - Mozilla Firefox\nELEMENT CheckBox \"I authorize verification of the information provided\" unchecked",
+      "gold_option": "check",
+      "selected_options": {
+        "upstream_pytorch": "check",
+        "baseline": "check",
+        "int8-weights": "check"
+      }
+    },
+    {
+      "row": 8714,
+      "context": "TASK fill the form from the document, then submit\nFORM Evergreen Rentals - Tenant Application\nELEMENT Edit \"Registration number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 8891,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment - Google Chrome\nELEMENT Edit \"Last\" value=\"Ivanova\" hint=\"Family name\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 9193,
+      "context": "TASK fill the form from the document, then submit\nFORM Blue Sun Freight - Vendor Onboarding\nELEMENT Edit \"DL number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 9253,
+      "context": "TASK fill the form from the document, then submit\nFORM Form\nELEMENT Edit \"Homepage\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 9977,
+      "context": "TASK fill the form from the document, then submit\nFORM \nELEMENT Edit \"Country\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 9981,
+      "context": "TASK fill the form from the document, then submit\nFORM \nELEMENT Edit \"City\" value=\"Burlington\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 10703,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment - JevBrowser\nELEMENT Edit \"Last\" value=\"Silva\" hint=\"Family name\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Family name: Silva",
+        "baseline": "fill Family name: Silva",
+        "int8-weights": "fill Family name: Silva"
+      }
+    },
+    {
+      "row": 11122,
+      "context": "TASK fill the form from the document, then submit\nFORM JevBrowser\nELEMENT Edit \"EIN\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 11257,
+      "context": "TASK fill the form from the document, then submit\nFORM Harbor University - Graduate Application - Microsoft Edge\nELEMENT CheckBox \"I am legally authorized to work in this country\" unchecked",
+      "gold_option": "check",
+      "selected_options": {
+        "upstream_pytorch": "check",
+        "baseline": "check",
+        "int8-weights": "check"
+      }
+    },
+    {
+      "row": 11880,
+      "context": "TASK fill the form from the document, then submit\nFORM Blue Sun Freight - Vendor Onboarding - Google Chrome\nELEMENT Edit \"Date of loss\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 12255,
+      "context": "TASK fill the form from the document, then submit\nFORM Globex Insurance - Auto Claim Form - JevBrowser\nELEMENT Edit \"Taxpayer identification number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill National ID: 813-20-3361",
+        "baseline": "fill National ID: 813-20-3361",
+        "int8-weights": "fill National ID: 813-20-3361"
+      }
+    },
+    {
+      "row": 12725,
+      "context": "TASK fill the form from the document, then submit\nFORM JevBrowser\nELEMENT Edit \"Company\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 12754,
+      "context": "TASK fill the form from the document, then submit\nFORM Northwind Clinic - New Patient Registration - Microsoft Edge\nELEMENT CheckBox \"I consent to treatment and agree to the privacy policy\" checked",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 13370,
+      "context": "TASK fill the form from the document, then submit\nFORM Evergreen Rentals - Tenant Application - Microsoft Edge\nELEMENT Edit \"Mailing address\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 13423,
+      "context": "TASK fill the form from the document, then submit\nFORM Skyline Travel - Visa Assistance Form\nELEMENT Edit \"Current job title\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 13480,
+      "context": "TASK fill the form from the document, then submit\nFORM Evergreen Rentals - Tenant Application\nELEMENT Edit \"Current employer\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Company reg: REG3878384",
+        "baseline": "fill Company reg: REG3878384",
+        "int8-weights": "fill Company reg: REG3878384"
+      }
+    },
+    {
+      "row": 13693,
+      "context": "TASK fill the form from the document, then submit\nFORM Lakeside Vet - Pet Owner Registration - Mozilla Firefox\nELEMENT Edit \"Organization\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 13768,
+      "context": "TASK fill the form from the document, then submit\nFORM Northwind Clinic - New Patient Registration\nELEMENT Edit \"Legal name\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 14656,
+      "context": "TASK fill the form from the document, then submit\nFORM Summit Bank - Direct Deposit Enrollment - Google Chrome\nELEMENT Edit \"Details of loss\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 15132,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment - Mozilla Firefox\nELEMENT Edit \"Town\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 15214,
+      "context": "TASK fill the form from the document, then submit\nFORM Riverside Dental - Patient Intake\nELEMENT Edit \"Relation to applicant\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Reg plate: WN9-8147",
+        "baseline": "fill Reg plate: WN9-8147",
+        "int8-weights": "fill Reg plate: WN9-8147"
+      }
+    },
+    {
+      "row": 15252,
+      "context": "TASK fill the form from the document, then submit\nFORM Northwind Clinic - New Patient Registration\nELEMENT Edit \"VIN number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 15698,
+      "context": "TASK fill the form from the document, then submit\nFORM Orbit Telecom - Account Transfer - Mozilla Firefox\nELEMENT CheckBox \"I am legally authorized to work in this country\" checked",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 16253,
+      "context": "TASK fill the form from the document, then submit\nFORM Harbor University - Graduate Application\nELEMENT Edit \"Relation to applicant\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 16920,
+      "context": "TASK fill the form from the document, then submit\nFORM Acme Robotics - Job Application - Google Chrome\nELEMENT Edit \"Total years experience\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 16951,
+      "context": "TASK fill the form from the document, then submit\nFORM Form\nELEMENT Edit \"State\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 17304,
+      "context": "TASK fill the form from the document, then submit\nFORM Metro Utilities - New Service Request - Google Chrome\nELEMENT CheckBox \"I am legally authorized to work in this country\" checked",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 18339,
+      "context": "TASK fill the form from the document, then submit\nFORM Summit Bank - Direct Deposit Enrollment - Mozilla Firefox\nELEMENT Edit \"Last name\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 18549,
+      "context": "TASK fill the form from the document, then submit\nFORM Lakeside Vet - Pet Owner Registration - Microsoft Edge\nELEMENT Edit \"ABA routing number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 18802,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment\nELEMENT Edit \"Unit number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 19270,
+      "context": "TASK fill the form from the document, then submit\nFORM JevBrowser\nELEMENT Edit \"Plate number\" value=\"old\"",
+      "gold_option": "fill License plate: SJ3-1115",
+      "selected_options": {
+        "upstream_pytorch": "fill License plate: SJ3-1115",
+        "baseline": "fill License plate: SJ3-1115",
+        "int8-weights": "fill License plate: SJ3-1115"
+      }
+    },
+    {
+      "row": 19308,
+      "context": "TASK fill the form from the document, then submit\nFORM City Library - Membership Sign-up - Google Chrome\nELEMENT Edit \"Last\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 19669,
+      "context": "TASK fill the form from the document, then submit\nFORM Acme Robotics - Job Application\nELEMENT Edit \"School\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 19957,
+      "context": "TASK fill the form from the document, then submit\nFORM Skyline Travel - Visa Assistance Form\nELEMENT Edit \"Plate number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 20462,
+      "context": "TASK fill the form from the document, then submit\nFORM Harbor University - Graduate Application - Mozilla Firefox\nELEMENT CheckBox \"I am legally authorized to work in this country\" checked",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 20476,
+      "context": "TASK fill the form from the document, then submit\nFORM JevBrowser\nELEMENT Edit \"Sort code\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 21022,
+      "context": "TASK fill the form from the document, then submit\nFORM Harbor University - Graduate Application - JevBrowser\nELEMENT Edit \"State\" value=\"NE\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill National ID: 794-03-1212",
+        "baseline": "fill National ID: 794-03-1212",
+        "int8-weights": "fill National ID: 794-03-1212"
+      }
+    },
+    {
+      "row": 21844,
+      "context": "TASK fill the form from the document, then submit\nFORM Cedar Pharmacy - Prescription Transfer - Mozilla Firefox\nELEMENT Edit \"Gender\" value=\"Female\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 21871,
+      "context": "TASK fill the form from the document, then submit\nFORM Riverside Dental - Patient Intake\nELEMENT Edit \"Sort code\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 22336,
+      "context": "TASK fill the form from the document, then submit\nFORM City Library - Membership Sign-up\nELEMENT Edit \"Car model\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 22714,
+      "context": "TASK fill the form from the document, then submit\nFORM Metro Utilities - New Service Request\nELEMENT Edit \"Describe your concern\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 22791,
+      "context": "TASK fill the form from the document, then submit\nFORM Riverside Dental - Patient Intake - Mozilla Firefox\nELEMENT CheckBox \"Consent to electronic communication (required)\" checked",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 23276,
+      "context": "TASK fill the form from the document, then submit\nFORM Pinewood Camp - Parent Consent Form - JevBrowser\nELEMENT Edit \"Relationship status\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 23670,
+      "context": "TASK fill the form from the document, then submit\nFORM \nELEMENT Edit \"Member number\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    },
+    {
+      "row": 23671,
+      "context": "TASK fill the form from the document, then submit\nFORM \nELEMENT Edit \"Email\" value=\"lchen6@yahoo.com\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "fill Email: lchen6@yahoo.com",
+        "baseline": "fill Email: lchen6@yahoo.com",
+        "int8-weights": "fill Email: lchen6@yahoo.com"
+      }
+    },
+    {
+      "row": 23992,
+      "context": "TASK fill the form from the document, then submit\nFORM Apex Gym - Member Enrollment - Microsoft Edge\nELEMENT Edit \"Total years experience\" value=\"\"",
+      "gold_option": "skip",
+      "selected_options": {
+        "upstream_pytorch": "skip",
+        "baseline": "skip",
+        "int8-weights": "skip"
+      }
+    }
+  ],
+  "trace": {
+    "file": "int8-synthetic-test-decisions.jsonl.gz",
+    "sha256": "69720b9e11430ecff7ea0718f14bf16792de6f39f114faba250cfe855e4e4156",
+    "rows": 24370
+  },
+  "upstream_evaluator_sha256": "49dccdeb4e2b456187cdc4491a6d78fabac6a45384272f8e3ec802a58872aa6e",
+  "harness_sha256": {
+    "benchmark-synthetic.py": "891da8408156e2b80849b4d799c77c9a31c091dfd999fcd044639deefac3e888",
+    "synthetic_test.py": "e2b68725236e6827eb422480c8cc8aa189b2490c1594650d8f2928771d1bc626",
+    "score-report.py": "c78e45caa86fb59f9a56ef18cc6a35020ea6e281a6a23d23618f56da85a0d95e"
+  },
+  "conversion_parity_passed": false,
+  "limitations": [
+    "The release file has 24370 rows; the model card claims approximately 15000. This is not a reconstruction of that unspecified manifest.",
+    "Upstream describes form-disjoint synthetic data; this is not an unseen real-world GUI or task-completion benchmark.",
+    "No fresh hosted Jev calls are included. Published hosted-Jev percentages are not a matched baseline for this run.",
+    "Local timings use CPU for PyTorch and CPU+ANE for Core ML; they are backend/deployment measurements, not equal-hardware algorithm comparisons."
+  ]
+}