Add experimental INT4 weights and full matched benchmark

Packed INT4 weights with FP16 compute: 451,595 bytes vs 1,511,080-byte matched FP16 control; iOS18/macOS15 required. Full 24,370-row test: 24,353 correct (99.9302%) vs 24,359 FP16. Fourteen choices change: 10 harmful flips and four corrections. Both medians 0.982 ms. Numerical parity fails: 356 rows above 0.005, max 0.754359; no INT4 sum failures. Include exact portable models, manifests, complete trace, demo and ANE reports. Original FP16, ANE-gather and INT8 artifacts unchanged; FP16 remains default.

Files changed (16) hide show

README.md +35 -0
checksums.json +15 -1
int4-source-fp16/conversion.json +73 -0
int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel +3 -0
int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin +3 -0
int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Manifest.json +18 -0
int4-weights/conversion.json +124 -0
int4-weights/cua_s1_forms_int4_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel +3 -0
int4-weights/cua_s1_forms_int4_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin +3 -0
int4-weights/cua_s1_forms_int4_options32.mlpackage/Manifest.json +18 -0
reports/int4-demo-verification.json +1799 -0
reports/int4-fallback.json +112 -0
reports/int4-profile.json +1105 -0
reports/int4-source-demo-verification.json +1748 -0
reports/int4-synthetic-test-decisions.jsonl.gz +3 -0
reports/int4-synthetic-test.json +0 -0

README.md CHANGED Viewed

@@ -308,6 +308,41 @@ compression does not establish an INT8 activation path or a speed/energy gain.
 No calibration, training, or test-based tuning was performed. Numerical parity
 remains failed; the default FP16 artifacts are unchanged.
 ## Application responsibilities
 The application must extract document entities, describe UI elements, build

 No calibration, training, or test-based tuning was performed. Numerical parity
 remains failed; the default FP16 artifacts are unchanged.
+## INT4 weight trial
+A matched run over all **24,370 synthetic decisions** on M5 Pro, CPU+ANE:
+| Export | Package size | Accuracy | Median | p95 |
+| --- | ---: | ---: | ---: | ---: |
+| FP16 control, iOS 18 target | 1.51 MB | 99.9549% | 0.982 ms | 1.081 ms |
+| INT4 weights, FP16 compute | 0.45 MB | 99.9302% | 0.982 ms | 1.082 ms |
+**70.1% smaller**, with essentially unchanged latency. INT4 makes **17 errors
+versus 11** for FP16: 14 choices change, introducing 10 errors and correcting four.
+Numerical parity fails: 356 rows exceed the 0.005 probability-error limit
+(maximum 0.754359); no INT4 probability-sum violations were observed.
+Packed INT4 requires **iOS 18/macOS 15**. Both exports use the same decomposed
+attention graph and FP16 computation. Batch-1 timing excludes encoding/loading/UI.
+INT8 preserves all choices at 0.81 MB; the original FP16 remains the default.
+The experimental INT4 package is
+`int4-weights/cua_s1_forms_int4_options32.mlpackage`. Its matched FP16 control is
+`int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage`. Nineteen weight tensors
+use per-channel symmetric INT4 with a 2,048-element threshold; no calibration or
+retraining. This is weight compression with FP16 computation. The graph is
+retargeted from the verified original to preserve decomposed attention.
+[Full report](reports/int4-synthetic-test.json) ·
+[Complete per-row trace](reports/int4-synthetic-test-decisions.jsonl.gz) ·
+[Quantization manifest](int4-weights/conversion.json) ·
+[FP16 control manifest](int4-source-fp16/conversion.json) ·
+[Reproduction](https://github.com/FluidInference/mobius/tree/codex/cua-s1-forms/models/computer-use/cua-s1-forms/coreml#int4-weight-trial)
+The demo retains 196/196 choices but fails numerical parity. The full benchmark
+above exposes the accuracy loss. Public compute-plan placement is 149 ANE / 32 CPU
+operations, plus 19 unassigned constant dequantizations; no speed/energy claim.
 ## Application responsibilities
 The application must extract document entities, describe UI elements, build

checksums.json CHANGED Viewed

@@ -1,7 +1,7 @@
 {
   "LICENSE": "c0779290c1d4783169aa3dbfb55feb505e563ef8a004bbf55298ceffcfbda8d9",
   "NOTICES.md": "027c72741eaa695e60d7b6cebd3666372c81d443a96b673b72a897cf432ddfd0",
-  "README.md": "ba32d1ab1fc14a855bd30bc07a8e8474f6b095f17c8126f6a9fc6a810b6ab31f",
   "UPSTREAM-THIRD-PARTY-NOTICES.md": "4091e69b45c8cc97e30a066fbbd56148dbef66ab716432c048d2333c9c464213",
   "ane-gather/conversion.json": "4d569a4020f1e9711da3f7fff0d11acef1812c4652414cf64728c77bc2bae4c4",
   "ane-gather/cua_s1_forms_fp16_options32.mlmodelc/analytics/coremldata.bin": "526dccb87bdc036df1e8add0b50dd6540db534808fc2f8caa97cd897b4fc1fa3",
@@ -23,6 +23,14 @@
   "demo/browser-demo.gif": "6cf597927e7bcb5617719453eecb86c7c06ab9a91fcaf46de272b42abe12d4fc",
   "demo/browser-demo.mp4": "5fbb3925c4824d15e6c24c504db91a96d318f4e751a20a070fdedf64e587d1a0",
   "demo/browser-preview.png": "e624ec2a797d0b3b050e66055d6a9fb09a1f6a9ace67b8eb071ce5f2f9e282b1",
   "int8-weights/conversion.json": "b8a51ceeb65fe8020b2a186fae2ebf586443956111acfbac1a3f208a062c1c80",
   "int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
   "int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
@@ -35,6 +43,12 @@
   "reports/ane-gather-verification.json": "fa51fe0537dcd9cc8d1d75fc0be4747d887aa2e3af199ec08edd8f9400ce3318",
   "reports/ane-profile.json": "8f654717397a294acc793861676a1480c160bb3f7ab5be6ff67ebb6a4473b832",
   "reports/browser-validation.json": "299641c7b9f6cc2464cfcd984b4ef4445385e4710bc15abae7cee8ba262331b6",
   "reports/int8-demo-verification.json": "5602c8c254c0ae4c696e771291b25d1196ccc0d63f48eaaccdde0245803a5368",
   "reports/int8-fallback.json": "9bb86a529a8e7e353efa12449a5ea930817dc7b1afdacec6ae24f3c318bc5835",
   "reports/int8-profile.json": "0b962528dc7e5bb1d53abeb4b549f93b93f549a1e1935c3d77ef2606e83bf6a9",

 {
   "LICENSE": "c0779290c1d4783169aa3dbfb55feb505e563ef8a004bbf55298ceffcfbda8d9",
   "NOTICES.md": "027c72741eaa695e60d7b6cebd3666372c81d443a96b673b72a897cf432ddfd0",
+  "README.md": "e629edbefc0592b8e978dae6ff0905ea4b98fafd956c6668f3cd43aaaa21eff8",
   "UPSTREAM-THIRD-PARTY-NOTICES.md": "4091e69b45c8cc97e30a066fbbd56148dbef66ab716432c048d2333c9c464213",
   "ane-gather/conversion.json": "4d569a4020f1e9711da3f7fff0d11acef1812c4652414cf64728c77bc2bae4c4",
   "ane-gather/cua_s1_forms_fp16_options32.mlmodelc/analytics/coremldata.bin": "526dccb87bdc036df1e8add0b50dd6540db534808fc2f8caa97cd897b4fc1fa3",
   "demo/browser-demo.gif": "6cf597927e7bcb5617719453eecb86c7c06ab9a91fcaf46de272b42abe12d4fc",
   "demo/browser-demo.mp4": "5fbb3925c4824d15e6c24c504db91a96d318f4e751a20a070fdedf64e587d1a0",
   "demo/browser-preview.png": "e624ec2a797d0b3b050e66055d6a9fb09a1f6a9ace67b8eb071ce5f2f9e282b1",
+  "int4-source-fp16/conversion.json": "01105dc8bef7a30fc3430e150caa2a143dc0ba23a295b6280de3b8ff3b40af55",
+  "int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel": "6ab2df320df7081480b76876df0f4bfd2c841592d93f354c568ba504f7269627",
+  "int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+  "int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Manifest.json": "e5b1c7dc66b2014b3985c180a75548a709788c6e314d9d2ae14e4e6d441c3c68",
+  "int4-weights/conversion.json": "4675febbdef9cc15d46bf8aed17d7c7aaa0e91b3168a525c31b81ad2cfdb6875",
+  "int4-weights/cua_s1_forms_int4_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel": "cadad6bfa6875383a7ee3de68d0d22871148aa21cf117e7fe44c2c971287c4e5",
+  "int4-weights/cua_s1_forms_int4_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin": "0f766fcb9dbd612ed4f6c7a40117a5993a4789b3ae94afba0df37193c831f63c",
+  "int4-weights/cua_s1_forms_int4_options32.mlpackage/Manifest.json": "fb91c8b5ec3168111567de663bb3140d2790c6e7c48f32e0df3f5135af0673af",
   "int8-weights/conversion.json": "b8a51ceeb65fe8020b2a186fae2ebf586443956111acfbac1a3f208a062c1c80",
   "int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel": "a7d9747308b8b754a46f3e70c8465b821f90e12376b3a71dc909f7364ae73c3a",
   "int8-weights/cua_s1_forms_int8_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin": "f600c4434b7f7e3e789de8f1a4aa54a55f84df51fa9b788e18f5c97b68e4840f",
   "reports/ane-gather-verification.json": "fa51fe0537dcd9cc8d1d75fc0be4747d887aa2e3af199ec08edd8f9400ce3318",
   "reports/ane-profile.json": "8f654717397a294acc793861676a1480c160bb3f7ab5be6ff67ebb6a4473b832",
   "reports/browser-validation.json": "299641c7b9f6cc2464cfcd984b4ef4445385e4710bc15abae7cee8ba262331b6",
+  "reports/int4-demo-verification.json": "7465aecddc5c6806a0c1882db3a587eb7ea0f8de7978f60bcbc7e39ffe2cfa32",
+  "reports/int4-fallback.json": "7e14ab1bb6b8c2ede589773af3d39aea1a7349d199a27d597f1cd47a40217b07",
+  "reports/int4-profile.json": "71f6b3ae5c9340a77936e368529201acef0d1e3b987b7436b1d35a829b6e350e",
+  "reports/int4-source-demo-verification.json": "f0c026f462dde1a8f86851d55ec35ab33ab8e68e5af3e4f1eb4b9c52f3dd08b1",
+  "reports/int4-synthetic-test-decisions.jsonl.gz": "6922913041f95ac8872958b92312cb5b869b9ce88116c4684b36ea7753284674",
+  "reports/int4-synthetic-test.json": "9d6a691aeb2ddc352ab7c12327472c92814991c201b67c7da8db39a76c972b32",
   "reports/int8-demo-verification.json": "5602c8c254c0ae4c696e771291b25d1196ccc0d63f48eaaccdde0245803a5368",
   "reports/int8-fallback.json": "9bb86a529a8e7e353efa12449a5ea930817dc7b1afdacec6ae24f3c318bc5835",
   "reports/int8-profile.json": "0b962528dc7e5bb1d53abeb4b549f93b93f549a1e1935c3d77ef2606e83bf6a9",

int4-source-fp16/conversion.json ADDED Viewed

	@@ -0,0 +1,73 @@

+{
+  "model": "cua_s1_forms_fp16_options32.mlpackage",
+  "precision": "float16",
+  "minimum_target": "iOS18/macOS15",
+  "limits": {
+    "context_bytes": 224,
+    "option_bytes": 96,
+    "max_options": 32
+  },
+  "model_config": {
+    "context_tokens": 224,
+    "encoder": "tinyx",
+    "heads": 4,
+    "hf_model": "Qwen/Qwen2.5-0.5B",
+    "layers": 2,
+    "option_tokens": 96,
+    "rank": 128,
+    "width": 128
+  },
+  "parameters": 706048,
+  "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "trace_row": 0,
+  "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "export_seconds": 0.5374167499830946,
+  "python": "3.11.11",
+  "torch": "2.7.0",
+  "coremltools": "9.0",
+  "package_files": {
+    "Data/com.apple.CoreML/model.mlmodel": "6ab2df320df7081480b76876df0f4bfd2c841592d93f354c568ba504f7269627",
+    "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+    "Manifest.json": "e5b1c7dc66b2014b3985c180a75548a709788c6e314d9d2ae14e4e6d441c3c68"
+  },
+  "target_upgrade": {
+    "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+    "source_package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+      "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+      "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+    },
+    "method": "Load original MIL at specification 9; empty pass pipeline; retain decomposed attention",
+    "operation_counts": {
+      "add": 13,
+      "cast": 11,
+      "concat": 2,
+      "const": 239,
+      "equal": 2,
+      "expand_dims": 5,
+      "fill_like": 3,
+      "gather": 2,
+      "greater_equal": 2,
+      "layer_norm": 8,
+      "linear": 15,
+      "logical_not": 2,
+      "matmul": 8,
+      "maximum": 1,
+      "mul": 7,
+      "not_equal": 3,
+      "real_div": 1,
+      "reduce_sum": 3,
+      "relu": 3,
+      "reshape": 31,
+      "select": 7,
+      "slice_by_index": 13,
+      "softmax": 5,
+      "squeeze": 3,
+      "tile": 2,
+      "transpose": 21
+    },
+    "script_sha256": "33db3295450428d90c1de5a804ab47bf3507c686fbbdef67cd276ce116758b5f"
+  }
+}

int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6ab2df320df7081480b76876df0f4bfd2c841592d93f354c568ba504f7269627
+size 63743

int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895
+size 1446720

int4-source-fp16/cua_s1_forms_fp16_options32.mlpackage/Manifest.json ADDED Viewed

	@@ -0,0 +1,18 @@

+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "15C1AFE1-37B2-4975-949D-B817E3C00088": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "B997D799-579D-437C-9F80-DE24FF3EFB11": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "15C1AFE1-37B2-4975-949D-B817E3C00088"
+}

int4-weights/conversion.json ADDED Viewed

	@@ -0,0 +1,124 @@

+{
+  "model": "cua_s1_forms_int4_options32.mlpackage",
+  "precision": "int4_weights_float16_compute",
+  "minimum_target": "iOS18/macOS15",
+  "limits": {
+    "context_bytes": 224,
+    "option_bytes": 96,
+    "max_options": 32
+  },
+  "model_config": {
+    "context_tokens": 224,
+    "encoder": "tinyx",
+    "heads": 4,
+    "hf_model": "Qwen/Qwen2.5-0.5B",
+    "layers": 2,
+    "option_tokens": 96,
+    "rank": 128,
+    "width": 128
+  },
+  "parameters": 706048,
+  "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "trace_row": 0,
+  "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "export_seconds": 0.5374167499830946,
+  "python": "3.11.11",
+  "torch": "2.7.0",
+  "coremltools": "9.0",
+  "package_files": {
+    "Data/com.apple.CoreML/model.mlmodel": "cadad6bfa6875383a7ee3de68d0d22871148aa21cf117e7fe44c2c971287c4e5",
+    "Data/com.apple.CoreML/weights/weight.bin": "0f766fcb9dbd612ed4f6c7a40117a5993a4789b3ae94afba0df37193c831f63c",
+    "Manifest.json": "fb91c8b5ec3168111567de663bb3140d2790c6e7c48f32e0df3f5135af0673af"
+  },
+  "target_upgrade": {
+    "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+    "source_package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+      "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+      "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+    },
+    "method": "Load original MIL at specification 9; empty pass pipeline; retain decomposed attention",
+    "operation_counts": {
+      "add": 13,
+      "cast": 11,
+      "concat": 2,
+      "const": 239,
+      "equal": 2,
+      "expand_dims": 5,
+      "fill_like": 3,
+      "gather": 2,
+      "greater_equal": 2,
+      "layer_norm": 8,
+      "linear": 15,
+      "logical_not": 2,
+      "matmul": 8,
+      "maximum": 1,
+      "mul": 7,
+      "not_equal": 3,
+      "real_div": 1,
+      "reduce_sum": 3,
+      "relu": 3,
+      "reshape": 31,
+      "select": 7,
+      "slice_by_index": 13,
+      "softmax": 5,
+      "squeeze": 3,
+      "tile": 2,
+      "transpose": 21
+    },
+    "script_sha256": "33db3295450428d90c1de5a804ab47bf3507c686fbbdef67cd276ce116758b5f"
+  },
+  "quantization": {
+    "name": "int4-weights",
+    "settings": {
+      "mode": "linear_symmetric",
+      "dtype": "int4",
+      "granularity": "per_channel",
+      "weight_threshold": 2048
+    },
+    "activations": "float16",
+    "calibration": "None; weight-only, data-free quantization",
+    "source_conversion_sha256": "01105dc8bef7a30fc3430e150caa2a143dc0ba23a295b6280de3b8ff3b40af55",
+    "source_package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "6ab2df320df7081480b76876df0f4bfd2c841592d93f354c568ba504f7269627",
+      "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+      "Manifest.json": "e5b1c7dc66b2014b3985c180a75548a709788c6e314d9d2ae14e4e6d441c3c68"
+    },
+    "source_package_bytes": 1511080,
+    "package_bytes": 451595,
+    "compression_seconds": 0.2996572910051327,
+    "packed_int4_tensors": 19,
+    "operation_counts": {
+      "add": 15,
+      "cast": 13,
+      "concat": 2,
+      "const": 226,
+      "constexpr_blockwise_shift_scale": 19,
+      "equal": 2,
+      "expand_dims": 5,
+      "fill_like": 3,
+      "gather": 2,
+      "greater_equal": 4,
+      "layer_norm": 8,
+      "linear": 15,
+      "logical_not": 2,
+      "matmul": 8,
+      "maximum": 1,
+      "mul": 7,
+      "not_equal": 3,
+      "real_div": 1,
+      "reduce_sum": 3,
+      "relu": 3,
+      "reshape": 31,
+      "select": 9,
+      "slice_by_index": 13,
+      "softmax": 5,
+      "squeeze": 3,
+      "tile": 2,
+      "transpose": 21
+    },
+    "script_sha256": "a91636e539d8934253b5e7628ca706346fbee538fa1aa9e3eaf794f440d49dcc"
+  }
+}

int4-weights/cua_s1_forms_int4_options32.mlpackage/Data/com.apple.CoreML/model.mlmodel ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cadad6bfa6875383a7ee3de68d0d22871148aa21cf117e7fe44c2c971287c4e5
+size 67938

int4-weights/cua_s1_forms_int4_options32.mlpackage/Data/com.apple.CoreML/weights/weight.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0f766fcb9dbd612ed4f6c7a40117a5993a4789b3ae94afba0df37193c831f63c
+size 383040

int4-weights/cua_s1_forms_int4_options32.mlpackage/Manifest.json ADDED Viewed

	@@ -0,0 +1,18 @@

+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "D6D01355-3106-40A1-999B-B56148C73691": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        },
+        "F0E58006-D8D0-4891-AD75-BB9E48868A99": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        }
+    },
+    "rootModelIdentifier": "F0E58006-D8D0-4891-AD75-BB9E48868A99"
+}

reports/int4-demo-verification.json ADDED Viewed

	@@ -0,0 +1,1799 @@

+{
+  "purpose": "Local conversion parity on the upstream demo; not a generalization or live GUI benchmark",
+  "created_utc": "2026-09-19T22:25:44.244422+00:00",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "dataset_file": "demo.jsonl",
+  "dataset_sha256": "4f43b442e79ba2e2ce731e27e9b8e340c2b5dfcaffc92d8ff564c34f115ff1ca",
+  "conversion": {
+    "model": "cua_s1_forms_int4_options32.mlpackage",
+    "precision": "int4_weights_float16_compute",
+    "minimum_target": "iOS18/macOS15",
+    "limits": {
+      "context_bytes": 224,
+      "option_bytes": 96,
+      "max_options": 32
+    },
+    "model_config": {
+      "context_tokens": 224,
+      "encoder": "tinyx",
+      "heads": 4,
+      "hf_model": "Qwen/Qwen2.5-0.5B",
+      "layers": 2,
+      "option_tokens": 96,
+      "rank": 128,
+      "width": 128
+    },
+    "parameters": 706048,
+    "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+    "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+    "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+    "trace_row": 0,
+    "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+    "export_seconds": 0.5374167499830946,
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "cadad6bfa6875383a7ee3de68d0d22871148aa21cf117e7fe44c2c971287c4e5",
+      "Data/com.apple.CoreML/weights/weight.bin": "0f766fcb9dbd612ed4f6c7a40117a5993a4789b3ae94afba0df37193c831f63c",
+      "Manifest.json": "fb91c8b5ec3168111567de663bb3140d2790c6e7c48f32e0df3f5135af0673af"
+    },
+    "target_upgrade": {
+      "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+      "source_package_files": {
+        "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+        "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+        "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+      },
+      "method": "Load original MIL at specification 9; empty pass pipeline; retain decomposed attention",
+      "operation_counts": {
+        "add": 13,
+        "cast": 11,
+        "concat": 2,
+        "const": 239,
+        "equal": 2,
+        "expand_dims": 5,
+        "fill_like": 3,
+        "gather": 2,
+        "greater_equal": 2,
+        "layer_norm": 8,
+        "linear": 15,
+        "logical_not": 2,
+        "matmul": 8,
+        "maximum": 1,
+        "mul": 7,
+        "not_equal": 3,
+        "real_div": 1,
+        "reduce_sum": 3,
+        "relu": 3,
+        "reshape": 31,
+        "select": 7,
+        "slice_by_index": 13,
+        "softmax": 5,
+        "squeeze": 3,
+        "tile": 2,
+        "transpose": 21
+      },
+      "script_sha256": "33db3295450428d90c1de5a804ab47bf3507c686fbbdef67cd276ce116758b5f"
+    },
+    "quantization": {
+      "name": "int4-weights",
+      "settings": {
+        "mode": "linear_symmetric",
+        "dtype": "int4",
+        "granularity": "per_channel",
+        "weight_threshold": 2048
+      },
+      "activations": "float16",
+      "calibration": "None; weight-only, data-free quantization",
+      "source_conversion_sha256": "01105dc8bef7a30fc3430e150caa2a143dc0ba23a295b6280de3b8ff3b40af55",
+      "source_package_files": {
+        "Data/com.apple.CoreML/model.mlmodel": "6ab2df320df7081480b76876df0f4bfd2c841592d93f354c568ba504f7269627",
+        "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+        "Manifest.json": "e5b1c7dc66b2014b3985c180a75548a709788c6e314d9d2ae14e4e6d441c3c68"
+      },
+      "source_package_bytes": 1511080,
+      "package_bytes": 451595,
+      "compression_seconds": 0.2996572910051327,
+      "packed_int4_tensors": 19,
+      "operation_counts": {
+        "add": 15,
+        "cast": 13,
+        "concat": 2,
+        "const": 226,
+        "constexpr_blockwise_shift_scale": 19,
+        "equal": 2,
+        "expand_dims": 5,
+        "fill_like": 3,
+        "gather": 2,
+        "greater_equal": 4,
+        "layer_norm": 8,
+        "linear": 15,
+        "logical_not": 2,
+        "matmul": 8,
+        "maximum": 1,
+        "mul": 7,
+        "not_equal": 3,
+        "real_div": 1,
+        "reduce_sum": 3,
+        "relu": 3,
+        "reshape": 31,
+        "select": 9,
+        "slice_by_index": 13,
+        "softmax": 5,
+        "squeeze": 3,
+        "tile": 2,
+        "transpose": 21
+      },
+      "script_sha256": "a91636e539d8934253b5e7628ca706346fbee538fa1aa9e3eaf794f440d49dcc"
+    }
+  },
+  "environment": {
+    "machine": "Apple M5 Pro",
+    "os": "27.0",
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "torch_cpu_threads": 2
+  },
+  "rows": 196,
+  "upstream": {
+    "correct": 196,
+    "accuracy": 1.0,
+    "inference_latency_ms": {
+      "count": 196,
+      "median_ms": 2.3931670002639294,
+      "p95_ms": 3.033687506103888,
+      "min_ms": 1.7705829814076424,
+      "max_ms": 3.819000005023554
+    }
+  },
+  "export_adapter_fp32": {
+    "argmax_agreement": 196,
+    "max_abs_probability_error": 1.1324882507324219e-06
+  },
+  "host_preprocessing_latency_ms": {
+    "count": 196,
+    "median_ms": 0.02962502185255289,
+    "p95_ms": 0.038823003706056625,
+    "min_ms": 0.024125009076669812,
+    "max_ms": 0.06787499296478927
+  },
+  "thresholds": {
+    "argmax_agreement": 1.0,
+    "max_abs_probability_error": 0.005,
+    "export_adapter_max_abs_probability_error": 0.0001,
+    "allow_accuracy_loss": false
+  },
+  "backends": {
+    "CPU_AND_NE": {
+      "passed": false,
+      "correct": 196,
+      "accuracy": 1.0,
+      "argmax_agreement": 196,
+      "max_abs_probability_error": 0.20076976716518402,
+      "mean_row_max_abs_probability_error": 0.0011279110022456198,
+      "max_abs_logit_error": 6.7208967208862305,
+      "load_seconds": 0.6122147499991115,
+      "warm_inference_latency_ms": {
+        "count": 196,
+        "median_ms": 0.9312290058005601,
+        "p95_ms": 1.0691460047382861,
+        "min_ms": 0.8852920145727694,
+        "max_ms": 1.2105409987270832
+      },
+      "reversed_option_order": {
+        "rows": 6,
+        "max_abs_probability_error": 6.0313243011478335e-05
+      },
+      "per_action": {
+        "fill": {
+          "rows": 36,
+          "correct": 36
+        },
+        "skip": {
+          "rows": 150,
+          "correct": 150
+        },
+        "check": {
+          "rows": 4,
+          "correct": 4
+        },
+        "click": {
+          "rows": 6,
+          "correct": 6
+        }
+      },
+      "decisions": [
+        {
+          "row": 0,
+          "label": 22,
+          "upstream": 22,
+          "coreml": 22,
+          "max_abs_probability_error": 6.0313243011478335e-05,
+          "max_abs_logit_error": 2.864961624145508
+        },
+        {
+          "row": 1,
+          "label": 23,
+          "upstream": 23,
+          "coreml": 23,
+          "max_abs_probability_error": 3.814697265625e-05,
+          "max_abs_logit_error": 3.1371307373046875
+        },
+        {
+          "row": 2,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 2.247527599334717
+        },
+        {
+          "row": 3,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 0.00022571229783352464,
+          "max_abs_logit_error": 4.824214935302734
+        },
+        {
+          "row": 4,
+          "label": 4,
+          "upstream": 4,
+          "coreml": 4,
+          "max_abs_probability_error": 3.3578715374460444e-05,
+          "max_abs_logit_error": 3.618511438369751
+        },
+        {
+          "row": 5,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 6.828465848229825e-05,
+          "max_abs_logit_error": 3.7253708839416504
+        },
+        {
+          "row": 6,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 2.3365020751953125e-05,
+          "max_abs_logit_error": 2.4195241928100586
+        },
+        {
+          "row": 7,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 0.004736423492431641,
+          "max_abs_logit_error": 3.612703323364258
+        },
+        {
+          "row": 8,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 1.1920928955078125e-06,
+          "max_abs_logit_error": 3.931535482406616
+        },
+        {
+          "row": 9,
+          "label": 10,
+          "upstream": 10,
+          "coreml": 10,
+          "max_abs_probability_error": 2.1457672119140625e-05,
+          "max_abs_logit_error": 3.5902247428894043
+        },
+        {
+          "row": 10,
+          "label": 11,
+          "upstream": 11,
+          "coreml": 11,
+          "max_abs_probability_error": 9.810033816393116e-07,
+          "max_abs_logit_error": 4.762610912322998
+        },
+        {
+          "row": 11,
+          "label": 13,
+          "upstream": 13,
+          "coreml": 13,
+          "max_abs_probability_error": 4.136032657697797e-06,
+          "max_abs_logit_error": 3.376967430114746
+        },
+        {
+          "row": 12,
+          "label": 15,
+          "upstream": 15,
+          "coreml": 15,
+          "max_abs_probability_error": 3.272867388659506e-06,
+          "max_abs_logit_error": 2.41459059715271
+        },
+        {
+          "row": 13,
+          "label": 16,
+          "upstream": 16,
+          "coreml": 16,
+          "max_abs_probability_error": 0.006299853324890137,
+          "max_abs_logit_error": 2.80385684967041
+        },
+        {
+          "row": 14,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.480397471226752e-05,
+          "max_abs_logit_error": 2.1806278228759766
+        },
+        {
+          "row": 15,
+          "label": 24,
+          "upstream": 24,
+          "coreml": 24,
+          "max_abs_probability_error": 1.6689300537109375e-05,
+          "max_abs_logit_error": 1.947547435760498
+        },
+        {
+          "row": 16,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 4.5203719878372794e-08,
+          "max_abs_logit_error": 3.743783950805664
+        },
+        {
+          "row": 17,
+          "label": 25,
+          "upstream": 25,
+          "coreml": 25,
+          "max_abs_probability_error": 5.519996193470433e-06,
+          "max_abs_logit_error": 3.9063854217529297
+        },
+        {
+          "row": 18,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 8.430758313693332e-09,
+          "max_abs_logit_error": 1.3515470027923584
+        },
+        {
+          "row": 19,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.0266471551243512e-09,
+          "max_abs_logit_error": 3.501469850540161
+        },
+        {
+          "row": 20,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4500966588902884e-08,
+          "max_abs_logit_error": 3.2016849517822266
+        },
+        {
+          "row": 21,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.647379281599637e-11,
+          "max_abs_logit_error": 2.692106246948242
+        },
+        {
+          "row": 22,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 2.046893835067749
+        },
+        {
+          "row": 23,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 4.133575439453125
+        },
+        {
+          "row": 24,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.5055844551924444e-11,
+          "max_abs_logit_error": 2.4618735313415527
+        },
+        {
+          "row": 25,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.995705792434137e-09,
+          "max_abs_logit_error": 3.3376502990722656
+        },
+        {
+          "row": 26,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.470584068551943e-10,
+          "max_abs_logit_error": 2.2433438301086426
+        },
+        {
+          "row": 27,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.973024935241767e-11,
+          "max_abs_logit_error": 2.810356616973877
+        },
+        {
+          "row": 28,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.994321927715873e-09,
+          "max_abs_logit_error": 3.7122511863708496
+        },
+        {
+          "row": 29,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4426039316183257e-11,
+          "max_abs_logit_error": 2.806028366088867
+        },
+        {
+          "row": 30,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.498983349141781e-07,
+          "max_abs_logit_error": 2.376333236694336
+        },
+        {
+          "row": 31,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.78299757975276e-10,
+          "max_abs_logit_error": 2.9533097743988037
+        },
+        {
+          "row": 32,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.185003176284454e-09,
+          "max_abs_logit_error": 2.1958632469177246
+        },
+        {
+          "row": 33,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.4556297823229443e-09,
+          "max_abs_logit_error": 3.4985671043395996
+        },
+        {
+          "row": 34,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 0.0004820823669433594,
+          "max_abs_logit_error": 3.9588379859924316
+        },
+        {
+          "row": 35,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.462750809366753e-09,
+          "max_abs_logit_error": 2.490422248840332
+        },
+        {
+          "row": 36,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.7510341227277024e-10,
+          "max_abs_logit_error": 1.6093449592590332
+        },
+        {
+          "row": 37,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.74394488744656e-08,
+          "max_abs_logit_error": 1.8185267448425293
+        },
+        {
+          "row": 38,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 9.552543478452691e-11,
+          "max_abs_logit_error": 2.2058591842651367
+        },
+        {
+          "row": 39,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 4.04716482549361e-10,
+          "max_abs_logit_error": 2.799337863922119
+        },
+        {
+          "row": 40,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 0.0019283411093056202,
+          "max_abs_logit_error": 3.8344945907592773
+        },
+        {
+          "row": 41,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.345841103121529e-08,
+          "max_abs_logit_error": 2.8801822662353516
+        },
+        {
+          "row": 42,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8819494851385343e-09,
+          "max_abs_logit_error": 1.621840476989746
+        },
+        {
+          "row": 43,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 2.8215017318725586
+        },
+        {
+          "row": 44,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.780525865635596e-10,
+          "max_abs_logit_error": 1.614362359046936
+        },
+        {
+          "row": 45,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.414331317674879e-11,
+          "max_abs_logit_error": 1.6064977645874023
+        },
+        {
+          "row": 46,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.599257724142603e-10,
+          "max_abs_logit_error": 1.4032363891601562
+        },
+        {
+          "row": 47,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.212314607705437e-11,
+          "max_abs_logit_error": 1.6732616424560547
+        },
+        {
+          "row": 48,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.480397471226752e-05,
+          "max_abs_logit_error": 2.1806278228759766
+        },
+        {
+          "row": 49,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.932842080831776e-10,
+          "max_abs_logit_error": 1.8353347778320312
+        },
+        {
+          "row": 50,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 4.5203719878372794e-08,
+          "max_abs_logit_error": 3.743783950805664
+        },
+        {
+          "row": 51,
+          "label": 25,
+          "upstream": 25,
+          "coreml": 25,
+          "max_abs_probability_error": 5.519996193470433e-06,
+          "max_abs_logit_error": 3.9063854217529297
+        },
+        {
+          "row": 52,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 8.430758313693332e-09,
+          "max_abs_logit_error": 1.3515470027923584
+        },
+        {
+          "row": 53,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.0266471551243512e-09,
+          "max_abs_logit_error": 3.501469850540161
+        },
+        {
+          "row": 54,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4500966588902884e-08,
+          "max_abs_logit_error": 3.2016849517822266
+        },
+        {
+          "row": 55,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.647379281599637e-11,
+          "max_abs_logit_error": 2.692106246948242
+        },
+        {
+          "row": 56,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 2.046893835067749
+        },
+        {
+          "row": 57,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 4.133575439453125
+        },
+        {
+          "row": 58,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.5055844551924444e-11,
+          "max_abs_logit_error": 2.4618735313415527
+        },
+        {
+          "row": 59,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.995705792434137e-09,
+          "max_abs_logit_error": 3.3376502990722656
+        },
+        {
+          "row": 60,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.470584068551943e-10,
+          "max_abs_logit_error": 2.2433438301086426
+        },
+        {
+          "row": 61,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.973024935241767e-11,
+          "max_abs_logit_error": 2.810356616973877
+        },
+        {
+          "row": 62,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.994321927715873e-09,
+          "max_abs_logit_error": 3.7122511863708496
+        },
+        {
+          "row": 63,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4426039316183257e-11,
+          "max_abs_logit_error": 2.806028366088867
+        },
+        {
+          "row": 64,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.498983349141781e-07,
+          "max_abs_logit_error": 2.376333236694336
+        },
+        {
+          "row": 65,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.78299757975276e-10,
+          "max_abs_logit_error": 2.9533097743988037
+        },
+        {
+          "row": 66,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.185003176284454e-09,
+          "max_abs_logit_error": 2.1958632469177246
+        },
+        {
+          "row": 67,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.4556297823229443e-09,
+          "max_abs_logit_error": 3.4985671043395996
+        },
+        {
+          "row": 68,
+          "label": 0,
+          "upstream": 0,
+          "coreml": 0,
+          "max_abs_probability_error": 0.00011065257422160357,
+          "max_abs_logit_error": 2.1242733001708984
+        },
+        {
+          "row": 69,
+          "label": 1,
+          "upstream": 1,
+          "coreml": 1,
+          "max_abs_probability_error": 5.5948574299691245e-05,
+          "max_abs_logit_error": 4.156556606292725
+        },
+        {
+          "row": 70,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 3.2760276553744916e-06,
+          "max_abs_logit_error": 4.688884735107422
+        },
+        {
+          "row": 71,
+          "label": 3,
+          "upstream": 3,
+          "coreml": 3,
+          "max_abs_probability_error": 5.781622166978195e-05,
+          "max_abs_logit_error": 2.7651166915893555
+        },
+        {
+          "row": 72,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 1.0177071089856327e-05,
+          "max_abs_logit_error": 2.8724193572998047
+        },
+        {
+          "row": 73,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 2.4447179384878837e-05,
+          "max_abs_logit_error": 2.4996681213378906
+        },
+        {
+          "row": 74,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 4.470348358154297e-05,
+          "max_abs_logit_error": 6.7208967208862305
+        },
+        {
+          "row": 75,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 3.379677946213633e-05,
+          "max_abs_logit_error": 5.287045478820801
+        },
+        {
+          "row": 76,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 4.347872618382098e-06,
+          "max_abs_logit_error": 3.2474308013916016
+        },
+        {
+          "row": 77,
+          "label": 13,
+          "upstream": 13,
+          "coreml": 13,
+          "max_abs_probability_error": 0.0006129741668701172,
+          "max_abs_logit_error": 4.197287559509277
+        },
+        {
+          "row": 78,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.0681543446698925e-06,
+          "max_abs_logit_error": 2.6077210903167725
+        },
+        {
+          "row": 79,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 0.0035958904772996902,
+          "max_abs_logit_error": 1.8085737228393555
+        },
+        {
+          "row": 80,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.3343336604521028e-06,
+          "max_abs_logit_error": 1.546478271484375
+        },
+        {
+          "row": 81,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 1.7384634017944336
+        },
+        {
+          "row": 82,
+          "label": 19,
+          "upstream": 19,
+          "coreml": 19,
+          "max_abs_probability_error": 6.477808369709237e-07,
+          "max_abs_logit_error": 2.3306055068969727
+        },
+        {
+          "row": 83,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.0942090545902374e-09,
+          "max_abs_logit_error": 1.5485305786132812
+        },
+        {
+          "row": 84,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 1.5241689682006836
+        },
+        {
+          "row": 85,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.5405053888659666e-10,
+          "max_abs_logit_error": 2.1916251182556152
+        },
+        {
+          "row": 86,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.236582296674896e-07,
+          "max_abs_logit_error": 3.8295583724975586
+        },
+        {
+          "row": 87,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 3.709770679473877
+        },
+        {
+          "row": 88,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3188037328859537e-09,
+          "max_abs_logit_error": 3.0971665382385254
+        },
+        {
+          "row": 89,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9970605169561395e-09,
+          "max_abs_logit_error": 4.972865104675293
+        },
+        {
+          "row": 90,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7617375336342889e-09,
+          "max_abs_logit_error": 1.8527463674545288
+        },
+        {
+          "row": 91,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.612549260787688e-10,
+          "max_abs_logit_error": 2.7531187534332275
+        },
+        {
+          "row": 92,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.506330630512425e-12,
+          "max_abs_logit_error": 3.0166378021240234
+        },
+        {
+          "row": 93,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.5150824594911683e-08,
+          "max_abs_logit_error": 3.1289987564086914
+        },
+        {
+          "row": 94,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3666674458789885e-09,
+          "max_abs_logit_error": 1.6400773525238037
+        },
+        {
+          "row": 95,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.969476136678196e-10,
+          "max_abs_logit_error": 1.7170515060424805
+        },
+        {
+          "row": 96,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.657726538846106e-10,
+          "max_abs_logit_error": 2.2562785148620605
+        },
+        {
+          "row": 97,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 1.4498934745788574
+        },
+        {
+          "row": 98,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 2.764932632446289
+        },
+        {
+          "row": 99,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.352701585550676e-06,
+          "max_abs_logit_error": 3.0918569564819336
+        },
+        {
+          "row": 100,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 2.7188186645507812
+        },
+        {
+          "row": 101,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.1713853704550274e-08,
+          "max_abs_logit_error": 1.6199064254760742
+        },
+        {
+          "row": 102,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.5862225694339713e-09,
+          "max_abs_logit_error": 4.226329803466797
+        },
+        {
+          "row": 103,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 0.00024586915969848633,
+          "max_abs_logit_error": 2.840273380279541
+        },
+        {
+          "row": 104,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.841255187988281e-06,
+          "max_abs_logit_error": 3.6580047607421875
+        },
+        {
+          "row": 105,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.3792570402747515e-07,
+          "max_abs_logit_error": 6.307399272918701
+        },
+        {
+          "row": 106,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.4136979082101107e-09,
+          "max_abs_logit_error": 1.3745036125183105
+        },
+        {
+          "row": 107,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.302839361538304e-10,
+          "max_abs_logit_error": 1.8023838996887207
+        },
+        {
+          "row": 108,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 3.241445541381836
+        },
+        {
+          "row": 109,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.0681543446698925e-06,
+          "max_abs_logit_error": 2.6077210903167725
+        },
+        {
+          "row": 110,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 2.4543886184692383
+        },
+        {
+          "row": 111,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.811981201171875e-05,
+          "max_abs_logit_error": 1.956678867340088
+        },
+        {
+          "row": 112,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 1.7384634017944336
+        },
+        {
+          "row": 113,
+          "label": 19,
+          "upstream": 19,
+          "coreml": 19,
+          "max_abs_probability_error": 6.477808369709237e-07,
+          "max_abs_logit_error": 2.3306055068969727
+        },
+        {
+          "row": 114,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.0942090545902374e-09,
+          "max_abs_logit_error": 1.5485305786132812
+        },
+        {
+          "row": 115,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 1.5241689682006836
+        },
+        {
+          "row": 116,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.5405053888659666e-10,
+          "max_abs_logit_error": 2.1916251182556152
+        },
+        {
+          "row": 117,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.236582296674896e-07,
+          "max_abs_logit_error": 3.8295583724975586
+        },
+        {
+          "row": 118,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 3.709770679473877
+        },
+        {
+          "row": 119,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3188037328859537e-09,
+          "max_abs_logit_error": 3.0971665382385254
+        },
+        {
+          "row": 120,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9970605169561395e-09,
+          "max_abs_logit_error": 4.972865104675293
+        },
+        {
+          "row": 121,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7617375336342889e-09,
+          "max_abs_logit_error": 1.8527463674545288
+        },
+        {
+          "row": 122,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.612549260787688e-10,
+          "max_abs_logit_error": 2.7531187534332275
+        },
+        {
+          "row": 123,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.506330630512425e-12,
+          "max_abs_logit_error": 3.0166378021240234
+        },
+        {
+          "row": 124,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.5150824594911683e-08,
+          "max_abs_logit_error": 3.1289987564086914
+        },
+        {
+          "row": 125,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3666674458789885e-09,
+          "max_abs_logit_error": 1.6400773525238037
+        },
+        {
+          "row": 126,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.969476136678196e-10,
+          "max_abs_logit_error": 1.7170515060424805
+        },
+        {
+          "row": 127,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.657726538846106e-10,
+          "max_abs_logit_error": 2.2562785148620605
+        },
+        {
+          "row": 128,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 1.4498934745788574
+        },
+        {
+          "row": 129,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 2.764932632446289
+        },
+        {
+          "row": 130,
+          "label": 1,
+          "upstream": 1,
+          "coreml": 1,
+          "max_abs_probability_error": 4.3291383917676285e-05,
+          "max_abs_logit_error": 3.559276580810547
+        },
+        {
+          "row": 131,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 3.2550487518310547
+        },
+        {
+          "row": 132,
+          "label": 3,
+          "upstream": 3,
+          "coreml": 3,
+          "max_abs_probability_error": 2.7894973754882812e-05,
+          "max_abs_logit_error": 3.068411111831665
+        },
+        {
+          "row": 133,
+          "label": 4,
+          "upstream": 4,
+          "coreml": 4,
+          "max_abs_probability_error": 0.0002166023332392797,
+          "max_abs_logit_error": 3.101825714111328
+        },
+        {
+          "row": 134,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 6.258487701416016e-05,
+          "max_abs_logit_error": 2.6334457397460938
+        },
+        {
+          "row": 135,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 1.2993812561035156e-05,
+          "max_abs_logit_error": 3.064577102661133
+        },
+        {
+          "row": 136,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 3.9360515074804425e-05,
+          "max_abs_logit_error": 3.498318672180176
+        },
+        {
+          "row": 137,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 0.20076976716518402,
+          "max_abs_logit_error": 4.9444427490234375
+        },
+        {
+          "row": 138,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 0.000747382640838623,
+          "max_abs_logit_error": 2.80314302444458
+        },
+        {
+          "row": 139,
+          "label": 10,
+          "upstream": 10,
+          "coreml": 10,
+          "max_abs_probability_error": 5.860753663000651e-05,
+          "max_abs_logit_error": 4.217677116394043
+        },
+        {
+          "row": 140,
+          "label": 11,
+          "upstream": 11,
+          "coreml": 11,
+          "max_abs_probability_error": 1.1063334568461869e-05,
+          "max_abs_logit_error": 3.9767203330993652
+        },
+        {
+          "row": 141,
+          "label": 12,
+          "upstream": 12,
+          "coreml": 12,
+          "max_abs_probability_error": 9.34600830078125e-05,
+          "max_abs_logit_error": 2.8596105575561523
+        },
+        {
+          "row": 142,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 3.8807570934295654
+        },
+        {
+          "row": 143,
+          "label": 16,
+          "upstream": 16,
+          "coreml": 16,
+          "max_abs_probability_error": 1.6191354461625451e-06,
+          "max_abs_logit_error": 2.416942596435547
+        },
+        {
+          "row": 144,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.6463640097063035e-06,
+          "max_abs_logit_error": 2.728714942932129
+        },
+        {
+          "row": 145,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.6954879760742188e-06,
+          "max_abs_logit_error": 3.8880889415740967
+        },
+        {
+          "row": 146,
+          "label": 17,
+          "upstream": 17,
+          "coreml": 17,
+          "max_abs_probability_error": 5.8316572904004715e-06,
+          "max_abs_logit_error": 4.496463298797607
+        },
+        {
+          "row": 147,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.429776876539563e-09,
+          "max_abs_logit_error": 2.1004066467285156
+        },
+        {
+          "row": 148,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 9.215698426601193e-09,
+          "max_abs_logit_error": 1.8194046020507812
+        },
+        {
+          "row": 149,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.84208304060607e-10,
+          "max_abs_logit_error": 4.05654764175415
+        },
+        {
+          "row": 150,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.947568818333917e-12,
+          "max_abs_logit_error": 2.169252395629883
+        },
+        {
+          "row": 151,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.291168614363073e-10,
+          "max_abs_logit_error": 2.1067094802856445
+        },
+        {
+          "row": 152,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.924417788629512e-10,
+          "max_abs_logit_error": 3.3871490955352783
+        },
+        {
+          "row": 153,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0083449808462319e-07,
+          "max_abs_logit_error": 4.667112827301025
+        },
+        {
+          "row": 154,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.3051329034750054e-11,
+          "max_abs_logit_error": 2.944230079650879
+        },
+        {
+          "row": 155,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.650468511860126e-10,
+          "max_abs_logit_error": 2.179288864135742
+        },
+        {
+          "row": 156,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.181078218703284e-12,
+          "max_abs_logit_error": 2.217021942138672
+        },
+        {
+          "row": 157,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.6455586893115992e-10,
+          "max_abs_logit_error": 2.0631823539733887
+        },
+        {
+          "row": 158,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.109900363426846e-10,
+          "max_abs_logit_error": 3.4993996620178223
+        },
+        {
+          "row": 159,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.693580800742893e-09,
+          "max_abs_logit_error": 2.16013240814209
+        },
+        {
+          "row": 160,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 4.588627788848498e-08,
+          "max_abs_logit_error": 3.608731269836426
+        },
+        {
+          "row": 161,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 3.2034151554107666
+        },
+        {
+          "row": 162,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.042114572413766e-08,
+          "max_abs_logit_error": 3.6694984436035156
+        },
+        {
+          "row": 163,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9131388140358752e-10,
+          "max_abs_logit_error": 2.9220056533813477
+        },
+        {
+          "row": 164,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 4.935061781452532e-08,
+          "max_abs_logit_error": 1.4873247146606445
+        },
+        {
+          "row": 165,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.15840287151903e-11,
+          "max_abs_logit_error": 1.6720411777496338
+        },
+        {
+          "row": 166,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.6797479790729994e-09,
+          "max_abs_logit_error": 1.2103171348571777
+        },
+        {
+          "row": 167,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.54509399403247e-10,
+          "max_abs_logit_error": 2.835115432739258
+        },
+        {
+          "row": 168,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 3.596881866455078
+        },
+        {
+          "row": 169,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.4169340140378637e-11,
+          "max_abs_logit_error": 1.5558462142944336
+        },
+        {
+          "row": 170,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.0329603412093675e-11,
+          "max_abs_logit_error": 2.1443252563476562
+        },
+        {
+          "row": 171,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 1.609299659729004
+        },
+        {
+          "row": 172,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 4.66337235494052e-09,
+          "max_abs_logit_error": 2.1106224060058594
+        },
+        {
+          "row": 173,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.2885730260592254e-09,
+          "max_abs_logit_error": 1.5084967613220215
+        },
+        {
+          "row": 174,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.1279766010119374e-09,
+          "max_abs_logit_error": 2.5634703636169434
+        },
+        {
+          "row": 175,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 3.8807570934295654
+        },
+        {
+          "row": 176,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0728836059570312e-06,
+          "max_abs_logit_error": 2.2860074043273926
+        },
+        {
+          "row": 177,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.665992202670168e-08,
+          "max_abs_logit_error": 2.732346534729004
+        },
+        {
+          "row": 178,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.6954879760742188e-06,
+          "max_abs_logit_error": 3.8880889415740967
+        },
+        {
+          "row": 179,
+          "label": 17,
+          "upstream": 17,
+          "coreml": 17,
+          "max_abs_probability_error": 5.8316572904004715e-06,
+          "max_abs_logit_error": 4.496463298797607
+        },
+        {
+          "row": 180,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.429776876539563e-09,
+          "max_abs_logit_error": 2.1004066467285156
+        },
+        {
+          "row": 181,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 9.215698426601193e-09,
+          "max_abs_logit_error": 1.8194046020507812
+        },
+        {
+          "row": 182,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.84208304060607e-10,
+          "max_abs_logit_error": 4.05654764175415
+        },
+        {
+          "row": 183,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.947568818333917e-12,
+          "max_abs_logit_error": 2.169252395629883
+        },
+        {
+          "row": 184,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.291168614363073e-10,
+          "max_abs_logit_error": 2.1067094802856445
+        },
+        {
+          "row": 185,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.924417788629512e-10,
+          "max_abs_logit_error": 3.3871490955352783
+        },
+        {
+          "row": 186,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0083449808462319e-07,
+          "max_abs_logit_error": 4.667112827301025
+        },
+        {
+          "row": 187,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.3051329034750054e-11,
+          "max_abs_logit_error": 2.944230079650879
+        },
+        {
+          "row": 188,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.650468511860126e-10,
+          "max_abs_logit_error": 2.179288864135742
+        },
+        {
+          "row": 189,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.181078218703284e-12,
+          "max_abs_logit_error": 2.217021942138672
+        },
+        {
+          "row": 190,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.6455586893115992e-10,
+          "max_abs_logit_error": 2.0631823539733887
+        },
+        {
+          "row": 191,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.109900363426846e-10,
+          "max_abs_logit_error": 3.4993996620178223
+        },
+        {
+          "row": 192,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.693580800742893e-09,
+          "max_abs_logit_error": 2.16013240814209
+        },
+        {
+          "row": 193,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 4.588627788848498e-08,
+          "max_abs_logit_error": 3.608731269836426
+        },
+        {
+          "row": 194,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 3.2034151554107666
+        },
+        {
+          "row": 195,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.042114572413766e-08,
+          "max_abs_logit_error": 3.6694984436035156
+        }
+      ]
+    }
+  },
+  "passed": false,
+  "code_sha256": {
+    "ane_gather.py": "c3379a3797041dcfed82eaaea3990771ec91f1dff7ec146b025988ad7a8c9aae",
+    "assets.py": "f7a5bc931075237e00c73bf21fee3575be35531f934f23b5e22627d50a79f92f",
+    "benchmark-synthetic.py": "ba8255fd3f63d87ef340c26c4c251800f2e20878d80858ad1e6557cf59f249a2",
+    "compare-ane.py": "4633df7ad5f90d6aed37d5ef4f7a2d2231fdadd272ef1aa3057c237b2f937df8",
+    "convert-coreml.py": "46fe0194df924033126286e601214b6789f5d667aae38c153cf4dbc1b0e629b0",
+    "export-reference.py": "ff9eab4d6e4e72ffaf074d82610f41fbea1e802da7c0e987ce562a89eb561b99",
+    "export_model.py": "c61da5d678c77030d1e9a78359626063ea31da46fb6434e49517fdeaa7800949",
+    "prepare-int4-source.py": "33db3295450428d90c1de5a804ab47bf3507c686fbbdef67cd276ce116758b5f",
+    "preprocessing.py": "18c74ede43f95f91cd638c7e9631923a353e3a24d3b522ddb8b18b480f9c2846",
+    "profile-coreml.py": "d5e1386f54fc6c482647120e593e7ffbbe4bbb277c2cc6d4949154660fe239fd",
+    "quantize-int4.py": "a91636e539d8934253b5e7628ca706346fbee538fa1aa9e3eaf794f440d49dcc",
+    "quantize-int8.py": "22071babfcb02ef77225203a479d9bb14b9fa35462ba38bcc4bf07d29c125a97",
+    "score-report.py": "c78e45caa86fb59f9a56ef18cc6a35020ea6e281a6a23d23618f56da85a0d95e",
+    "synthetic_test.py": "e2b68725236e6827eb422480c8cc8aa189b2490c1594650d8f2928771d1bc626",
+    "verify.py": "bc20ec9df47708e9b48e411076abe4eab7c18313f67e11334de2d9e3ecb3efa7"
+  }
+}

reports/int4-fallback.json ADDED Viewed

	@@ -0,0 +1,112 @@

+{
+  "hardware": {
+    "device": "arm64",
+    "chip": "Apple M5 Pro",
+    "ram": "24GB",
+    "os_version": "macOS 27.0",
+    "timestamp": "2026-09-19T22:28:18.483796+00:00"
+  },
+  "models": [
+    {
+      "model_path": "/Users/hanweng/Documents/FluidAudio/.mobius/cua-s1-prs/mobius/models/computer-use/cua-s1-forms/coreml/build/int4-weights/cua_s1_forms_int4_options32.mlmodelc",
+      "model_name": "cua_s1_forms_int4_options32",
+      "fallback": {
+        "compute_units": "cpu_and_neural_engine",
+        "total_ops": 200,
+        "ane_ops": 149,
+        "gpu_ops": 0,
+        "cpu_ops": 51,
+        "ane_percent": 74.5,
+        "reasons": [
+          {
+            "reason": "Unsupported tensor data type: int32",
+            "count": 25,
+            "estimated_cpu_runtime_ms": 0.0375,
+            "op_types": {
+              "ios18.cast": 10,
+              "ios18.slice_by_index": 4,
+              "ios18.add": 4,
+              "select": 4,
+              "ios18.gather": 2,
+              "ios18.reshape": 1
+            },
+            "ops": [
+              "var_27",
+              "var_43",
+              "var_43_to_fp16",
+              "context_ids_to_int16",
+              "cast_54",
+              "add_3",
+              "select_0",
+              "select_0_to_int16",
+              "cast_0",
+              "add_0",
+              "select_0_1",
+              "op_59_cast_fp16_cast_uint16_cast_uint16",
+              "flat_ids",
+              "var_333",
+              "var_349",
+              "var_349_to_fp16",
+              "flat_ids_to_int16",
+              "cast_55",
+              "add_4",
+              "select_1",
+              "select_1_to_int16",
+              "cast_1",
+              "add_1",
+              "select_1_1",
+              "op_365_cast_fp16_cast_uint16_cast_uint16"
+            ]
+          },
+          {
+            "reason": "ANE not available for this op",
+            "count": 19,
+            "estimated_cpu_runtime_ms": 0,
+            "op_types": {
+              "ios18.constexpr_blockwise_shift_scale": 19
+            },
+            "ops": [
+              "model_embedding_weight_to_fp16_quantized",
+              "model_position_weight_to_fp16_quantized",
+              "model_encoder_layers_0_self_attn_in_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_0_self_attn_out_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_0_linear1_weight_to_fp16_quantized",
+              "model_encoder_layers_0_linear2_weight_to_fp16_quantized",
+              "model_encoder_layers_1_self_attn_in_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_1_self_attn_out_proj_weight_to_fp16_quantized",
+              "model_encoder_layers_1_linear1_weight_to_fp16_quantized",
+              "model_encoder_layers_1_linear2_weight_to_fp16_quantized",
+              "op_369_to_fp16_quantized",
+              "op_394_to_fp16_quantized",
+              "model_option_encoder_layers_0_self_attn_in_proj_weight_to_fp16_quantized",
+              "model_option_encoder_layers_0_self_attn_out_proj_weight_to_fp16_quantized",
+              "model_option_encoder_layers_0_linear1_weight_to_fp16_quantized",
+              "model_option_encoder_layers_0_linear2_weight_to_fp16_quantized",
+              "model_head_query_weight_to_fp16_quantized",
+              "model_head_key_weight_to_fp16_quantized",
+              "model_head_value_weight_to_fp16_quantized"
+            ]
+          },
+          {
+            "reason": "Unable to resolve operation input \"y\".",
+            "count": 7,
+            "estimated_cpu_runtime_ms": 0.007,
+            "op_types": {
+              "ios18.greater_equal": 4,
+              "ios18.not_equal": 3
+            },
+            "ops": [
+              "option_mask_1",
+              "context_mask",
+              "greater_equal_0",
+              "greater_equal_0_1",
+              "flat_mask",
+              "greater_equal_1",
+              "greater_equal_1_1"
+            ]
+          }
+        ]
+      }
+    }
+  ]
+}

reports/int4-profile.json ADDED Viewed

	@@ -0,0 +1,1105 @@

+{
+  "created_utc": "2026-09-19T22:27:49.746625+00:00",
+  "purpose": "Bounded real-input latency and scheduler placement; not a general accuracy benchmark",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "package_files": {
+    "Data/com.apple.CoreML/model.mlmodel": "cadad6bfa6875383a7ee3de68d0d22871148aa21cf117e7fe44c2c971287c4e5",
+    "Data/com.apple.CoreML/weights/weight.bin": "0f766fcb9dbd612ed4f6c7a40117a5993a4789b3ae94afba0df37193c831f63c",
+    "Manifest.json": "fb91c8b5ec3168111567de663bb3140d2790c6e7c48f32e0df3f5135af0673af"
+  },
+  "dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "dataset_sha256": "4f43b442e79ba2e2ce731e27e9b8e340c2b5dfcaffc92d8ff564c34f115ff1ca",
+  "environment": {
+    "chip": "Apple M5 Pro",
+    "memory_bytes": 25769803776,
+    "macos": "27.0",
+    "os_build": "26A428",
+    "python": "3.11.11",
+    "coremltools": "9.0"
+  },
+  "protocol": {
+    "rows": [
+      0,
+      68,
+      130
+    ],
+    "pages": [
+      "patient-registration",
+      "job-application",
+      "auto-claim"
+    ],
+    "option_counts": [
+      27,
+      21,
+      19
+    ],
+    "first_call_row": 0,
+    "warmup_passes": 2,
+    "timed_passes": 10,
+    "backend_order": [
+      "CPU_AND_NE"
+    ],
+    "timing_scope": "Synchronous Python MLModel.predict; pre-encoded inputs; output checks outside timer",
+    "load_scope": "MLModel construction; system caches retained, not first-install cold start",
+    "first_call_scope": "First predict after this load; may use existing system/ANE caches",
+    "plan_order": "Compute plan loaded after timing each backend"
+  },
+  "backends": {
+    "CPU_AND_NE": {
+      "load_ms": 628.8241670117714,
+      "first_prediction_ms": 2.4197499733418226,
+      "warm_prediction": {
+        "count": 30,
+        "median_ms": 0.9370829939143732,
+        "p95_ms": 0.969368452206254,
+        "min_ms": 0.8955409866757691,
+        "max_ms": 0.985999999102205
+      },
+      "timed_predictions_correct": 30,
+      "calls": [
+        {
+          "row": 0,
+          "milliseconds": 0.9536660218145698
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9417080145794898
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9293750044889748
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9304159902967513
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9457910200580955
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9614159935154021
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9506249916739762
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9356669906992465
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9454160172026604
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9370410116389394
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9322499972768128
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.975875009316951
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.985999999102205
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.960250006755814
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9522080072201788
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9541249892208725
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9371249761898071
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9184999798890203
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9398749971296638
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9219589992426336
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9396669920533895
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9407500037923455
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9344589780084789
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.931665999814868
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9232910233549774
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9075840061996132
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.8955409866757691
+        },
+        {
+          "row": 0,
+          "milliseconds": 0.9247090201824903
+        },
+        {
+          "row": 68,
+          "milliseconds": 0.9097080037463456
+        },
+        {
+          "row": 130,
+          "milliseconds": 0.9349999891128391
+        }
+      ],
+      "compute_plan": {
+        "kind": "MLComputePlan preferred device; not a runtime utilization or power trace",
+        "assigned_operation_count": 181,
+        "device_counts": {
+          "cpu": 32,
+          "gpu": 0,
+          "ane": 149,
+          "unknown": 0
+        },
+        "unassigned_operation_types": {
+          "const": 226,
+          "ios18.constexpr_blockwise_shift_scale": 19
+        },
+        "operations": [
+          {
+            "name": "option_mask_1",
+            "type": "ios18.not_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "context_mask",
+            "type": "ios18.not_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "var_27",
+            "type": "ios18.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "fill_like_0_cast_fp16",
+            "type": "ios16.fill_like",
+            "device": "ane"
+          },
+          {
+            "name": "var_43",
+            "type": "ios18.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "var_43_to_fp16",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "safe_context_ids_cast_fp16",
+            "type": "ios18.concat",
+            "device": "ane"
+          },
+          {
+            "name": "context_ids_to_int16",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_54",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_0",
+            "type": "ios18.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_3",
+            "type": "ios18.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_0",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "select_0_to_int16",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_0",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_0_1",
+            "type": "ios18.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_0",
+            "type": "ios18.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_0_1",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "op_59_cast_fp16_cast_uint16_cast_uint16",
+            "type": "ios18.gather",
+            "device": "cpu"
+          },
+          {
+            "name": "src_1_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "mask_1_cast_fp16",
+            "type": "ios18.equal",
+            "device": "ane"
+          },
+          {
+            "name": "key_padding_mask_1_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "query_1_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "query_3_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "linear_0_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "var_124_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_125_cast_fp16",
+            "type": "ios18.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_126_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "var_127_cast_fp16",
+            "type": "ios18.squeeze",
+            "device": "ane"
+          },
+          {
+            "name": "q_1_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "k_1_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "v_1_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "var_136_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_143_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_150_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_154_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_156_cast_fp16",
+            "type": "tile",
+            "device": "ane"
+          },
+          {
+            "name": "q_3_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "q_5_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "k_3_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "k_5_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "v_3_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "v_5_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "mul_1_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_0_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "add_0_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "softmax_0_cast_fp16",
+            "type": "ios18.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_1_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_172_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_3_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "linear_1_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_7_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_3_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "input_5_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "input_7_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_2_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_11_cast_fp16",
+            "type": "ios18.relu",
+            "device": "ane"
+          },
+          {
+            "name": "linear_3_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_17_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "query_5_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "query_7_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "linear_4_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "var_231_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_232_cast_fp16",
+            "type": "ios18.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_233_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "var_234_cast_fp16",
+            "type": "ios18.squeeze",
+            "device": "ane"
+          },
+          {
+            "name": "q_7_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "k_7_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "v_7_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "var_243_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_250_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_257_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "q_9_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "q_11_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "k_9_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "k_11_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "v_9_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "v_11_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "mul_3_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_1_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "add_1_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "softmax_1_cast_fp16",
+            "type": "ios18.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_9_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_279_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_11_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "linear_5_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_15_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_19_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "input_21_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "input_23_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_6_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_27_cast_fp16",
+            "type": "ios18.relu",
+            "device": "ane"
+          },
+          {
+            "name": "linear_7_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "context_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "flat_ids",
+            "type": "ios18.reshape",
+            "device": "cpu"
+          },
+          {
+            "name": "flat_mask",
+            "type": "ios18.not_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "var_333",
+            "type": "ios18.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "fill_like_1_cast_fp16",
+            "type": "ios16.fill_like",
+            "device": "ane"
+          },
+          {
+            "name": "var_349",
+            "type": "ios18.slice_by_index",
+            "device": "cpu"
+          },
+          {
+            "name": "var_349_to_fp16",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "safe_ids_cast_fp16",
+            "type": "ios18.concat",
+            "device": "ane"
+          },
+          {
+            "name": "flat_ids_to_int16",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_55",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_1",
+            "type": "ios18.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_4",
+            "type": "ios18.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_1",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "select_1_to_int16",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "cast_1",
+            "type": "ios18.cast",
+            "device": "cpu"
+          },
+          {
+            "name": "greater_equal_1_1",
+            "type": "ios18.greater_equal",
+            "device": "cpu"
+          },
+          {
+            "name": "add_1",
+            "type": "ios18.add",
+            "device": "cpu"
+          },
+          {
+            "name": "select_1_1",
+            "type": "select",
+            "device": "cpu"
+          },
+          {
+            "name": "op_365_cast_fp16_cast_uint16_cast_uint16",
+            "type": "ios18.gather",
+            "device": "cpu"
+          },
+          {
+            "name": "src_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "mask_cast_fp16",
+            "type": "ios18.equal",
+            "device": "ane"
+          },
+          {
+            "name": "key_padding_mask_7_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "query_9_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "query_11_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "linear_8_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "var_428_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_429_cast_fp16",
+            "type": "ios18.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_430_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "var_431_cast_fp16",
+            "type": "ios18.squeeze",
+            "device": "ane"
+          },
+          {
+            "name": "q_13_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "k_13_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "v_13_cast_fp16",
+            "type": "ios18.slice_by_index",
+            "device": "ane"
+          },
+          {
+            "name": "var_440_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_447_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_454_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_458_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "var_460_cast_fp16",
+            "type": "tile",
+            "device": "ane"
+          },
+          {
+            "name": "q_15_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "q_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "k_15_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "k_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "v_15_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "v_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "mul_5_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_2_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "add_2_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "softmax_2_cast_fp16",
+            "type": "ios18.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_17_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_476_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_19_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "linear_9_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "attn_output_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_35_cast_fp16",
+            "type": "ios18.transpose",
+            "device": "ane"
+          },
+          {
+            "name": "input_37_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "input_39_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_10_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "input_43_cast_fp16",
+            "type": "ios18.relu",
+            "device": "ane"
+          },
+          {
+            "name": "linear_11_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "hidden_cast_fp16",
+            "type": "ios18.add",
+            "device": "ane"
+          },
+          {
+            "name": "var_504",
+            "type": "ios18.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_504_to_fp16",
+            "type": "ios18.cast",
+            "device": "ane"
+          },
+          {
+            "name": "var_510_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_515_cast_fp16",
+            "type": "ios16.reduce_sum",
+            "device": "ane"
+          },
+          {
+            "name": "var_520_cast_fp16",
+            "type": "ios16.reduce_sum",
+            "device": "ane"
+          },
+          {
+            "name": "var_522_cast_fp16",
+            "type": "ios18.maximum",
+            "device": "ane"
+          },
+          {
+            "name": "pooled_cast_fp16",
+            "type": "ios18.real_div",
+            "device": "ane"
+          },
+          {
+            "name": "options_cast_fp16",
+            "type": "ios18.reshape",
+            "device": "ane"
+          },
+          {
+            "name": "input_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "input_53_cast_fp16",
+            "type": "ios18.layer_norm",
+            "device": "ane"
+          },
+          {
+            "name": "linear_12_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "linear_13_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "linear_14_cast_fp16",
+            "type": "ios18.linear",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_3_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "_inversed_scores_1_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_567",
+            "type": "ios18.expand_dims",
+            "device": "ane"
+          },
+          {
+            "name": "var_569",
+            "type": "ios18.logical_not",
+            "device": "ane"
+          },
+          {
+            "name": "scores_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "var_571_cast_fp16",
+            "type": "ios18.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "matmul_4_cast_fp16",
+            "type": "ios18.matmul",
+            "device": "ane"
+          },
+          {
+            "name": "var_574_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_576_cast_fp16",
+            "type": "ios16.reduce_sum",
+            "device": "ane"
+          },
+          {
+            "name": "_inversed_logits_1_cast_fp16",
+            "type": "ios18.mul",
+            "device": "ane"
+          },
+          {
+            "name": "var_579",
+            "type": "ios18.logical_not",
+            "device": "ane"
+          },
+          {
+            "name": "logits_3_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "var_587_cast_fp16",
+            "type": "ios16.fill_like",
+            "device": "ane"
+          },
+          {
+            "name": "logits_cast_fp16",
+            "type": "select",
+            "device": "ane"
+          },
+          {
+            "name": "var_591_cast_fp16",
+            "type": "ios18.softmax",
+            "device": "ane"
+          },
+          {
+            "name": "logits",
+            "type": "ios18.cast",
+            "device": "ane"
+          },
+          {
+            "name": "probabilities",
+            "type": "ios18.cast",
+            "device": "ane"
+          }
+        ]
+      }
+    }
+  },
+  "script_sha256": "d5e1386f54fc6c482647120e593e7ffbbe4bbb277c2cc6d4949154660fe239fd"
+}

reports/int4-source-demo-verification.json ADDED Viewed

	@@ -0,0 +1,1748 @@

+{
+  "purpose": "Local conversion parity on the upstream demo; not a generalization or live GUI benchmark",
+  "created_utc": "2026-09-19T22:24:56.262849+00:00",
+  "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+  "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+  "dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+  "dataset_file": "demo.jsonl",
+  "dataset_sha256": "4f43b442e79ba2e2ce731e27e9b8e340c2b5dfcaffc92d8ff564c34f115ff1ca",
+  "conversion": {
+    "model": "cua_s1_forms_fp16_options32.mlpackage",
+    "precision": "float16",
+    "minimum_target": "iOS18/macOS15",
+    "limits": {
+      "context_bytes": 224,
+      "option_bytes": 96,
+      "max_options": 32
+    },
+    "model_config": {
+      "context_tokens": 224,
+      "encoder": "tinyx",
+      "heads": 4,
+      "hf_model": "Qwen/Qwen2.5-0.5B",
+      "layers": 2,
+      "option_tokens": 96,
+      "rank": 128,
+      "width": 128
+    },
+    "parameters": 706048,
+    "assets_lock_sha256": "8e65ad70af6bb814b571cdcfe828ba4bc339147da2d2211cbeac416163ef18ba",
+    "model_revision": "f54adbf447f4ca6ec259f529ee3f2e3e09f8cc71",
+    "source_revision": "83f142c4290a0f7d9ed545ae8532858c6e4f8145",
+    "trace_row": 0,
+    "trace_dataset_revision": "8273f34778b99ac2e12d9f6e7d57dad99ae20845",
+    "export_seconds": 0.5374167499830946,
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "package_files": {
+      "Data/com.apple.CoreML/model.mlmodel": "6ab2df320df7081480b76876df0f4bfd2c841592d93f354c568ba504f7269627",
+      "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+      "Manifest.json": "e5b1c7dc66b2014b3985c180a75548a709788c6e314d9d2ae14e4e6d441c3c68"
+    },
+    "target_upgrade": {
+      "source_conversion_sha256": "ac94239751ceb7e19602eb7276940ff53446b1776d4900405af9be0382fe4a94",
+      "source_package_files": {
+        "Data/com.apple.CoreML/model.mlmodel": "70485fc18cbb21785df833cbddddc0b5b59acb00d22394b76e55307e2c135dd0",
+        "Data/com.apple.CoreML/weights/weight.bin": "4da9259f798e44f5a1b50769ee1916fd3747c4d723dd9997b516c7fe238c7895",
+        "Manifest.json": "2bc0f5f62337b27fb6b0ecde248f1e3dc269e1ba4b65516aaeede2a60e293dcc"
+      },
+      "method": "Load original MIL at specification 9; empty pass pipeline; retain decomposed attention",
+      "operation_counts": {
+        "add": 13,
+        "cast": 11,
+        "concat": 2,
+        "const": 239,
+        "equal": 2,
+        "expand_dims": 5,
+        "fill_like": 3,
+        "gather": 2,
+        "greater_equal": 2,
+        "layer_norm": 8,
+        "linear": 15,
+        "logical_not": 2,
+        "matmul": 8,
+        "maximum": 1,
+        "mul": 7,
+        "not_equal": 3,
+        "real_div": 1,
+        "reduce_sum": 3,
+        "relu": 3,
+        "reshape": 31,
+        "select": 7,
+        "slice_by_index": 13,
+        "softmax": 5,
+        "squeeze": 3,
+        "tile": 2,
+        "transpose": 21
+      },
+      "script_sha256": "33db3295450428d90c1de5a804ab47bf3507c686fbbdef67cd276ce116758b5f"
+    }
+  },
+  "environment": {
+    "machine": "Apple M5 Pro",
+    "os": "27.0",
+    "python": "3.11.11",
+    "torch": "2.7.0",
+    "coremltools": "9.0",
+    "torch_cpu_threads": 2
+  },
+  "rows": 196,
+  "upstream": {
+    "correct": 196,
+    "accuracy": 1.0,
+    "inference_latency_ms": {
+      "count": 196,
+      "median_ms": 2.572771001723595,
+      "p95_ms": 3.2190517595154233,
+      "min_ms": 1.7389580025337636,
+      "max_ms": 4.086540982825682
+    }
+  },
+  "export_adapter_fp32": {
+    "argmax_agreement": 196,
+    "max_abs_probability_error": 1.1324882507324219e-06
+  },
+  "host_preprocessing_latency_ms": {
+    "count": 196,
+    "median_ms": 0.041624996811151505,
+    "p95_ms": 0.052635499741882086,
+    "min_ms": 0.029125018045306206,
+    "max_ms": 0.08079098188318312
+  },
+  "thresholds": {
+    "argmax_agreement": 1.0,
+    "max_abs_probability_error": 0.005,
+    "export_adapter_max_abs_probability_error": 0.0001,
+    "allow_accuracy_loss": false
+  },
+  "backends": {
+    "CPU_AND_NE": {
+      "passed": true,
+      "correct": 196,
+      "accuracy": 1.0,
+      "argmax_agreement": 196,
+      "max_abs_probability_error": 0.00233614444732666,
+      "mean_row_max_abs_probability_error": 2.934475840582507e-05,
+      "max_abs_logit_error": 0.07230281829833984,
+      "load_seconds": 0.6436757920018863,
+      "warm_inference_latency_ms": {
+        "count": 196,
+        "median_ms": 0.9285830019507557,
+        "p95_ms": 0.9653437373344786,
+        "min_ms": 0.8930829935707152,
+        "max_ms": 1.0020830086432397
+      },
+      "reversed_option_order": {
+        "rows": 6,
+        "max_abs_probability_error": 3.1828880310058594e-05
+      },
+      "per_action": {
+        "fill": {
+          "rows": 36,
+          "correct": 36
+        },
+        "skip": {
+          "rows": 150,
+          "correct": 150
+        },
+        "check": {
+          "rows": 4,
+          "correct": 4
+        },
+        "click": {
+          "rows": 6,
+          "correct": 6
+        }
+      },
+      "decisions": [
+        {
+          "row": 0,
+          "label": 22,
+          "upstream": 22,
+          "coreml": 22,
+          "max_abs_probability_error": 3.1828880310058594e-05,
+          "max_abs_logit_error": 0.03086566925048828
+        },
+        {
+          "row": 1,
+          "label": 23,
+          "upstream": 23,
+          "coreml": 23,
+          "max_abs_probability_error": 3.814697265625e-05,
+          "max_abs_logit_error": 0.020841598510742188
+        },
+        {
+          "row": 2,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.024377822875976562
+        },
+        {
+          "row": 3,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 3.218650817871094e-05,
+          "max_abs_logit_error": 0.030775785446166992
+        },
+        {
+          "row": 4,
+          "label": 4,
+          "upstream": 4,
+          "coreml": 4,
+          "max_abs_probability_error": 2.944469451904297e-05,
+          "max_abs_logit_error": 0.03573417663574219
+        },
+        {
+          "row": 5,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 1.2636184692382812e-05,
+          "max_abs_logit_error": 0.026729822158813477
+        },
+        {
+          "row": 6,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 2.3365020751953125e-05,
+          "max_abs_logit_error": 0.023685455322265625
+        },
+        {
+          "row": 7,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 0.00014638900756835938,
+          "max_abs_logit_error": 0.04038810729980469
+        },
+        {
+          "row": 8,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 1.1920928955078125e-06,
+          "max_abs_logit_error": 0.02460002899169922
+        },
+        {
+          "row": 9,
+          "label": 10,
+          "upstream": 10,
+          "coreml": 10,
+          "max_abs_probability_error": 2.1457672119140625e-05,
+          "max_abs_logit_error": 0.018064022064208984
+        },
+        {
+          "row": 10,
+          "label": 11,
+          "upstream": 11,
+          "coreml": 11,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 0.047691673040390015
+        },
+        {
+          "row": 11,
+          "label": 13,
+          "upstream": 13,
+          "coreml": 13,
+          "max_abs_probability_error": 3.2186508178710938e-06,
+          "max_abs_logit_error": 0.03340768814086914
+        },
+        {
+          "row": 12,
+          "label": 15,
+          "upstream": 15,
+          "coreml": 15,
+          "max_abs_probability_error": 2.384185791015625e-06,
+          "max_abs_logit_error": 0.053191184997558594
+        },
+        {
+          "row": 13,
+          "label": 16,
+          "upstream": 16,
+          "coreml": 16,
+          "max_abs_probability_error": 0.0005360841751098633,
+          "max_abs_logit_error": 0.03362560272216797
+        },
+        {
+          "row": 14,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8596649169921875e-05,
+          "max_abs_logit_error": 0.024669170379638672
+        },
+        {
+          "row": 15,
+          "label": 24,
+          "upstream": 24,
+          "coreml": 24,
+          "max_abs_probability_error": 1.6689300537109375e-05,
+          "max_abs_logit_error": 0.034501075744628906
+        },
+        {
+          "row": 16,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.440092489701783e-08,
+          "max_abs_logit_error": 0.033290743827819824
+        },
+        {
+          "row": 17,
+          "label": 25,
+          "upstream": 25,
+          "coreml": 25,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.03323173522949219
+        },
+        {
+          "row": 18,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 8.430758313693332e-09,
+          "max_abs_logit_error": 0.022179126739501953
+        },
+        {
+          "row": 19,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.0266471551243512e-09,
+          "max_abs_logit_error": 0.029880523681640625
+        },
+        {
+          "row": 20,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4500966588902884e-08,
+          "max_abs_logit_error": 0.031145095825195312
+        },
+        {
+          "row": 21,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.647379281599637e-11,
+          "max_abs_logit_error": 0.03861522674560547
+        },
+        {
+          "row": 22,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.027164459228515625
+        },
+        {
+          "row": 23,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 0.058716535568237305
+        },
+        {
+          "row": 24,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.5055844551924444e-11,
+          "max_abs_logit_error": 0.035381317138671875
+        },
+        {
+          "row": 25,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.995705792434137e-09,
+          "max_abs_logit_error": 0.04862499237060547
+        },
+        {
+          "row": 26,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.470584068551943e-10,
+          "max_abs_logit_error": 0.029034733772277832
+        },
+        {
+          "row": 27,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.973024935241767e-11,
+          "max_abs_logit_error": 0.03808116912841797
+        },
+        {
+          "row": 28,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.994321927715873e-09,
+          "max_abs_logit_error": 0.03319191932678223
+        },
+        {
+          "row": 29,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4426039316183257e-11,
+          "max_abs_logit_error": 0.030045509338378906
+        },
+        {
+          "row": 30,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.026231765747070312
+        },
+        {
+          "row": 31,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.78299757975276e-10,
+          "max_abs_logit_error": 0.033112093806266785
+        },
+        {
+          "row": 32,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.185003176284454e-09,
+          "max_abs_logit_error": 0.022128582000732422
+        },
+        {
+          "row": 33,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.4556297823229443e-09,
+          "max_abs_logit_error": 0.038306236267089844
+        },
+        {
+          "row": 34,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 0.0004820823669433594,
+          "max_abs_logit_error": 0.032683372497558594
+        },
+        {
+          "row": 35,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.462750809366753e-09,
+          "max_abs_logit_error": 0.029730796813964844
+        },
+        {
+          "row": 36,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.7510341227277024e-10,
+          "max_abs_logit_error": 0.02333354949951172
+        },
+        {
+          "row": 37,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.657592190757896e-08,
+          "max_abs_logit_error": 0.022855758666992188
+        },
+        {
+          "row": 38,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 9.552543478452691e-11,
+          "max_abs_logit_error": 0.023164749145507812
+        },
+        {
+          "row": 39,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 4.04716482549361e-10,
+          "max_abs_logit_error": 0.02880859375
+        },
+        {
+          "row": 40,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 0.00021719932556152344,
+          "max_abs_logit_error": 0.04735088348388672
+        },
+        {
+          "row": 41,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.146233744175333e-09,
+          "max_abs_logit_error": 0.03671276569366455
+        },
+        {
+          "row": 42,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8819494851385343e-09,
+          "max_abs_logit_error": 0.023815155029296875
+        },
+        {
+          "row": 43,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.02039623260498047
+        },
+        {
+          "row": 44,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.780525865635596e-10,
+          "max_abs_logit_error": 0.022927284240722656
+        },
+        {
+          "row": 45,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.414331317674879e-11,
+          "max_abs_logit_error": 0.030417919158935547
+        },
+        {
+          "row": 46,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.599257724142603e-10,
+          "max_abs_logit_error": 0.025608062744140625
+        },
+        {
+          "row": 47,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.212314607705437e-11,
+          "max_abs_logit_error": 0.022733211517333984
+        },
+        {
+          "row": 48,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.8596649169921875e-05,
+          "max_abs_logit_error": 0.024669170379638672
+        },
+        {
+          "row": 49,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.932842080831776e-10,
+          "max_abs_logit_error": 0.030317306518554688
+        },
+        {
+          "row": 50,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.440092489701783e-08,
+          "max_abs_logit_error": 0.033290743827819824
+        },
+        {
+          "row": 51,
+          "label": 25,
+          "upstream": 25,
+          "coreml": 25,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.03323173522949219
+        },
+        {
+          "row": 52,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 8.430758313693332e-09,
+          "max_abs_logit_error": 0.022179126739501953
+        },
+        {
+          "row": 53,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.0266471551243512e-09,
+          "max_abs_logit_error": 0.029880523681640625
+        },
+        {
+          "row": 54,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4500966588902884e-08,
+          "max_abs_logit_error": 0.031145095825195312
+        },
+        {
+          "row": 55,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 1.647379281599637e-11,
+          "max_abs_logit_error": 0.03861522674560547
+        },
+        {
+          "row": 56,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.027164459228515625
+        },
+        {
+          "row": 57,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.960464477539062e-07,
+          "max_abs_logit_error": 0.058716535568237305
+        },
+        {
+          "row": 58,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.5055844551924444e-11,
+          "max_abs_logit_error": 0.035381317138671875
+        },
+        {
+          "row": 59,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.995705792434137e-09,
+          "max_abs_logit_error": 0.04862499237060547
+        },
+        {
+          "row": 60,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.470584068551943e-10,
+          "max_abs_logit_error": 0.029034733772277832
+        },
+        {
+          "row": 61,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.973024935241767e-11,
+          "max_abs_logit_error": 0.03808116912841797
+        },
+        {
+          "row": 62,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 5.994321927715873e-09,
+          "max_abs_logit_error": 0.03319191932678223
+        },
+        {
+          "row": 63,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.4426039316183257e-11,
+          "max_abs_logit_error": 0.030045509338378906
+        },
+        {
+          "row": 64,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.026231765747070312
+        },
+        {
+          "row": 65,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 7.78299757975276e-10,
+          "max_abs_logit_error": 0.033112093806266785
+        },
+        {
+          "row": 66,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 6.185003176284454e-09,
+          "max_abs_logit_error": 0.022128582000732422
+        },
+        {
+          "row": 67,
+          "label": 26,
+          "upstream": 26,
+          "coreml": 26,
+          "max_abs_probability_error": 3.4556297823229443e-09,
+          "max_abs_logit_error": 0.038306236267089844
+        },
+        {
+          "row": 68,
+          "label": 0,
+          "upstream": 0,
+          "coreml": 0,
+          "max_abs_probability_error": 3.147125244140625e-05,
+          "max_abs_logit_error": 0.03265953063964844
+        },
+        {
+          "row": 69,
+          "label": 1,
+          "upstream": 1,
+          "coreml": 1,
+          "max_abs_probability_error": 5.030632019042969e-05,
+          "max_abs_logit_error": 0.038794517517089844
+        },
+        {
+          "row": 70,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.040470123291015625
+        },
+        {
+          "row": 71,
+          "label": 3,
+          "upstream": 3,
+          "coreml": 3,
+          "max_abs_probability_error": 5.2928924560546875e-05,
+          "max_abs_logit_error": 0.040035247802734375
+        },
+        {
+          "row": 72,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 4.649162292480469e-06,
+          "max_abs_logit_error": 0.042186737060546875
+        },
+        {
+          "row": 73,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 2.1576881408691406e-05,
+          "max_abs_logit_error": 0.033161163330078125
+        },
+        {
+          "row": 74,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 4.470348358154297e-05,
+          "max_abs_logit_error": 0.0293731689453125
+        },
+        {
+          "row": 75,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 1.1563301086425781e-05,
+          "max_abs_logit_error": 0.03249359130859375
+        },
+        {
+          "row": 76,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 2.86102294921875e-06,
+          "max_abs_logit_error": 0.03498554229736328
+        },
+        {
+          "row": 77,
+          "label": 13,
+          "upstream": 13,
+          "coreml": 13,
+          "max_abs_probability_error": 0.0003635883331298828,
+          "max_abs_logit_error": 0.023473739624023438
+        },
+        {
+          "row": 78,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.034421443939208984
+        },
+        {
+          "row": 79,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 0.0003609657287597656,
+          "max_abs_logit_error": 0.03937721252441406
+        },
+        {
+          "row": 80,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.03752422332763672
+        },
+        {
+          "row": 81,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.025417327880859375
+        },
+        {
+          "row": 82,
+          "label": 19,
+          "upstream": 19,
+          "coreml": 19,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.04453086853027344
+        },
+        {
+          "row": 83,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.0942090545902374e-09,
+          "max_abs_logit_error": 0.023950576782226562
+        },
+        {
+          "row": 84,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.022706031799316406
+        },
+        {
+          "row": 85,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.5405053888659666e-10,
+          "max_abs_logit_error": 0.035375118255615234
+        },
+        {
+          "row": 86,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.76837158203125e-07,
+          "max_abs_logit_error": 0.03717994689941406
+        },
+        {
+          "row": 87,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.04058122634887695
+        },
+        {
+          "row": 88,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3188037328859537e-09,
+          "max_abs_logit_error": 0.032048702239990234
+        },
+        {
+          "row": 89,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9970605169561395e-09,
+          "max_abs_logit_error": 0.030427932739257812
+        },
+        {
+          "row": 90,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7617375336342889e-09,
+          "max_abs_logit_error": 0.0281219482421875
+        },
+        {
+          "row": 91,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.612549260787688e-10,
+          "max_abs_logit_error": 0.030226707458496094
+        },
+        {
+          "row": 92,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.506330630512425e-12,
+          "max_abs_logit_error": 0.04007530212402344
+        },
+        {
+          "row": 93,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.5150824594911683e-08,
+          "max_abs_logit_error": 0.02728891372680664
+        },
+        {
+          "row": 94,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3666674458789885e-09,
+          "max_abs_logit_error": 0.0217437744140625
+        },
+        {
+          "row": 95,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.969476136678196e-10,
+          "max_abs_logit_error": 0.03214454650878906
+        },
+        {
+          "row": 96,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.657726538846106e-10,
+          "max_abs_logit_error": 0.03669023513793945
+        },
+        {
+          "row": 97,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.019012451171875
+        },
+        {
+          "row": 98,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.026651382446289062
+        },
+        {
+          "row": 99,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.1457672119140625e-06,
+          "max_abs_logit_error": 0.04161381721496582
+        },
+        {
+          "row": 100,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.04680633544921875
+        },
+        {
+          "row": 101,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.789079107084035e-08,
+          "max_abs_logit_error": 0.029542922973632812
+        },
+        {
+          "row": 102,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.5862225694339713e-09,
+          "max_abs_logit_error": 0.023795127868652344
+        },
+        {
+          "row": 103,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 0.00024586915969848633,
+          "max_abs_logit_error": 0.030111312866210938
+        },
+        {
+          "row": 104,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.841255187988281e-06,
+          "max_abs_logit_error": 0.03126239776611328
+        },
+        {
+          "row": 105,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.928799035575082e-10,
+          "max_abs_logit_error": 0.026149272918701172
+        },
+        {
+          "row": 106,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.4136979082101107e-09,
+          "max_abs_logit_error": 0.024585723876953125
+        },
+        {
+          "row": 107,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.302839361538304e-10,
+          "max_abs_logit_error": 0.024765491485595703
+        },
+        {
+          "row": 108,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.03492164611816406
+        },
+        {
+          "row": 109,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.034421443939208984
+        },
+        {
+          "row": 110,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 9.5367431640625e-07,
+          "max_abs_logit_error": 0.0320439338684082
+        },
+        {
+          "row": 111,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.811981201171875e-05,
+          "max_abs_logit_error": 0.032814979553222656
+        },
+        {
+          "row": 112,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.025417327880859375
+        },
+        {
+          "row": 113,
+          "label": 19,
+          "upstream": 19,
+          "coreml": 19,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.04453086853027344
+        },
+        {
+          "row": 114,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 5.0942090545902374e-09,
+          "max_abs_logit_error": 0.023950576782226562
+        },
+        {
+          "row": 115,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.022706031799316406
+        },
+        {
+          "row": 116,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.5405053888659666e-10,
+          "max_abs_logit_error": 0.035375118255615234
+        },
+        {
+          "row": 117,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 4.76837158203125e-07,
+          "max_abs_logit_error": 0.03717994689941406
+        },
+        {
+          "row": 118,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.04058122634887695
+        },
+        {
+          "row": 119,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3188037328859537e-09,
+          "max_abs_logit_error": 0.032048702239990234
+        },
+        {
+          "row": 120,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9970605169561395e-09,
+          "max_abs_logit_error": 0.030427932739257812
+        },
+        {
+          "row": 121,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.7617375336342889e-09,
+          "max_abs_logit_error": 0.0281219482421875
+        },
+        {
+          "row": 122,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.612549260787688e-10,
+          "max_abs_logit_error": 0.030226707458496094
+        },
+        {
+          "row": 123,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 6.506330630512425e-12,
+          "max_abs_logit_error": 0.04007530212402344
+        },
+        {
+          "row": 124,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.5150824594911683e-08,
+          "max_abs_logit_error": 0.02728891372680664
+        },
+        {
+          "row": 125,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.3666674458789885e-09,
+          "max_abs_logit_error": 0.0217437744140625
+        },
+        {
+          "row": 126,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 3.969476136678196e-10,
+          "max_abs_logit_error": 0.03214454650878906
+        },
+        {
+          "row": 127,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.657726538846106e-10,
+          "max_abs_logit_error": 0.03669023513793945
+        },
+        {
+          "row": 128,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.019012451171875
+        },
+        {
+          "row": 129,
+          "label": 20,
+          "upstream": 20,
+          "coreml": 20,
+          "max_abs_probability_error": 2.384185791015625e-07,
+          "max_abs_logit_error": 0.026651382446289062
+        },
+        {
+          "row": 130,
+          "label": 1,
+          "upstream": 1,
+          "coreml": 1,
+          "max_abs_probability_error": 3.170967102050781e-05,
+          "max_abs_logit_error": 0.071075439453125
+        },
+        {
+          "row": 131,
+          "label": 2,
+          "upstream": 2,
+          "coreml": 2,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.051357269287109375
+        },
+        {
+          "row": 132,
+          "label": 3,
+          "upstream": 3,
+          "coreml": 3,
+          "max_abs_probability_error": 2.7894973754882812e-05,
+          "max_abs_logit_error": 0.033801641315221786
+        },
+        {
+          "row": 133,
+          "label": 4,
+          "upstream": 4,
+          "coreml": 4,
+          "max_abs_probability_error": 1.0967254638671875e-05,
+          "max_abs_logit_error": 0.039325714111328125
+        },
+        {
+          "row": 134,
+          "label": 5,
+          "upstream": 5,
+          "coreml": 5,
+          "max_abs_probability_error": 6.258487701416016e-05,
+          "max_abs_logit_error": 0.033056676387786865
+        },
+        {
+          "row": 135,
+          "label": 6,
+          "upstream": 6,
+          "coreml": 6,
+          "max_abs_probability_error": 1.2993812561035156e-05,
+          "max_abs_logit_error": 0.02561807632446289
+        },
+        {
+          "row": 136,
+          "label": 7,
+          "upstream": 7,
+          "coreml": 7,
+          "max_abs_probability_error": 6.198883056640625e-06,
+          "max_abs_logit_error": 0.03448677062988281
+        },
+        {
+          "row": 137,
+          "label": 8,
+          "upstream": 8,
+          "coreml": 8,
+          "max_abs_probability_error": 0.00233614444732666,
+          "max_abs_logit_error": 0.04230833053588867
+        },
+        {
+          "row": 138,
+          "label": 9,
+          "upstream": 9,
+          "coreml": 9,
+          "max_abs_probability_error": 0.00022917985916137695,
+          "max_abs_logit_error": 0.05339241027832031
+        },
+        {
+          "row": 139,
+          "label": 10,
+          "upstream": 10,
+          "coreml": 10,
+          "max_abs_probability_error": 4.851818084716797e-05,
+          "max_abs_logit_error": 0.032756805419921875
+        },
+        {
+          "row": 140,
+          "label": 11,
+          "upstream": 11,
+          "coreml": 11,
+          "max_abs_probability_error": 3.0994415283203125e-06,
+          "max_abs_logit_error": 0.029809951782226562
+        },
+        {
+          "row": 141,
+          "label": 12,
+          "upstream": 12,
+          "coreml": 12,
+          "max_abs_probability_error": 9.34600830078125e-05,
+          "max_abs_logit_error": 0.026587963104248047
+        },
+        {
+          "row": 142,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.03671073913574219
+        },
+        {
+          "row": 143,
+          "label": 16,
+          "upstream": 16,
+          "coreml": 16,
+          "max_abs_probability_error": 7.152557373046875e-07,
+          "max_abs_logit_error": 0.05281543731689453
+        },
+        {
+          "row": 144,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.1920928955078125e-06,
+          "max_abs_logit_error": 0.04511451721191406
+        },
+        {
+          "row": 145,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.6954879760742188e-06,
+          "max_abs_logit_error": 0.07230281829833984
+        },
+        {
+          "row": 146,
+          "label": 17,
+          "upstream": 17,
+          "coreml": 17,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.03114461898803711
+        },
+        {
+          "row": 147,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.429776876539563e-09,
+          "max_abs_logit_error": 0.020772457122802734
+        },
+        {
+          "row": 148,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 9.215698426601193e-09,
+          "max_abs_logit_error": 0.03475606441497803
+        },
+        {
+          "row": 149,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.84208304060607e-10,
+          "max_abs_logit_error": 0.020298004150390625
+        },
+        {
+          "row": 150,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.947568818333917e-12,
+          "max_abs_logit_error": 0.04283332824707031
+        },
+        {
+          "row": 151,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.291168614363073e-10,
+          "max_abs_logit_error": 0.028450965881347656
+        },
+        {
+          "row": 152,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.924417788629512e-10,
+          "max_abs_logit_error": 0.02250051498413086
+        },
+        {
+          "row": 153,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.837478791344438e-08,
+          "max_abs_logit_error": 0.06335592269897461
+        },
+        {
+          "row": 154,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.3051329034750054e-11,
+          "max_abs_logit_error": 0.02735137939453125
+        },
+        {
+          "row": 155,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.650468511860126e-10,
+          "max_abs_logit_error": 0.015047073364257812
+        },
+        {
+          "row": 156,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.181078218703284e-12,
+          "max_abs_logit_error": 0.04868888854980469
+        },
+        {
+          "row": 157,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.6455586893115992e-10,
+          "max_abs_logit_error": 0.026497364044189453
+        },
+        {
+          "row": 158,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.109900363426846e-10,
+          "max_abs_logit_error": 0.022236347198486328
+        },
+        {
+          "row": 159,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.693580800742893e-09,
+          "max_abs_logit_error": 0.020044326782226562
+        },
+        {
+          "row": 160,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.3718365998727222e-08,
+          "max_abs_logit_error": 0.05533123016357422
+        },
+        {
+          "row": 161,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.0247042179107666
+        },
+        {
+          "row": 162,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.042114572413766e-08,
+          "max_abs_logit_error": 0.028873443603515625
+        },
+        {
+          "row": 163,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9131388140358752e-10,
+          "max_abs_logit_error": 0.015494346618652344
+        },
+        {
+          "row": 164,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0254025184508464e-08,
+          "max_abs_logit_error": 0.02497100830078125
+        },
+        {
+          "row": 165,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.15840287151903e-11,
+          "max_abs_logit_error": 0.02679443359375
+        },
+        {
+          "row": 166,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.6797479790729994e-09,
+          "max_abs_logit_error": 0.028203964233398438
+        },
+        {
+          "row": 167,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.54509399403247e-10,
+          "max_abs_logit_error": 0.03371429443359375
+        },
+        {
+          "row": 168,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.028280355036258698
+        },
+        {
+          "row": 169,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 2.4169340140378637e-11,
+          "max_abs_logit_error": 0.038245439529418945
+        },
+        {
+          "row": 170,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.0329603412093675e-11,
+          "max_abs_logit_error": 0.030043363571166992
+        },
+        {
+          "row": 171,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.7881393432617188e-06,
+          "max_abs_logit_error": 0.02963542938232422
+        },
+        {
+          "row": 172,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 4.66337235494052e-09,
+          "max_abs_logit_error": 0.03510093688964844
+        },
+        {
+          "row": 173,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.2885730260592254e-09,
+          "max_abs_logit_error": 0.04009199142456055
+        },
+        {
+          "row": 174,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.1279766010119374e-09,
+          "max_abs_logit_error": 0.031810760498046875
+        },
+        {
+          "row": 175,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.9073486328125e-06,
+          "max_abs_logit_error": 0.03671073913574219
+        },
+        {
+          "row": 176,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.0728836059570312e-06,
+          "max_abs_logit_error": 0.05123615264892578
+        },
+        {
+          "row": 177,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.2799782567185503e-08,
+          "max_abs_logit_error": 0.03157186508178711
+        },
+        {
+          "row": 178,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.6954879760742188e-06,
+          "max_abs_logit_error": 0.07230281829833984
+        },
+        {
+          "row": 179,
+          "label": 17,
+          "upstream": 17,
+          "coreml": 17,
+          "max_abs_probability_error": 1.1920928955078125e-07,
+          "max_abs_logit_error": 0.03114461898803711
+        },
+        {
+          "row": 180,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.429776876539563e-09,
+          "max_abs_logit_error": 0.020772457122802734
+        },
+        {
+          "row": 181,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 9.215698426601193e-09,
+          "max_abs_logit_error": 0.03475606441497803
+        },
+        {
+          "row": 182,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.84208304060607e-10,
+          "max_abs_logit_error": 0.020298004150390625
+        },
+        {
+          "row": 183,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.947568818333917e-12,
+          "max_abs_logit_error": 0.04283332824707031
+        },
+        {
+          "row": 184,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 5.291168614363073e-10,
+          "max_abs_logit_error": 0.028450965881347656
+        },
+        {
+          "row": 185,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 7.924417788629512e-10,
+          "max_abs_logit_error": 0.02250051498413086
+        },
+        {
+          "row": 186,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.837478791344438e-08,
+          "max_abs_logit_error": 0.06335592269897461
+        },
+        {
+          "row": 187,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 3.3051329034750054e-11,
+          "max_abs_logit_error": 0.02735137939453125
+        },
+        {
+          "row": 188,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.650468511860126e-10,
+          "max_abs_logit_error": 0.015047073364257812
+        },
+        {
+          "row": 189,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.181078218703284e-12,
+          "max_abs_logit_error": 0.04868888854980469
+        },
+        {
+          "row": 190,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.6455586893115992e-10,
+          "max_abs_logit_error": 0.026497364044189453
+        },
+        {
+          "row": 191,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.109900363426846e-10,
+          "max_abs_logit_error": 0.022236347198486328
+        },
+        {
+          "row": 192,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 6.693580800742893e-09,
+          "max_abs_logit_error": 0.020044326782226562
+        },
+        {
+          "row": 193,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.3718365998727222e-08,
+          "max_abs_logit_error": 0.05533123016357422
+        },
+        {
+          "row": 194,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 8.344650268554688e-07,
+          "max_abs_logit_error": 0.0247042179107666
+        },
+        {
+          "row": 195,
+          "label": 18,
+          "upstream": 18,
+          "coreml": 18,
+          "max_abs_probability_error": 1.042114572413766e-08,
+          "max_abs_logit_error": 0.028873443603515625
+        }
+      ]
+    }
+  },
+  "passed": true,
+  "code_sha256": {
+    "ane_gather.py": "c3379a3797041dcfed82eaaea3990771ec91f1dff7ec146b025988ad7a8c9aae",
+    "assets.py": "f7a5bc931075237e00c73bf21fee3575be35531f934f23b5e22627d50a79f92f",
+    "benchmark-synthetic.py": "ba8255fd3f63d87ef340c26c4c251800f2e20878d80858ad1e6557cf59f249a2",
+    "compare-ane.py": "4633df7ad5f90d6aed37d5ef4f7a2d2231fdadd272ef1aa3057c237b2f937df8",
+    "convert-coreml.py": "46fe0194df924033126286e601214b6789f5d667aae38c153cf4dbc1b0e629b0",
+    "export-reference.py": "ff9eab4d6e4e72ffaf074d82610f41fbea1e802da7c0e987ce562a89eb561b99",
+    "export_model.py": "c61da5d678c77030d1e9a78359626063ea31da46fb6434e49517fdeaa7800949",
+    "prepare-int4-source.py": "33db3295450428d90c1de5a804ab47bf3507c686fbbdef67cd276ce116758b5f",
+    "preprocessing.py": "18c74ede43f95f91cd638c7e9631923a353e3a24d3b522ddb8b18b480f9c2846",
+    "profile-coreml.py": "d5e1386f54fc6c482647120e593e7ffbbe4bbb277c2cc6d4949154660fe239fd",
+    "quantize-int4.py": "a91636e539d8934253b5e7628ca706346fbee538fa1aa9e3eaf794f440d49dcc",
+    "quantize-int8.py": "22071babfcb02ef77225203a479d9bb14b9fa35462ba38bcc4bf07d29c125a97",
+    "score-report.py": "c78e45caa86fb59f9a56ef18cc6a35020ea6e281a6a23d23618f56da85a0d95e",
+    "synthetic_test.py": "e2b68725236e6827eb422480c8cc8aa189b2490c1594650d8f2928771d1bc626",
+    "verify.py": "bc20ec9df47708e9b48e411076abe4eab7c18313f67e11334de2d9e3ecb3efa7"
+  }
+}

reports/int4-synthetic-test-decisions.jsonl.gz ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6922913041f95ac8872958b92312cb5b869b9ce88116c4684b36ea7753284674
+size 2410463

reports/int4-synthetic-test.json ADDED Viewed

The diff for this file is too large to render. See raw diff