Bind R4 bounded ZIP staging, question recovery and exact qualified image
Browse files- EVALUATION.json +0 -0
- EVALUATION.md +3 -3
- IMAGE_IDENTITY.json +58 -24
- LINEAGE.json +61 -27
- LINEAGE.md +9 -0
- LOAD.md +1 -1
- PLATFORM_COMPATIBILITY.json +0 -0
- PLATFORM_COMPATIBILITY.md +5 -5
- README.md +3 -3
- REQUIRED_LINKS.json +5 -2
- platform_wrapper/CONTRACT.json +18 -4
- platform_wrapper/bootstrap.py +269 -18
- platform_wrapper/probe_entrypoint.py +1118 -0
- platform_wrapper/segment_zip_input.py +345 -0
EVALUATION.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
EVALUATION.md
CHANGED
|
@@ -1,12 +1,12 @@
|
|
| 1 |
# SEGMENT evaluation evidence
|
| 2 |
|
| 3 |
-
**
|
| 4 |
|
| 5 |
The completed development comparison is between two registered serving implementations under the same fixed Qwen3.5-27B judge and 20-case batching protocol: candidate 2373/4000 (59.325%) versus named reference 1940/4000 (48.500%), +10.825 percentage points. Strict sensitivity is 2334/3940 versus 1906/3940. All ten original-source-video differences are positive; 5b causal consequence reasoning regressed 17.742 micro points on 62 cases from six sources. These are adaptively used development results with inherited training/data-exposure limits, not an independent holdout, an official finals score or an all-candidate ranking. The registered local scorer passed14 convenience controls (6positive,8negative). These check response/parser boundaries; they do not estimate organizer-judge agreement, real-question false-positive rates, official calibration or a finals score. All 4,000 were not run through a Docker archive. Source-derived routing equality on the 4,000 intervals 1–299s does not by itself transfer these scores to new image bytes.
|
| 6 |
|
| 7 |
-
|
| 8 |
|
| 9 |
-
|
| 10 |
|
| 11 |
Measured grader completions: September 10, 2026, 10:53:44 UTC (candidate) and 10:55:13 UTC (reference). Complete readback and actual GPU-return checks passed for both. Paired reporting completed at 11:07:14 UTC.
|
| 12 |
|
|
|
|
| 1 |
# SEGMENT evaluation evidence
|
| 2 |
|
| 3 |
+
**R4 local runtime, archive and import checks completed; official platform qualification remains unproven.** The exact archive is `e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00` (35,613,293,722 bytes), immutable generation `1789092734891901`. The build manifest is `ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0` and configuration `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4`. Full compressed/raw archive verification covers all 26 ordered DiffIDs; actual same-generation classic Docker import reports config `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4` and no imported manifest digest. The agreed eight cases completed 9 application invocations and 1593 answer executions across 533 unique qIDs. Healthy20 and loose/ZIP500 matched saved content; reverse20 matched the same-image ordinary20, and both ZIP quotas matched the same-image loose500. Canonical repeated calls retained the exact disclosed deferrals. The mixed recovery case blanked only its declared failed questions and preserved later healthy outputs. These are bounded runtime/interface/content checks; no new semantic grade or official final result is claimed.
|
| 4 |
|
| 5 |
The completed development comparison is between two registered serving implementations under the same fixed Qwen3.5-27B judge and 20-case batching protocol: candidate 2373/4000 (59.325%) versus named reference 1940/4000 (48.500%), +10.825 percentage points. Strict sensitivity is 2334/3940 versus 1906/3940. All ten original-source-video differences are positive; 5b causal consequence reasoning regressed 17.742 micro points on 62 cases from six sources. These are adaptively used development results with inherited training/data-exposure limits, not an independent holdout, an official finals score or an all-candidate ranking. The registered local scorer passed14 convenience controls (6positive,8negative). These check response/parser boundaries; they do not estimate organizer-judge agreement, real-question false-positive rates, official calibration or a finals score. All 4,000 were not run through a Docker archive. Source-derived routing equality on the 4,000 intervals 1–299s does not by itself transfer these scores to new image bytes.
|
| 6 |
|
| 7 |
+
R4 retains the image-owned literal SEGMENT task and the original request clocks. It stages only the selected ZIP member for each startup probe or answer, verifies that member through CRC/EOF and SHA-256, closes decoder handles, and removes the lease before continuing. Loose input remains a direct read. Model initialization is still eager. An attributed ordinary per-question exception produces empty content for that qID with measured latency and failure telemetry; successful answers are preserved and later questions continue. Unsafe or ambiguous archives, source mutation, cleanup integrity, required model initialization, environment and fixed-track failures remain fatal. The inherited precise pre-generation rules deferrals remain separately disclosed as degraded coverage. The four changed image files are the bootstrap, contract, probe entrypoint and new ZIP helper; all 51 unchanged inherited serving source files, including cap_fmt, all 50 model/prior assets, prompts, model arithmetic and generation settings remain unchanged.
|
| 8 |
|
| 9 |
+
The full retained ZIP500 was qualified locally at the stated 4 GiB and 16 GiB tmpfs profiles, alongside loose500, healthy20, reversed20, default and persistent canonical calls, and controlled recovery. These fixtures do not establish the platform scratch quota, all possible input sizes, continuous resource peaks, cold-cache requirements, general order invariance, new semantic scores, independent-holdout performance or official platform success. Canonical deferrals and deliberately failed recovery rows are reduced scientific coverage. The prior R3 4 GiB eager-ZIP failure is a conditional local capacity reproduction, not proof of the organizer failure cause. Earlier images and failed attempts remain historical.
|
| 10 |
|
| 11 |
Measured grader completions: September 10, 2026, 10:53:44 UTC (candidate) and 10:55:13 UTC (reference). Complete readback and actual GPU-return checks passed for both. Paired reporting completed at 11:07:14 UTC.
|
| 12 |
|
IMAGE_IDENTITY.json
CHANGED
|
@@ -36,35 +36,38 @@
|
|
| 36 |
"sha256": "2f4dd5055dc7c33c3e9a43c5a443f2946a6cf999f60aa3e453e8d4ce610ac13b"
|
| 37 |
},
|
| 38 |
"current_image": {
|
| 39 |
-
"CPU_acceptance_sha256": "
|
| 40 |
"actual_platform_result_uuid": null,
|
| 41 |
-
"
|
| 42 |
-
"archive_bytes":
|
| 43 |
-
"archive_generation": "
|
| 44 |
-
"archive_receipt_sha256": "
|
| 45 |
-
"archive_sha256": "
|
| 46 |
-
"archive_uri": "gs://heydonto-surgfield-research/submissions/
|
| 47 |
-
"base_archive_sha256": "
|
| 48 |
-
"base_config_sha256": "
|
| 49 |
-
"
|
| 50 |
-
"
|
| 51 |
-
"
|
| 52 |
-
"
|
| 53 |
-
"
|
|
|
|
| 54 |
"cap_fmt_inferrer_sha256": "2861921a562c19937e38346aec219114e64e419bc7dbc7f1eaee4fcaf7f20713",
|
| 55 |
-
"contract_sha256": "
|
| 56 |
-
"docker_config_bytes":
|
| 57 |
-
"docker_config_sha256": "
|
| 58 |
-
"docker_loaded_image_id": "sha256:
|
| 59 |
"docker_loaded_image_id_kind": "classic_config_digest",
|
| 60 |
-
"layer_count":
|
| 61 |
"normalized_archive_import_manifest_sha256": null,
|
| 62 |
-
"overlay_blob_sha256": "
|
| 63 |
-
"overlay_diff_id": "
|
| 64 |
"platform_qualified": false,
|
| 65 |
-
"
|
| 66 |
-
"
|
| 67 |
-
"
|
|
|
|
|
|
|
| 68 |
},
|
| 69 |
"manifest_identity_note": "Build-store manifest, classic loaded configuration ID and any physically observed imported manifest are separate. Pending fields remain null.",
|
| 70 |
"preserved_asset_manifest_scope": {
|
|
@@ -133,6 +136,37 @@
|
|
| 133 |
"runtime_rehearsal_status": "passed_targeted_derivative_runtime_not_platform",
|
| 134 |
"status": "passed_local_targeted_runtime_and_full_archive_load_not_platform"
|
| 135 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
"schema": "segment-associated-container-identity/2",
|
| 137 |
"shared_serving_implementation": {
|
| 138 |
"asset_revision": "35fc28af1d55a5a9ae8096ff2e2cd2c961aa7119",
|
|
|
|
| 36 |
"sha256": "2f4dd5055dc7c33c3e9a43c5a443f2946a6cf999f60aa3e453e8d4ce610ac13b"
|
| 37 |
},
|
| 38 |
"current_image": {
|
| 39 |
+
"CPU_acceptance_sha256": "64a3e419b9b0ea9d320c0de0d0e3d2359a010dd77119927bdf4060d32094f5c3",
|
| 40 |
"actual_platform_result_uuid": null,
|
| 41 |
+
"all25_ancestor_layers_preserved": true,
|
| 42 |
+
"archive_bytes": 35613293722,
|
| 43 |
+
"archive_generation": "1789092734891901",
|
| 44 |
+
"archive_receipt_sha256": "a1e4f11381d1be8cc62a8d3d9f8317324c99426c87f5b6be561b9b9224c627a8",
|
| 45 |
+
"archive_sha256": "e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00",
|
| 46 |
+
"archive_uri": "gs://heydonto-surgfield-research/submissions/segment_final_recovery_20260911_r1/output/segment-final.tar.gz#1789092734891901",
|
| 47 |
+
"base_archive_sha256": "3f86686527e2695697d7e2958c6b6026abba4dd0ad07918c6ddc3dfc4d39c51d",
|
| 48 |
+
"base_config_sha256": "fa8e31b08886a361c45c80425485b08de0fa3f837b4951a0fab636a18952fe79",
|
| 49 |
+
"base_manifest_sha256": "5752df0b2ac35a0660fd4c5627c59c383ad33ac0b5823dd56647a999ba4af095",
|
| 50 |
+
"base_ordered_layer_count": 25,
|
| 51 |
+
"bootstrap_sha256": "c6846ff32df7e7027a3de520536ee917b0927f1d55915dfcf530fd035d0a35c8",
|
| 52 |
+
"build_receipt_sha256": "9ab1411d614c7142568102381fa3c3db5355e465eec7da708db31d61a31764f8",
|
| 53 |
+
"build_store_manifest_sha256": "ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0",
|
| 54 |
+
"candidate_source_sha256": "6bb01e766b15a196ba78779fd8169dff20a1af2ada0a47db1aa85cefe26fc2c1",
|
| 55 |
"cap_fmt_inferrer_sha256": "2861921a562c19937e38346aec219114e64e419bc7dbc7f1eaee4fcaf7f20713",
|
| 56 |
+
"contract_sha256": "2426fca2beaec028bf506085083f9c0eb681620dd667fb0386ffc41236dcc1de",
|
| 57 |
+
"docker_config_bytes": 28323,
|
| 58 |
+
"docker_config_sha256": "3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4",
|
| 59 |
+
"docker_loaded_image_id": "sha256:3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4",
|
| 60 |
"docker_loaded_image_id_kind": "classic_config_digest",
|
| 61 |
+
"layer_count": 26,
|
| 62 |
"normalized_archive_import_manifest_sha256": null,
|
| 63 |
+
"overlay_blob_sha256": "e0f8eb14b59dd28adacb89945febed19517e4e5268db36c96a6cc17fe4a3681c",
|
| 64 |
+
"overlay_diff_id": "1e3951eb1a4e2a1460649a3b4721adada757d5339f84509dc6c33345d9b21392",
|
| 65 |
"platform_qualified": false,
|
| 66 |
+
"probe_entrypoint_sha256": "40e91b3780417c3503fe14164385aa1d91ab7571265de05962026228f8cf4e33",
|
| 67 |
+
"runtime_rehearsal_receipt_sha256": "8b5daa59751678265917a64d96726115edf01c1b10bc8703bb45ec62df4488f0",
|
| 68 |
+
"runtime_rehearsal_status": "passed_complete_agreed_local_matrix_not_platform",
|
| 69 |
+
"segment_zip_input_sha256": "aef9806482af9b03ce08946c01240183393fbf15c6a2046bad9e00c27a3f58a7",
|
| 70 |
+
"status": "passed_local_complete_runtime_matrix_archive_load_not_platform"
|
| 71 |
},
|
| 72 |
"manifest_identity_note": "Build-store manifest, classic loaded configuration ID and any physically observed imported manifest are separate. Pending fields remain null.",
|
| 73 |
"preserved_asset_manifest_scope": {
|
|
|
|
| 136 |
"runtime_rehearsal_status": "passed_targeted_derivative_runtime_not_platform",
|
| 137 |
"status": "passed_local_targeted_runtime_and_full_archive_load_not_platform"
|
| 138 |
},
|
| 139 |
+
"prior_fixed_track_R3_image": {
|
| 140 |
+
"CPU_acceptance_sha256": "c20a1680bf867569101d04c79910ca1f9c04d67caf08d65729e87e8b9800e030",
|
| 141 |
+
"actual_platform_result_uuid": null,
|
| 142 |
+
"all24_ancestor_layers_preserved": true,
|
| 143 |
+
"archive_bytes": 35613234136,
|
| 144 |
+
"archive_generation": "1789069382415008",
|
| 145 |
+
"archive_receipt_sha256": "2eeeb4daa5f57840ea15c8ab094eaed4b59d56cd666c77da1a562381833337b1",
|
| 146 |
+
"archive_sha256": "3f86686527e2695697d7e2958c6b6026abba4dd0ad07918c6ddc3dfc4d39c51d",
|
| 147 |
+
"archive_uri": "gs://heydonto-surgfield-research/submissions/segment_forward_fixed_track_cloudbuild_20260910_r5/output/segment-final.tar.gz#1789069382415008",
|
| 148 |
+
"base_archive_sha256": "43d4fc4fd7a39e4884a5179176a6b6a436956a146494876783d6287855a82825",
|
| 149 |
+
"base_config_sha256": "f7e22f1822d7a152ec91999dbb639ec63378346ef3f5144723c5b453748f97b9",
|
| 150 |
+
"base_ordered_layer_count": 24,
|
| 151 |
+
"bootstrap_sha256": "4078387df9a46989ae0ab8b25275dc6d43e82afd5459018d3eaa12df1cf0cd76",
|
| 152 |
+
"build_receipt_sha256": "0d360628bdf70daf245d6c0ef1c1e7fe7f7db5f6189a4699672daeb6a94c29fa",
|
| 153 |
+
"build_store_manifest_sha256": "5752df0b2ac35a0660fd4c5627c59c383ad33ac0b5823dd56647a999ba4af095",
|
| 154 |
+
"candidate_source_sha256": "88ae763faa28c7333031876d4c6e6f2f005b19fb9b1362b6d6ffdaf3cd992e37",
|
| 155 |
+
"cap_fmt_inferrer_sha256": "2861921a562c19937e38346aec219114e64e419bc7dbc7f1eaee4fcaf7f20713",
|
| 156 |
+
"contract_sha256": "6af2fb47ced6f8bf0bb9b547c5b6580c1a4663869a6eeceb3e2ab90fe7dcc8b7",
|
| 157 |
+
"docker_config_bytes": 28206,
|
| 158 |
+
"docker_config_sha256": "fa8e31b08886a361c45c80425485b08de0fa3f837b4951a0fab636a18952fe79",
|
| 159 |
+
"docker_loaded_image_id": "sha256:fa8e31b08886a361c45c80425485b08de0fa3f837b4951a0fab636a18952fe79",
|
| 160 |
+
"docker_loaded_image_id_kind": "classic_config_digest",
|
| 161 |
+
"layer_count": 25,
|
| 162 |
+
"normalized_archive_import_manifest_sha256": null,
|
| 163 |
+
"overlay_blob_sha256": "64309f5fa0609fe53792678b90973e41f9ca34490c547dbcae741d7ddd785220",
|
| 164 |
+
"overlay_diff_id": "91029e508c7a21f763ab8bf3eeca620f1b5ba76cd9a6280331f34c13af596ee8",
|
| 165 |
+
"platform_qualified": false,
|
| 166 |
+
"runtime_rehearsal_receipt_sha256": "0a8bc8e0cf2d71bebe9085fefd4b2a968595ad69701ea10f592afccc8c35fcfa",
|
| 167 |
+
"runtime_rehearsal_status": "passed_final_runtime_interface_and_strict_parity_not_platform",
|
| 168 |
+
"status": "passed_local_interface_strict_parity_archive_load_not_platform"
|
| 169 |
+
},
|
| 170 |
"schema": "segment-associated-container-identity/2",
|
| 171 |
"shared_serving_implementation": {
|
| 172 |
"asset_revision": "35fc28af1d55a5a9ae8096ff2e2cd2c961aa7119",
|
LINEAGE.json
CHANGED
|
@@ -49,42 +49,45 @@
|
|
| 49 |
},
|
| 50 |
"attribution": "HeyDonto Labs",
|
| 51 |
"current_serving_image": {
|
| 52 |
-
"CPU_acceptance_sha256": "
|
| 53 |
"actual_platform_result_uuid": null,
|
| 54 |
-
"
|
| 55 |
-
"archive_bytes":
|
| 56 |
-
"archive_generation": "
|
| 57 |
-
"archive_receipt_sha256": "
|
| 58 |
-
"archive_sha256": "
|
| 59 |
-
"archive_uri": "gs://heydonto-surgfield-research/submissions/
|
| 60 |
-
"base_archive_sha256": "
|
| 61 |
-
"base_config_sha256": "
|
| 62 |
-
"
|
| 63 |
-
"
|
| 64 |
-
"
|
| 65 |
-
"
|
| 66 |
-
"
|
|
|
|
| 67 |
"cap_fmt_inferrer_sha256": "2861921a562c19937e38346aec219114e64e419bc7dbc7f1eaee4fcaf7f20713",
|
| 68 |
-
"contract_sha256": "
|
| 69 |
-
"docker_config_bytes":
|
| 70 |
-
"docker_config_sha256": "
|
| 71 |
-
"docker_loaded_image_id": "sha256:
|
| 72 |
"docker_loaded_image_id_kind": "classic_config_digest",
|
| 73 |
-
"layer_count":
|
| 74 |
"normalized_archive_import_manifest_sha256": null,
|
| 75 |
-
"overlay_blob_sha256": "
|
| 76 |
-
"overlay_diff_id": "
|
| 77 |
"platform_qualified": false,
|
| 78 |
-
"
|
| 79 |
-
"
|
| 80 |
-
"
|
|
|
|
|
|
|
| 81 |
},
|
| 82 |
-
"current_serving_image_reference": "IMAGE_IDENTITY.json
|
| 83 |
"derived_annotation_current_scope_clarification": "The known derived packet contains all 13,177 listed extensions: 4,621 paraphrases, 5,217 count remints, 2,317 convention remints and 1,022 adoption rows. Exclusion of the 33,581 organizer rows is intentional, not evidence that those rows must be republished. The unresolved items are the separate private annotation URL/access/delivery receipts and complete signature provenance.",
|
| 84 |
"forward_change": "Only tested frame_sampler; same selected models/priors/precision/rendering/generation.",
|
| 85 |
"forward_gradient_updates": 0,
|
| 86 |
-
"full_algorithm_and_image_qualification": "
|
| 87 |
-
"image_compatibility_change": "
|
| 88 |
"image_identity": {
|
| 89 |
"added_diff_id": "ee65a0fd5c5057739b208abb4bab056ff0b874e40fd9dfa43c18acf1ddbf5e1e",
|
| 90 |
"adoption_sha256": "7e46d6b9ff4472e894fbcb2279e79c8ecc25daf251e0be11b4c47f96c6dff4b2",
|
|
@@ -279,6 +282,37 @@
|
|
| 279 |
"runtime_rehearsal_status": "passed_targeted_derivative_runtime_not_platform",
|
| 280 |
"status": "passed_local_targeted_runtime_and_full_archive_load_not_platform"
|
| 281 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 282 |
"responsible_contact": {
|
| 283 |
"name": "Reza Nehzati, Ph.D.",
|
| 284 |
"profile": "https://huggingface.co/rezanehzati"
|
|
|
|
| 49 |
},
|
| 50 |
"attribution": "HeyDonto Labs",
|
| 51 |
"current_serving_image": {
|
| 52 |
+
"CPU_acceptance_sha256": "64a3e419b9b0ea9d320c0de0d0e3d2359a010dd77119927bdf4060d32094f5c3",
|
| 53 |
"actual_platform_result_uuid": null,
|
| 54 |
+
"all25_ancestor_layers_preserved": true,
|
| 55 |
+
"archive_bytes": 35613293722,
|
| 56 |
+
"archive_generation": "1789092734891901",
|
| 57 |
+
"archive_receipt_sha256": "a1e4f11381d1be8cc62a8d3d9f8317324c99426c87f5b6be561b9b9224c627a8",
|
| 58 |
+
"archive_sha256": "e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00",
|
| 59 |
+
"archive_uri": "gs://heydonto-surgfield-research/submissions/segment_final_recovery_20260911_r1/output/segment-final.tar.gz#1789092734891901",
|
| 60 |
+
"base_archive_sha256": "3f86686527e2695697d7e2958c6b6026abba4dd0ad07918c6ddc3dfc4d39c51d",
|
| 61 |
+
"base_config_sha256": "fa8e31b08886a361c45c80425485b08de0fa3f837b4951a0fab636a18952fe79",
|
| 62 |
+
"base_manifest_sha256": "5752df0b2ac35a0660fd4c5627c59c383ad33ac0b5823dd56647a999ba4af095",
|
| 63 |
+
"base_ordered_layer_count": 25,
|
| 64 |
+
"bootstrap_sha256": "c6846ff32df7e7027a3de520536ee917b0927f1d55915dfcf530fd035d0a35c8",
|
| 65 |
+
"build_receipt_sha256": "9ab1411d614c7142568102381fa3c3db5355e465eec7da708db31d61a31764f8",
|
| 66 |
+
"build_store_manifest_sha256": "ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0",
|
| 67 |
+
"candidate_source_sha256": "6bb01e766b15a196ba78779fd8169dff20a1af2ada0a47db1aa85cefe26fc2c1",
|
| 68 |
"cap_fmt_inferrer_sha256": "2861921a562c19937e38346aec219114e64e419bc7dbc7f1eaee4fcaf7f20713",
|
| 69 |
+
"contract_sha256": "2426fca2beaec028bf506085083f9c0eb681620dd667fb0386ffc41236dcc1de",
|
| 70 |
+
"docker_config_bytes": 28323,
|
| 71 |
+
"docker_config_sha256": "3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4",
|
| 72 |
+
"docker_loaded_image_id": "sha256:3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4",
|
| 73 |
"docker_loaded_image_id_kind": "classic_config_digest",
|
| 74 |
+
"layer_count": 26,
|
| 75 |
"normalized_archive_import_manifest_sha256": null,
|
| 76 |
+
"overlay_blob_sha256": "e0f8eb14b59dd28adacb89945febed19517e4e5268db36c96a6cc17fe4a3681c",
|
| 77 |
+
"overlay_diff_id": "1e3951eb1a4e2a1460649a3b4721adada757d5339f84509dc6c33345d9b21392",
|
| 78 |
"platform_qualified": false,
|
| 79 |
+
"probe_entrypoint_sha256": "40e91b3780417c3503fe14164385aa1d91ab7571265de05962026228f8cf4e33",
|
| 80 |
+
"runtime_rehearsal_receipt_sha256": "8b5daa59751678265917a64d96726115edf01c1b10bc8703bb45ec62df4488f0",
|
| 81 |
+
"runtime_rehearsal_status": "passed_complete_agreed_local_matrix_not_platform",
|
| 82 |
+
"segment_zip_input_sha256": "aef9806482af9b03ce08946c01240183393fbf15c6a2046bad9e00c27a3f58a7",
|
| 83 |
+
"status": "passed_local_complete_runtime_matrix_archive_load_not_platform"
|
| 84 |
},
|
| 85 |
+
"current_serving_image_reference": "IMAGE_IDENTITY.json current_image (R4); all prior image identities are explicitly historical.",
|
| 86 |
"derived_annotation_current_scope_clarification": "The known derived packet contains all 13,177 listed extensions: 4,621 paraphrases, 5,217 count remints, 2,317 convention remints and 1,022 adoption rows. Exclusion of the 33,581 organizer rows is intentional, not evidence that those rows must be republished. The unresolved items are the separate private annotation URL/access/delivery receipts and complete signature provenance.",
|
| 87 |
"forward_change": "Only tested frame_sampler; same selected models/priors/precision/rendering/generation.",
|
| 88 |
"forward_gradient_updates": 0,
|
| 89 |
+
"full_algorithm_and_image_qualification": "R4 local runtime, archive and import checks completed; official platform qualification remains unproven. The exact archive is `e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00` (35,613,293,722 bytes), immutable generation `1789092734891901`. The build manifest is `ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0` and configuration `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4`. Full compressed/raw archive verification covers all 26 ordered DiffIDs; actual same-generation classic Docker import reports config `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4` and no imported manifest digest. The agreed eight cases completed 9 application invocations and 1593 answer executions across 533 unique qIDs. Healthy20 and loose/ZIP500 matched saved content; reverse20 matched the same-image ordinary20, and both ZIP quotas matched the same-image loose500. Canonical repeated calls retained the exact disclosed deferrals. The mixed recovery case blanked only its declared failed questions and preserved later healthy outputs. These are bounded runtime/interface/content checks; no new semantic grade or official final result is claimed. The full retained ZIP500 was qualified locally at the stated 4 GiB and 16 GiB tmpfs profiles, alongside loose500, healthy20, reversed20, default and persistent canonical calls, and controlled recovery. These fixtures do not establish the platform scratch quota, all possible input sizes, continuous resource peaks, cold-cache requirements, general order invariance, new semantic scores, independent-holdout performance or official platform success. Canonical deferrals and deliberately failed recovery rows are reduced scientific coverage. The prior R3 4 GiB eager-ZIP failure is a conditional local capacity reproduction, not proof of the organizer failure cause. Earlier images and failed attempts remain historical.",
|
| 90 |
+
"image_compatibility_change": "R4 retains the image-owned literal SEGMENT task and the original request clocks. It stages only the selected ZIP member for each startup probe or answer, verifies that member through CRC/EOF and SHA-256, closes decoder handles, and removes the lease before continuing. Loose input remains a direct read. Model initialization is still eager. An attributed ordinary per-question exception produces empty content for that qID with measured latency and failure telemetry; successful answers are preserved and later questions continue. Unsafe or ambiguous archives, source mutation, cleanup integrity, required model initialization, environment and fixed-track failures remain fatal. The inherited precise pre-generation rules deferrals remain separately disclosed as degraded coverage. The four changed image files are the bootstrap, contract, probe entrypoint and new ZIP helper; all 51 unchanged inherited serving source files, including cap_fmt, all 50 model/prior assets, prompts, model arithmetic and generation settings remain unchanged.",
|
| 91 |
"image_identity": {
|
| 92 |
"added_diff_id": "ee65a0fd5c5057739b208abb4bab056ff0b874e40fd9dfa43c18acf1ddbf5e1e",
|
| 93 |
"adoption_sha256": "7e46d6b9ff4472e894fbcb2279e79c8ecc25daf251e0be11b4c47f96c6dff4b2",
|
|
|
|
| 282 |
"runtime_rehearsal_status": "passed_targeted_derivative_runtime_not_platform",
|
| 283 |
"status": "passed_local_targeted_runtime_and_full_archive_load_not_platform"
|
| 284 |
},
|
| 285 |
+
"prior_fixed_track_R3_serving_image": {
|
| 286 |
+
"CPU_acceptance_sha256": "c20a1680bf867569101d04c79910ca1f9c04d67caf08d65729e87e8b9800e030",
|
| 287 |
+
"actual_platform_result_uuid": null,
|
| 288 |
+
"all24_ancestor_layers_preserved": true,
|
| 289 |
+
"archive_bytes": 35613234136,
|
| 290 |
+
"archive_generation": "1789069382415008",
|
| 291 |
+
"archive_receipt_sha256": "2eeeb4daa5f57840ea15c8ab094eaed4b59d56cd666c77da1a562381833337b1",
|
| 292 |
+
"archive_sha256": "3f86686527e2695697d7e2958c6b6026abba4dd0ad07918c6ddc3dfc4d39c51d",
|
| 293 |
+
"archive_uri": "gs://heydonto-surgfield-research/submissions/segment_forward_fixed_track_cloudbuild_20260910_r5/output/segment-final.tar.gz#1789069382415008",
|
| 294 |
+
"base_archive_sha256": "43d4fc4fd7a39e4884a5179176a6b6a436956a146494876783d6287855a82825",
|
| 295 |
+
"base_config_sha256": "f7e22f1822d7a152ec91999dbb639ec63378346ef3f5144723c5b453748f97b9",
|
| 296 |
+
"base_ordered_layer_count": 24,
|
| 297 |
+
"bootstrap_sha256": "4078387df9a46989ae0ab8b25275dc6d43e82afd5459018d3eaa12df1cf0cd76",
|
| 298 |
+
"build_receipt_sha256": "0d360628bdf70daf245d6c0ef1c1e7fe7f7db5f6189a4699672daeb6a94c29fa",
|
| 299 |
+
"build_store_manifest_sha256": "5752df0b2ac35a0660fd4c5627c59c383ad33ac0b5823dd56647a999ba4af095",
|
| 300 |
+
"candidate_source_sha256": "88ae763faa28c7333031876d4c6e6f2f005b19fb9b1362b6d6ffdaf3cd992e37",
|
| 301 |
+
"cap_fmt_inferrer_sha256": "2861921a562c19937e38346aec219114e64e419bc7dbc7f1eaee4fcaf7f20713",
|
| 302 |
+
"contract_sha256": "6af2fb47ced6f8bf0bb9b547c5b6580c1a4663869a6eeceb3e2ab90fe7dcc8b7",
|
| 303 |
+
"docker_config_bytes": 28206,
|
| 304 |
+
"docker_config_sha256": "fa8e31b08886a361c45c80425485b08de0fa3f837b4951a0fab636a18952fe79",
|
| 305 |
+
"docker_loaded_image_id": "sha256:fa8e31b08886a361c45c80425485b08de0fa3f837b4951a0fab636a18952fe79",
|
| 306 |
+
"docker_loaded_image_id_kind": "classic_config_digest",
|
| 307 |
+
"layer_count": 25,
|
| 308 |
+
"normalized_archive_import_manifest_sha256": null,
|
| 309 |
+
"overlay_blob_sha256": "64309f5fa0609fe53792678b90973e41f9ca34490c547dbcae741d7ddd785220",
|
| 310 |
+
"overlay_diff_id": "91029e508c7a21f763ab8bf3eeca620f1b5ba76cd9a6280331f34c13af596ee8",
|
| 311 |
+
"platform_qualified": false,
|
| 312 |
+
"runtime_rehearsal_receipt_sha256": "0a8bc8e0cf2d71bebe9085fefd4b2a968595ad69701ea10f592afccc8c35fcfa",
|
| 313 |
+
"runtime_rehearsal_status": "passed_final_runtime_interface_and_strict_parity_not_platform",
|
| 314 |
+
"status": "passed_local_interface_strict_parity_archive_load_not_platform"
|
| 315 |
+
},
|
| 316 |
"responsible_contact": {
|
| 317 |
"name": "Reza Nehzati, Ph.D.",
|
| 318 |
"profile": "https://huggingface.co/rezanehzati"
|
LINEAGE.md
CHANGED
|
@@ -17,3 +17,12 @@ The six inherited prior files remain byte-identical. `cholec_priors_v38.json` ar
|
|
| 17 |
The original forward candidate changed the frame sampler (`cd146570…`). The later R3 compatibility derivative additionally fixes image-owned SEGMENT routing in cap_fmt_inferrer.py and records exactly attributed pre-generation deferrals in its bootstrap; the original timestamps, sampler/model behavior and all other 51 serving files remain unchanged. Public Qwen, private N3, both selected adapters, all six priors, prompts, precision, frame limits and generation settings are unchanged. `TRAINING_RESOURCES.json` preserves selected-save timing/peak scopes and producer identities; `MODEL_IDENTITY.json` preserves every selected file identity.
|
| 18 |
|
| 19 |
The known derived packet contains all 13,177 listed extensions: 4,621 paraphrases, 5,217 count remints, 2,317 convention remints and 1,022 adoption rows. Exclusion of the 33,581 organizer rows is intentional, not evidence that those rows must be republished. The unresolved items are the separate private annotation URL/access/delivery receipts and complete signature provenance.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
The original forward candidate changed the frame sampler (`cd146570…`). The later R3 compatibility derivative additionally fixes image-owned SEGMENT routing in cap_fmt_inferrer.py and records exactly attributed pre-generation deferrals in its bootstrap; the original timestamps, sampler/model behavior and all other 51 serving files remain unchanged. Public Qwen, private N3, both selected adapters, all six priors, prompts, precision, frame limits and generation settings are unchanged. `TRAINING_RESOURCES.json` preserves selected-save timing/peak scopes and producer identities; `MODEL_IDENTITY.json` preserves every selected file identity.
|
| 18 |
|
| 19 |
The known derived packet contains all 13,177 listed extensions: 4,621 paraphrases, 5,217 count remints, 2,317 convention remints and 1,022 adoption rows. Exclusion of the 33,581 organizer rows is intentional, not evidence that those rows must be republished. The unresolved items are the separate private annotation URL/access/delivery receipts and complete signature provenance.
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
## R4 serving compatibility update
|
| 23 |
+
|
| 24 |
+
R4 local runtime, archive and import checks completed; official platform qualification remains unproven. The exact archive is `e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00` (35,613,293,722 bytes), immutable generation `1789092734891901`. The build manifest is `ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0` and configuration `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4`. Full compressed/raw archive verification covers all 26 ordered DiffIDs; actual same-generation classic Docker import reports config `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4` and no imported manifest digest. The agreed eight cases completed 9 application invocations and 1593 answer executions across 533 unique qIDs. Healthy20 and loose/ZIP500 matched saved content; reverse20 matched the same-image ordinary20, and both ZIP quotas matched the same-image loose500. Canonical repeated calls retained the exact disclosed deferrals. The mixed recovery case blanked only its declared failed questions and preserved later healthy outputs. These are bounded runtime/interface/content checks; no new semantic grade or official final result is claimed.
|
| 25 |
+
|
| 26 |
+
R4 retains the image-owned literal SEGMENT task and the original request clocks. It stages only the selected ZIP member for each startup probe or answer, verifies that member through CRC/EOF and SHA-256, closes decoder handles, and removes the lease before continuing. Loose input remains a direct read. Model initialization is still eager. An attributed ordinary per-question exception produces empty content for that qID with measured latency and failure telemetry; successful answers are preserved and later questions continue. Unsafe or ambiguous archives, source mutation, cleanup integrity, required model initialization, environment and fixed-track failures remain fatal. The inherited precise pre-generation rules deferrals remain separately disclosed as degraded coverage. The four changed image files are the bootstrap, contract, probe entrypoint and new ZIP helper; all 51 unchanged inherited serving source files, including cap_fmt, all 50 model/prior assets, prompts, model arithmetic and generation settings remain unchanged.
|
| 27 |
+
|
| 28 |
+
The full retained ZIP500 was qualified locally at the stated 4 GiB and 16 GiB tmpfs profiles, alongside loose500, healthy20, reversed20, default and persistent canonical calls, and controlled recovery. These fixtures do not establish the platform scratch quota, all possible input sizes, continuous resource peaks, cold-cache requirements, general order invariance, new semantic scores, independent-holdout performance or official platform success. Canonical deferrals and deliberately failed recovery rows are reduced scientific coverage. The prior R3 4 GiB eager-ZIP failure is a conditional local capacity reproduction, not proof of the organizer failure cause. Earlier images and failed attempts remain historical.
|
LOAD.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Local loading from two immutable revisions
|
| 2 |
|
| 3 |
-
These revisions reproduce the unchanged **model components and loading tools**. For full serving, the locally qualified
|
| 4 |
|
| 5 |
The metadata download below remains pinned to revision M and excludes its archived `README.md`, `LOAD.md` and `IMAGE_IDENTITY.json`. The retained `MODEL_FILES.json` and lineage/binding documents describe component ancestry and may contain the ed85 ancestor's image identity; those bindings do not select the current platform artifact. The 50 model/prior files, immutable A/M revisions, manifests and component loaders remain unchanged. The fixed wrapper is already inside the current Docker archive and is identified by the current records linked above.
|
| 6 |
|
|
|
|
| 1 |
# Local loading from two immutable revisions
|
| 2 |
|
| 3 |
+
These revisions reproduce the unchanged **model components and loading tools**. For full serving, the locally qualified R4 archive has SHA-256 `e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00`. Its exact GCS generation, configuration, four compatibility-source identities and bounded local runtime evidence are recorded in [IMAGE_IDENTITY.json](IMAGE_IDENTITY.json) and [PLATFORM_COMPATIBILITY.md](PLATFORM_COMPATIBILITY.md) at the repository revision containing this page. Prior archives remain historical. Use those current records to select the serving archive.
|
| 4 |
|
| 5 |
The metadata download below remains pinned to revision M and excludes its archived `README.md`, `LOAD.md` and `IMAGE_IDENTITY.json`. The retained `MODEL_FILES.json` and lineage/binding documents describe component ancestry and may contain the ed85 ancestor's image identity; those bindings do not select the current platform artifact. The 50 model/prior files, immutable A/M revisions, manifests and component loaders remain unchanged. The fixed wrapper is already inside the current Docker archive and is identified by the current records linked above.
|
| 6 |
|
PLATFORM_COMPATIBILITY.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
PLATFORM_COMPATIBILITY.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
# Fixed SEGMENT task and explicit degraded coverage
|
| 2 |
|
| 3 |
-
**
|
| 4 |
|
| 5 |
-
|
| 6 |
|
| 7 |
-
|
| 8 |
|
| 9 |
-
The earlier 43d4 qualification covered six H100 invocations and 600 answer executions across 520 unique questions, including repeated20 cases and a loose500 case. It omitted the owner canonical long intervals and invalid-media control. Its151 positive 500 startup probes left 349 unmeasured; its ZIP coverage was 20, not 500. Those measurements remain historical and do not qualify
|
| 10 |
|
| 11 |
-
The exact
|
|
|
|
| 1 |
# Fixed SEGMENT task and explicit degraded coverage
|
| 2 |
|
| 3 |
+
**R4 local runtime, archive and import checks completed; official platform qualification remains unproven.** The exact archive is `e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00` (35,613,293,722 bytes), immutable generation `1789092734891901`. The build manifest is `ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0` and configuration `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4`. Full compressed/raw archive verification covers all 26 ordered DiffIDs; actual same-generation classic Docker import reports config `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4` and no imported manifest digest. The agreed eight cases completed 9 application invocations and 1593 answer executions across 533 unique qIDs. Healthy20 and loose/ZIP500 matched saved content; reverse20 matched the same-image ordinary20, and both ZIP quotas matched the same-image loose500. Canonical repeated calls retained the exact disclosed deferrals. The mixed recovery case blanked only its declared failed questions and preserved later healthy outputs. These are bounded runtime/interface/content checks; no new semantic grade or official final result is claimed.
|
| 4 |
|
| 5 |
+
R4 retains the image-owned literal SEGMENT task and the original request clocks. It stages only the selected ZIP member for each startup probe or answer, verifies that member through CRC/EOF and SHA-256, closes decoder handles, and removes the lease before continuing. Loose input remains a direct read. Model initialization is still eager. An attributed ordinary per-question exception produces empty content for that qID with measured latency and failure telemetry; successful answers are preserved and later questions continue. Unsafe or ambiguous archives, source mutation, cleanup integrity, required model initialization, environment and fixed-track failures remain fatal. The inherited precise pre-generation rules deferrals remain separately disclosed as degraded coverage. The four changed image files are the bootstrap, contract, probe entrypoint and new ZIP helper; all 51 unchanged inherited serving source files, including cap_fmt, all 50 model/prior assets, prompts, model arithmetic and generation settings remain unchanged.
|
| 6 |
|
| 7 |
+
The full retained ZIP500 was qualified locally at the stated 4 GiB and 16 GiB tmpfs profiles, alongside loose500, healthy20, reversed20, default and persistent canonical calls, and controlled recovery. These fixtures do not establish the platform scratch quota, all possible input sizes, continuous resource peaks, cold-cache requirements, general order invariance, new semantic scores, independent-holdout performance or official platform success. Canonical deferrals and deliberately failed recovery rows are reduced scientific coverage. The prior R3 4 GiB eager-ZIP failure is a conditional local capacity reproduction, not proof of the organizer failure cause. Earlier images and failed attempts remain historical.
|
| 8 |
|
| 9 |
+
The earlier 43d4 qualification covered six H100 invocations and 600 answer executions across 520 unique questions, including repeated20 cases and a loose500 case. It omitted the owner canonical long intervals and invalid-media control. Its151 positive 500 startup probes left 349 unmeasured; its ZIP coverage was 20, not 500. Those measurements remain historical and do not qualify R4. The completed R2 diagnostic prototype returned 10 answers twice with identical contents but failed its guard on two adaptive_n duration-domain deferrals and one no-frame control. That interface diagnosis is not an all-neural-health pass.
|
| 10 |
|
| 11 |
+
The exact four R4 image files are available for inspection under platform_wrapper/. The inherited component_source/cap_fmt_inferrer.py and all component loaders remain unchanged. R4 retains the image-owned literal SEGMENT task and the original request clocks. It stages only the selected ZIP member for each startup probe or answer, verifies that member through CRC/EOF and SHA-256, closes decoder handles, and removes the lease before continuing. Loose input remains a direct read. Model initialization is still eager. An attributed ordinary per-question exception produces empty content for that qID with measured latency and failure telemetry; successful answers are preserved and later questions continue. Unsafe or ambiguous archives, source mutation, cleanup integrity, required model initialization, environment and fixed-track failures remain fatal. The inherited precise pre-generation rules deferrals remain separately disclosed as degraded coverage. The four changed image files are the bootstrap, contract, probe entrypoint and new ZIP helper; all 51 unchanged inherited serving source files, including cap_fmt, all 50 model/prior assets, prompts, model arithmetic and generation settings remain unchanged.
|
README.md
CHANGED
|
@@ -16,7 +16,7 @@ tags:
|
|
| 16 |
---
|
| 17 |
# SURGFIELD ORena2026 SEGMENT FINAL
|
| 18 |
|
| 19 |
-
**
|
| 20 |
|
| 21 |
Private model and resource handoff by **HeyDonto Labs**. Responsible contact: **Reza Nehzati, Ph.D.**, via [rezanehzati](https://huggingface.co/rezanehzati).
|
| 22 |
|
|
@@ -24,11 +24,11 @@ This candidate answers segment-level surgical-video questions through fixed rule
|
|
| 24 |
|
| 25 |
`SERVING_ASSETS.json` lists all **50 required model/prior files**: 22 private assets, 14 pinned public Qwen files, 8 pinned OWLv2 files and 6 priors. `LOAD.md` downloads immutable asset and metadata revisions into separate directories and explains the frozen loading contract. LOAD pins asset revision `35fc28af1d55a5a9ae8096ff2e2cd2c961aa7119` and metadata/tools revision `bc9e092422beac9ae51005694824cdc70205e0d5` separately. The private N3 backbone and public Qwen backbone are distinct; both are required. The inactive OWLv2 `.bin` duplicate and obsolete adapters are excluded.
|
| 26 |
|
| 27 |
-
|
| 28 |
|
| 29 |
The completed development comparison is between two registered serving implementations under the same fixed Qwen3.5-27B judge and 20-case batching protocol: candidate 2373/4000 (59.325%) versus named reference 1940/4000 (48.500%), +10.825 percentage points. Strict sensitivity is 2334/3940 versus 1906/3940. All ten original-source-video differences are positive; 5b causal consequence reasoning regressed 17.742 micro points on 62 cases from six sources. These are adaptively used development results with inherited training/data-exposure limits, not an independent holdout, an official finals score or an all-candidate ranking. The registered local scorer passed14 convenience controls (6positive,8negative). These check response/parser boundaries; they do not estimate organizer-judge agreement, real-question false-positive rates, official calibration or a finals score. All 4,000 were not run through a Docker archive. Source-derived routing equality on the 4,000 intervals 1–299s does not by itself transfer these scores to new image bytes.
|
| 30 |
|
| 31 |
-
|
| 32 |
|
| 33 |
Training lineage and selected-prefix exposure are disclosed in `LINEAGE.md` and the accompanying JSONs. The separately retained annotation packet has 13,177 derived rows, not the complete 46,758-row ancestor corpus; it contains no underlying video pixels or complete signature ledger.
|
| 34 |
|
|
|
|
| 16 |
---
|
| 17 |
# SURGFIELD ORena2026 SEGMENT FINAL
|
| 18 |
|
| 19 |
+
**R4 local runtime, archive and import checks completed; official platform qualification remains unproven.** The exact archive is `e477d307ff2e09f469d83569ec1ce8703daeef34edbba633a3f240eab955de00` (35,613,293,722 bytes), immutable generation `1789092734891901`. The build manifest is `ba9ddb1e78a45040803ae6f0e68c66de7a538f5e3e86397777e5fc2c488106b0` and configuration `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4`. Full compressed/raw archive verification covers all 26 ordered DiffIDs; actual same-generation classic Docker import reports config `3f43fce1081dc5db454447dbe4678947e5cefec014635e611d8d1032a0ac71c4` and no imported manifest digest. The agreed eight cases completed 9 application invocations and 1593 answer executions across 533 unique qIDs. Healthy20 and loose/ZIP500 matched saved content; reverse20 matched the same-image ordinary20, and both ZIP quotas matched the same-image loose500. Canonical repeated calls retained the exact disclosed deferrals. The mixed recovery case blanked only its declared failed questions and preserved later healthy outputs. These are bounded runtime/interface/content checks; no new semantic grade or official final result is claimed.
|
| 20 |
|
| 21 |
Private model and resource handoff by **HeyDonto Labs**. Responsible contact: **Reza Nehzati, Ph.D.**, via [rezanehzati](https://huggingface.co/rezanehzati).
|
| 22 |
|
|
|
|
| 24 |
|
| 25 |
`SERVING_ASSETS.json` lists all **50 required model/prior files**: 22 private assets, 14 pinned public Qwen files, 8 pinned OWLv2 files and 6 priors. `LOAD.md` downloads immutable asset and metadata revisions into separate directories and explains the frozen loading contract. LOAD pins asset revision `35fc28af1d55a5a9ae8096ff2e2cd2c961aa7119` and metadata/tools revision `bc9e092422beac9ae51005694824cdc70205e0d5` separately. The private N3 backbone and public Qwen backbone are distinct; both are required. The inactive OWLv2 `.bin` duplicate and obsolete adapters are excluded.
|
| 26 |
|
| 27 |
+
R4 retains the image-owned literal SEGMENT task and the original request clocks. It stages only the selected ZIP member for each startup probe or answer, verifies that member through CRC/EOF and SHA-256, closes decoder handles, and removes the lease before continuing. Loose input remains a direct read. Model initialization is still eager. An attributed ordinary per-question exception produces empty content for that qID with measured latency and failure telemetry; successful answers are preserved and later questions continue. Unsafe or ambiguous archives, source mutation, cleanup integrity, required model initialization, environment and fixed-track failures remain fatal. The inherited precise pre-generation rules deferrals remain separately disclosed as degraded coverage. The four changed image files are the bootstrap, contract, probe entrypoint and new ZIP helper; all 51 unchanged inherited serving source files, including cap_fmt, all 50 model/prior assets, prompts, model arithmetic and generation settings remain unchanged.
|
| 28 |
|
| 29 |
The completed development comparison is between two registered serving implementations under the same fixed Qwen3.5-27B judge and 20-case batching protocol: candidate 2373/4000 (59.325%) versus named reference 1940/4000 (48.500%), +10.825 percentage points. Strict sensitivity is 2334/3940 versus 1906/3940. All ten original-source-video differences are positive; 5b causal consequence reasoning regressed 17.742 micro points on 62 cases from six sources. These are adaptively used development results with inherited training/data-exposure limits, not an independent holdout, an official finals score or an all-candidate ranking. The registered local scorer passed14 convenience controls (6positive,8negative). These check response/parser boundaries; they do not estimate organizer-judge agreement, real-question false-positive rates, official calibration or a finals score. All 4,000 were not run through a Docker archive. Source-derived routing equality on the 4,000 intervals 1–299s does not by itself transfer these scores to new image bytes.
|
| 30 |
|
| 31 |
+
The full retained ZIP500 was qualified locally at the stated 4 GiB and 16 GiB tmpfs profiles, alongside loose500, healthy20, reversed20, default and persistent canonical calls, and controlled recovery. These fixtures do not establish the platform scratch quota, all possible input sizes, continuous resource peaks, cold-cache requirements, general order invariance, new semantic scores, independent-holdout performance or official platform success. Canonical deferrals and deliberately failed recovery rows are reduced scientific coverage. The prior R3 4 GiB eager-ZIP failure is a conditional local capacity reproduction, not proof of the organizer failure cause. Earlier images and failed attempts remain historical.
|
| 32 |
|
| 33 |
Training lineage and selected-prefix exposure are disclosed in `LINEAGE.md` and the accompanying JSONs. The separately retained annotation packet has 13,177 derived rows, not the complete 46,758-row ancestor corpus; it contains no underlying video pixels or complete signature ledger.
|
| 34 |
|
REQUIRED_LINKS.json
CHANGED
|
@@ -1,6 +1,7 @@
|
|
| 1 |
{
|
| 2 |
-
"current_image_publication_status": "
|
| 3 |
"current_records": {
|
|
|
|
| 4 |
"asset_inventory": "MODEL_FILES.json",
|
| 5 |
"asset_revision": "35fc28af1d55a5a9ae8096ff2e2cd2c961aa7119",
|
| 6 |
"current_loading_instructions": "LOAD.md",
|
|
@@ -17,7 +18,9 @@
|
|
| 17 |
"licenses": "NOTICES.md",
|
| 18 |
"lineage": "LINEAGE.md",
|
| 19 |
"loading": "LOAD.md",
|
| 20 |
-
"model_card": "README.md"
|
|
|
|
|
|
|
| 21 |
},
|
| 22 |
"historical_availability_verification": {
|
| 23 |
"asset_rows_matching_immutable_A_custody": 50,
|
|
|
|
| 1 |
{
|
| 2 |
+
"current_image_publication_status": "R4 payload locally qualified; exact replacement private commit/readback must be bound separately.",
|
| 3 |
"current_records": {
|
| 4 |
+
"R4_source": "platform_wrapper/ (four exact image source files)",
|
| 5 |
"asset_inventory": "MODEL_FILES.json",
|
| 6 |
"asset_revision": "35fc28af1d55a5a9ae8096ff2e2cd2c961aa7119",
|
| 7 |
"current_loading_instructions": "LOAD.md",
|
|
|
|
| 18 |
"licenses": "NOTICES.md",
|
| 19 |
"lineage": "LINEAGE.md",
|
| 20 |
"loading": "LOAD.md",
|
| 21 |
+
"model_card": "README.md",
|
| 22 |
+
"r4_probe_source": "platform_wrapper/probe_entrypoint.py",
|
| 23 |
+
"r4_zip_source": "platform_wrapper/segment_zip_input.py"
|
| 24 |
},
|
| 25 |
"historical_availability_verification": {
|
| 26 |
"asset_rows_matching_immutable_A_custody": 50,
|
platform_wrapper/CONTRACT.json
CHANGED
|
@@ -133,6 +133,14 @@
|
|
| 133 |
},
|
| 134 |
"input_root": "/input",
|
| 135 |
"output_root": "/output",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
"platform_root_compatibility": {
|
| 137 |
"linked_input_parent": "/opt/ml/input/data",
|
| 138 |
"linked_input_suffix": "-input",
|
|
@@ -196,7 +204,7 @@
|
|
| 196 |
"TRITON_CACHE_DIR": "/tmp/orena-segment/triton",
|
| 197 |
"XDG_CACHE_HOME": "/tmp/orena-segment/cache"
|
| 198 |
},
|
| 199 |
-
"runtime_validation": "
|
| 200 |
"schema_version": "orena-segment-runtime/1",
|
| 201 |
"source_files": [
|
| 202 |
{
|
|
@@ -260,10 +268,10 @@
|
|
| 260 |
"sha256": "a55143dfdbeca52503e225a8a1d2529ef359d3f905ee00ae4d919c6c885c065f"
|
| 261 |
},
|
| 262 |
{
|
| 263 |
-
"bytes":
|
| 264 |
-
"package_path": "
|
| 265 |
"path": "/app/solution/entrypoint/probe_entrypoint.py",
|
| 266 |
-
"sha256": "
|
| 267 |
},
|
| 268 |
{
|
| 269 |
"bytes": 4419,
|
|
@@ -289,6 +297,12 @@
|
|
| 289 |
"path": "/app/solution/entrypoint/segment_unmerged_main.py",
|
| 290 |
"sha256": "7a4df6453eba63c01e7638a5ea402e78653a47f83fb7e25d4cf53f030ef06cf6"
|
| 291 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 292 |
{
|
| 293 |
"bytes": 21221,
|
| 294 |
"package_path": "reference/333cd7fe/solution/entrypoint/test_procedure_byte_identity.py",
|
|
|
|
| 133 |
},
|
| 134 |
"input_root": "/input",
|
| 135 |
"output_root": "/output",
|
| 136 |
+
"per_question_failure_policy": {
|
| 137 |
+
"fatal": "initialization, source/environment/fixed-track, unsafe archive/path/identity/cleanup integrity, non-Exception BaseException (e.g. SystemExit/KeyboardInterrupt)",
|
| 138 |
+
"intentional_deferrals": "existing rules behavior retained and disclosed",
|
| 139 |
+
"ordinary_attributed_exceptions": "blank only affected qID and continue; degraded scientific qualification",
|
| 140 |
+
"output": "exact qID census and finite nonnegative latency; empty content only with attributed failure",
|
| 141 |
+
"revision": "R4",
|
| 142 |
+
"zip_staging": "one selected member lease at a time; full member EOF/CRC before model access; no compressed archive copy"
|
| 143 |
+
},
|
| 144 |
"platform_root_compatibility": {
|
| 145 |
"linked_input_parent": "/opt/ml/input/data",
|
| 146 |
"linked_input_suffix": "-input",
|
|
|
|
| 204 |
"TRITON_CACHE_DIR": "/tmp/orena-segment/triton",
|
| 205 |
"XDG_CACHE_HOME": "/tmp/orena-segment/cache"
|
| 206 |
},
|
| 207 |
+
"runtime_validation": "Successful fixed-source and model initialization, exact per-question attempt/skip and output census; attributed question failures produce blank answers and degraded scientific qualification; source/environment/task/archive-integrity failures remain fatal.",
|
| 208 |
"schema_version": "orena-segment-runtime/1",
|
| 209 |
"source_files": [
|
| 210 |
{
|
|
|
|
| 268 |
"sha256": "a55143dfdbeca52503e225a8a1d2529ef359d3f905ee00ae4d919c6c885c065f"
|
| 269 |
},
|
| 270 |
{
|
| 271 |
+
"bytes": 57832,
|
| 272 |
+
"package_path": "runtime/platform_input_r4/probe_entrypoint.py",
|
| 273 |
"path": "/app/solution/entrypoint/probe_entrypoint.py",
|
| 274 |
+
"sha256": "40e91b3780417c3503fe14164385aa1d91ab7571265de05962026228f8cf4e33"
|
| 275 |
},
|
| 276 |
{
|
| 277 |
"bytes": 4419,
|
|
|
|
| 297 |
"path": "/app/solution/entrypoint/segment_unmerged_main.py",
|
| 298 |
"sha256": "7a4df6453eba63c01e7638a5ea402e78653a47f83fb7e25d4cf53f030ef06cf6"
|
| 299 |
},
|
| 300 |
+
{
|
| 301 |
+
"bytes": 16014,
|
| 302 |
+
"package_path": "runtime/platform_input_r4/segment_zip_input.py",
|
| 303 |
+
"path": "/app/solution/entrypoint/segment_zip_input.py",
|
| 304 |
+
"sha256": "aef9806482af9b03ce08946c01240183393fbf15c6a2046bad9e00c27a3f58a7"
|
| 305 |
+
},
|
| 306 |
{
|
| 307 |
"bytes": 21221,
|
| 308 |
"package_path": "reference/333cd7fe/solution/entrypoint/test_procedure_byte_identity.py",
|
platform_wrapper/bootstrap.py
CHANGED
|
@@ -25,7 +25,7 @@ import zipfile
|
|
| 25 |
|
| 26 |
TRACK = "segment"
|
| 27 |
PLATFORM_ROOT_POLICY = {'schema': 'segment-declared-platform-input-root/1', 'logical_input_root': '/input', 'linked_input_parent': '/opt/ml/input/data', 'linked_input_suffix': '-input', 'production_scratch_policy': 'exclusive_per_invocation_children', 'output_and_tmp_aliases': 'refused'}
|
| 28 |
-
CONTRACT_SHA256 = '
|
| 29 |
IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".bmp", ".webp"}
|
| 30 |
VIDEO_EXTENSIONS = {".mp4", ".avi", ".mov", ".mkv", ".webm", ".m4v", ".mpg", ".mpeg"}
|
| 31 |
SAFE_COMPONENT = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_.-]{0,254}\Z")
|
|
@@ -39,7 +39,7 @@ PROTECTED_NAMES = {"PATH", "LD_LIBRARY_PATH", "LD_PRELOAD", "LD_AUDIT", "HOME",
|
|
| 39 |
"OWLV2_REVISION", "CUDA_VISIBLE_DEVICES", "NVIDIA_VISIBLE_DEVICES",
|
| 40 |
"NVIDIA_DRIVER_CAPABILITIES"}
|
| 41 |
DEGRADED_MARKERS = ("PROBESHELL DEGRADED:", "PROBESHELL: launcher floor stands",
|
| 42 |
-
"SHIP-CONFIG ASSERT FAILED",
|
| 43 |
"TRACK: FRAME RESOLVE-FAIL", "WARN: Owlv2 model load failed:",
|
| 44 |
"keeping the pre-seeded fallback", "L17 eager-load: failed",
|
| 45 |
"L17 eager-load raised:")
|
|
@@ -308,6 +308,15 @@ def safe_component(value, label):
|
|
| 308 |
return value
|
| 309 |
|
| 310 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 311 |
def validate_inputs(root: Path):
|
| 312 |
data, request_sha = read_json(root / "request.json")
|
| 313 |
rows = [data] if isinstance(data, dict) else data
|
|
@@ -317,7 +326,7 @@ def validate_inputs(root: Path):
|
|
| 317 |
for row in rows:
|
| 318 |
if not isinstance(row, dict):
|
| 319 |
raise ValueError("request row must be an object")
|
| 320 |
-
qid =
|
| 321 |
if qid in seen:
|
| 322 |
raise ValueError("duplicate qID")
|
| 323 |
seen.add(qid)
|
|
@@ -325,8 +334,8 @@ def validate_inputs(root: Path):
|
|
| 325 |
raise ValueError("request track mismatch")
|
| 326 |
if not isinstance(row.get("question"), str) or not row["question"].strip():
|
| 327 |
raise ValueError("question must be nonempty text")
|
| 328 |
-
if row.get("videoID") is not None:
|
| 329 |
-
|
| 330 |
for key in ("start_time", "end_time"):
|
| 331 |
if key not in row or type(row[key]) not in (int, float) or not math.isfinite(row[key]) or row[key] < 0:
|
| 332 |
raise ValueError(f"invalid or missing {key}")
|
|
@@ -380,6 +389,7 @@ def validate_inputs(root: Path):
|
|
| 380 |
if any(name.suffix.lower() in ((VIDEO_EXTENSIONS if TRACK == "frame" else IMAGE_EXTENSIONS)) for name, _ in staged.values()):
|
| 381 |
raise ValueError("other-track media present")
|
| 382 |
selected = []
|
|
|
|
| 383 |
for row in rows:
|
| 384 |
qid = row["qID"]
|
| 385 |
hits = [name for name in media if name.stem == qid]
|
|
@@ -387,11 +397,14 @@ def validate_inputs(root: Path):
|
|
| 387 |
groups.append(hits)
|
| 388 |
groups.append([p for p in media if p.stem == row.get("videoID")])
|
| 389 |
candidates = next((group for group in groups if group), [])
|
| 390 |
-
if len(candidates)
|
| 391 |
-
raise ValueError(f"
|
| 392 |
-
|
|
|
|
|
|
|
|
|
|
| 393 |
return {"requests": len(rows), "input_files": len(files), "archives": archive_count,
|
| 394 |
-
"media_files": len(media), "resolved_requests": len(selected), "request_sha256": request_sha,
|
| 395 |
"validation": "names, layout and supplied interval metadata only; no media decoding"}
|
| 396 |
|
| 397 |
|
|
@@ -399,7 +412,7 @@ def protected(name):
|
|
| 399 |
return name.startswith(PROTECTED_PREFIXES) or name in PROTECTED_NAMES
|
| 400 |
|
| 401 |
|
| 402 |
-
def validate_outputs(output_root: Path, input_root: Path):
|
| 403 |
regular_path(output_root, directory=True)
|
| 404 |
if sorted(p.name for p in output_root.iterdir()) != ["answer.json"]:
|
| 405 |
raise ValueError("official output must contain only answer.json")
|
|
@@ -413,12 +426,14 @@ def validate_outputs(output_root: Path, input_root: Path):
|
|
| 413 |
for answer in answers:
|
| 414 |
if not isinstance(answer, dict) or set(answer) != {"qID", "content", "latency"}:
|
| 415 |
raise ValueError("answer schema requires exactly qID/content/latency")
|
| 416 |
-
qid =
|
| 417 |
if qid in seen:
|
| 418 |
raise ValueError("duplicate answer qID")
|
| 419 |
seen.add(qid)
|
| 420 |
-
if not isinstance(answer["content"], str)
|
| 421 |
-
raise ValueError("answer content must be
|
|
|
|
|
|
|
| 422 |
latency = answer["latency"]
|
| 423 |
if type(latency) not in (int, float) or not math.isfinite(latency) or latency < 0:
|
| 424 |
raise ValueError("answer latency must be a finite nonnegative number")
|
|
@@ -602,6 +617,12 @@ class EngineObservations:
|
|
| 602 |
self.temporal = temporal
|
| 603 |
self.temporal_handler = None
|
| 604 |
self.input_deferrals = []
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 605 |
self.allowed_deferrals_by_layer = {}
|
| 606 |
owner = self
|
| 607 |
active = {"layer": None, "qid": None, "sample": None, "sample_return": None}
|
|
@@ -610,6 +631,7 @@ class EngineObservations:
|
|
| 610 |
original = getattr(cls, name)
|
| 611 |
self.originals.append((cls, name, original))
|
| 612 |
def observed(instance, *args, **kwargs):
|
|
|
|
| 613 |
try:
|
| 614 |
if name == "answer":
|
| 615 |
# The original caller supplies infer_all's scratch dict.
|
|
@@ -620,6 +642,13 @@ class EngineObservations:
|
|
| 620 |
if ("scratch_init" in kwargs and len(args) > 1) or not isinstance(scratch, dict) or scratch.get("track") != TRACK:
|
| 621 |
owner.events.append("answer scratch_init track missing, ambiguous or foreign")
|
| 622 |
raise ValueError("answer scratch_init must carry the fixed track")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 623 |
owner.counts["answers_track_checked"] += 1
|
| 624 |
result = original(instance, *args, **kwargs)
|
| 625 |
if result_check is not None and not result_check(instance, result):
|
|
@@ -628,10 +657,18 @@ class EngineObservations:
|
|
| 628 |
owner.counts[counter] += 1
|
| 629 |
if name == "__init__":
|
| 630 |
owner.containers.append(instance)
|
|
|
|
| 631 |
return result
|
| 632 |
except BaseException as error:
|
| 633 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 634 |
raise
|
|
|
|
|
|
|
|
|
|
| 635 |
setattr(cls, name, observed)
|
| 636 |
|
| 637 |
loaded = lambda instance, result: getattr(instance, "_model", None) is not None and getattr(instance, "_processor", None) is not None
|
|
@@ -704,6 +741,18 @@ class EngineObservations:
|
|
| 704 |
key = id(instance)
|
| 705 |
owner.allowed_deferrals_by_layer[key] = owner.allowed_deferrals_by_layer.get(key, 0) + 1
|
| 706 |
active["sample"] = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 707 |
return original_failure(instance, message, error)
|
| 708 |
|
| 709 |
layer_class.predict = observed_predict
|
|
@@ -796,27 +845,86 @@ class EngineObservations:
|
|
| 796 |
"actual_grid_t": local.get("grid_t"),
|
| 797 |
"expected_grid_t": local["expected_t"]})
|
| 798 |
return
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 799 |
if any(marker in message for marker in ("inference FAILED", "R1/build defer", "frames (<2)", "model load FAILED")):
|
| 800 |
owner.events.append("temporal seam reported failure: " + message[:500])
|
| 801 |
self.temporal_handler = TemporalHandler()
|
| 802 |
temporal.log.addHandler(self.temporal_handler)
|
| 803 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 804 |
def finish(self, requests):
|
| 805 |
-
|
|
|
|
| 806 |
self.events.append("engine initialization/answer census mismatch")
|
| 807 |
if self.counts["detector_load_completed"] < 1 or self.counts["vlm_eager_load_completed"] < 1:
|
| 808 |
self.events.append("required model initialization was not completed")
|
| 809 |
if self.temporal is not None and (self.counts["temporal_load_completed"] < 1 or self.temporal._LOAD_FAILED):
|
| 810 |
self.events.append("required temporal initialization was not completed")
|
| 811 |
for container in self.containers:
|
| 812 |
-
|
| 813 |
-
self.
|
|
|
|
| 814 |
for layer in getattr(container, "layers", []):
|
| 815 |
allowed = self.allowed_deferrals_by_layer.get(id(layer), 0)
|
| 816 |
if getattr(layer, "_load_failed", False) or getattr(layer, "_n_infer_fail", 0) != allowed:
|
| 817 |
self.events.append("layer recorded model load/inference failure")
|
| 818 |
return {"counts": dict(self.counts), "failures": list(dict.fromkeys(self.events)),
|
| 819 |
"observed_input_deferrals": list(self.input_deferrals),
|
|
|
|
|
|
|
|
|
|
|
|
|
| 820 |
"input_deferral_summary": {
|
| 821 |
"events": len(self.input_deferrals),
|
| 822 |
"affected_qIDs": sorted({item["qID"] for item in self.input_deferrals}),
|
|
@@ -861,6 +969,131 @@ def verify_shell_counters(counters, requests):
|
|
| 861 |
"scope": "original structured shell counters; no correctness or timing conclusion"}
|
| 862 |
|
| 863 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 864 |
def execute(contract, *, input_root: Path, output_root: Path, evidence_root: Path | None = None, backend=None, setup_imports=False, production_invocation=False):
|
| 865 |
"""Synthetic-fixture seam; production supplies only baked paths and launcher."""
|
| 866 |
receipt = {"track": TRACK, "status": "failed", "source_validation": None,
|
|
@@ -962,14 +1195,32 @@ def execute(contract, *, input_root: Path, output_root: Path, evidence_root: Pat
|
|
| 962 |
guard.violations.append({"key": "environment API", "operation": "guard replaced"})
|
| 963 |
guard.final_check()
|
| 964 |
receipt["configuration_violations"] = list(guard.violations)
|
|
|
|
| 965 |
if observations is not None:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 966 |
receipt["engine_observations"] = observations.finish(receipt["input_validation"]["requests"])
|
| 967 |
if receipt["engine_observations"]["failures"]:
|
| 968 |
raise ValueError("original engine initialization or answer failure")
|
| 969 |
verify_sources(contract)
|
| 970 |
if validate_inputs(input_root) != receipt["input_validation"]:
|
| 971 |
raise ValueError("input metadata or layout changed during backend execution")
|
| 972 |
-
receipt["
|
|
|
|
| 973 |
check_input_binding(binding, policy)
|
| 974 |
if guard.violations or receipt["degraded_markers"]:
|
| 975 |
raise ValueError("backend degraded or attempted fixed configuration mutation")
|
|
|
|
| 25 |
|
| 26 |
TRACK = "segment"
|
| 27 |
PLATFORM_ROOT_POLICY = {'schema': 'segment-declared-platform-input-root/1', 'logical_input_root': '/input', 'linked_input_parent': '/opt/ml/input/data', 'linked_input_suffix': '-input', 'production_scratch_policy': 'exclusive_per_invocation_children', 'output_and_tmp_aliases': 'refused'}
|
| 28 |
+
CONTRACT_SHA256 = '2426fca2beaec028bf506085083f9c0eb681620dd667fb0386ffc41236dcc1de'
|
| 29 |
IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".bmp", ".webp"}
|
| 30 |
VIDEO_EXTENSIONS = {".mp4", ".avi", ".mov", ".mkv", ".webm", ".m4v", ".mpg", ".mpeg"}
|
| 31 |
SAFE_COMPONENT = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_.-]{0,254}\Z")
|
|
|
|
| 39 |
"OWLV2_REVISION", "CUDA_VISIBLE_DEVICES", "NVIDIA_VISIBLE_DEVICES",
|
| 40 |
"NVIDIA_DRIVER_CAPABILITIES"}
|
| 41 |
DEGRADED_MARKERS = ("PROBESHELL DEGRADED:", "PROBESHELL: launcher floor stands",
|
| 42 |
+
"SHIP-CONFIG ASSERT FAILED",
|
| 43 |
"TRACK: FRAME RESOLVE-FAIL", "WARN: Owlv2 model load failed:",
|
| 44 |
"keeping the pre-seeded fallback", "L17 eager-load: failed",
|
| 45 |
"L17 eager-load raised:")
|
|
|
|
| 308 |
return value
|
| 309 |
|
| 310 |
|
| 311 |
+
def literal_qid(value, label):
|
| 312 |
+
import unicodedata
|
| 313 |
+
if (not isinstance(value, str) or not value or value in (".", "..")
|
| 314 |
+
or "/" in value or "\\" in value
|
| 315 |
+
or any(unicodedata.category(c) in ("Cc", "Cs") for c in value)):
|
| 316 |
+
raise ValueError(f"unsafe {label}")
|
| 317 |
+
return value
|
| 318 |
+
|
| 319 |
+
|
| 320 |
def validate_inputs(root: Path):
|
| 321 |
data, request_sha = read_json(root / "request.json")
|
| 322 |
rows = [data] if isinstance(data, dict) else data
|
|
|
|
| 326 |
for row in rows:
|
| 327 |
if not isinstance(row, dict):
|
| 328 |
raise ValueError("request row must be an object")
|
| 329 |
+
qid = literal_qid(row.get("qID"), "qID")
|
| 330 |
if qid in seen:
|
| 331 |
raise ValueError("duplicate qID")
|
| 332 |
seen.add(qid)
|
|
|
|
| 334 |
raise ValueError("request track mismatch")
|
| 335 |
if not isinstance(row.get("question"), str) or not row["question"].strip():
|
| 336 |
raise ValueError("question must be nonempty text")
|
| 337 |
+
if row.get("videoID") is not None and not isinstance(row["videoID"], str):
|
| 338 |
+
raise ValueError("videoID must be metadata text")
|
| 339 |
for key in ("start_time", "end_time"):
|
| 340 |
if key not in row or type(row[key]) not in (int, float) or not math.isfinite(row[key]) or row[key] < 0:
|
| 341 |
raise ValueError(f"invalid or missing {key}")
|
|
|
|
| 389 |
if any(name.suffix.lower() in ((VIDEO_EXTENSIONS if TRACK == "frame" else IMAGE_EXTENSIONS)) for name, _ in staged.values()):
|
| 390 |
raise ValueError("other-track media present")
|
| 391 |
selected = []
|
| 392 |
+
unresolved = []
|
| 393 |
for row in rows:
|
| 394 |
qid = row["qID"]
|
| 395 |
hits = [name for name in media if name.stem == qid]
|
|
|
|
| 397 |
groups.append(hits)
|
| 398 |
groups.append([p for p in media if p.stem == row.get("videoID")])
|
| 399 |
candidates = next((group for group in groups if group), [])
|
| 400 |
+
if len(candidates) > 1:
|
| 401 |
+
raise ValueError(f"ambiguous media for qID: {qid}")
|
| 402 |
+
if not candidates:
|
| 403 |
+
unresolved.append(qid)
|
| 404 |
+
else:
|
| 405 |
+
selected.append(str(candidates[0]))
|
| 406 |
return {"requests": len(rows), "input_files": len(files), "archives": archive_count,
|
| 407 |
+
"media_files": len(media), "resolved_requests": len(selected), "unresolved_qIDs": unresolved, "request_sha256": request_sha,
|
| 408 |
"validation": "names, layout and supplied interval metadata only; no media decoding"}
|
| 409 |
|
| 410 |
|
|
|
|
| 412 |
return name.startswith(PROTECTED_PREFIXES) or name in PROTECTED_NAMES
|
| 413 |
|
| 414 |
|
| 415 |
+
def validate_outputs(output_root: Path, input_root: Path, *, empty_qids=frozenset()):
|
| 416 |
regular_path(output_root, directory=True)
|
| 417 |
if sorted(p.name for p in output_root.iterdir()) != ["answer.json"]:
|
| 418 |
raise ValueError("official output must contain only answer.json")
|
|
|
|
| 426 |
for answer in answers:
|
| 427 |
if not isinstance(answer, dict) or set(answer) != {"qID", "content", "latency"}:
|
| 428 |
raise ValueError("answer schema requires exactly qID/content/latency")
|
| 429 |
+
qid = literal_qid(answer["qID"], "answer qID")
|
| 430 |
if qid in seen:
|
| 431 |
raise ValueError("duplicate answer qID")
|
| 432 |
seen.add(qid)
|
| 433 |
+
if not isinstance(answer["content"], str):
|
| 434 |
+
raise ValueError("answer content must be text")
|
| 435 |
+
if not answer["content"].strip() and qid not in empty_qids:
|
| 436 |
+
raise ValueError("empty answer lacks an attributed per-question failure")
|
| 437 |
latency = answer["latency"]
|
| 438 |
if type(latency) not in (int, float) or not math.isfinite(latency) or latency < 0:
|
| 439 |
raise ValueError("answer latency must be a finite nonnegative number")
|
|
|
|
| 617 |
self.temporal = temporal
|
| 618 |
self.temporal_handler = None
|
| 619 |
self.input_deferrals = []
|
| 620 |
+
self.question_failures = []
|
| 621 |
+
self.answer_qids = []
|
| 622 |
+
self.answer_calls_failed = 0
|
| 623 |
+
self.current_answer_qid = None
|
| 624 |
+
self.allowed_layer_exceptions = {}
|
| 625 |
+
self.skipped_qids = set()
|
| 626 |
self.allowed_deferrals_by_layer = {}
|
| 627 |
owner = self
|
| 628 |
active = {"layer": None, "qid": None, "sample": None, "sample_return": None}
|
|
|
|
| 631 |
original = getattr(cls, name)
|
| 632 |
self.originals.append((cls, name, original))
|
| 633 |
def observed(instance, *args, **kwargs):
|
| 634 |
+
admitted_answer = False
|
| 635 |
try:
|
| 636 |
if name == "answer":
|
| 637 |
# The original caller supplies infer_all's scratch dict.
|
|
|
|
| 642 |
if ("scratch_init" in kwargs and len(args) > 1) or not isinstance(scratch, dict) or scratch.get("track") != TRACK:
|
| 643 |
owner.events.append("answer scratch_init track missing, ambiguous or foreign")
|
| 644 |
raise ValueError("answer scratch_init must carry the fixed track")
|
| 645 |
+
request = args[0] if args else kwargs.get("request")
|
| 646 |
+
qid = request.get("qID") if isinstance(request, dict) else getattr(request, "qID", None)
|
| 647 |
+
if not isinstance(qid, str) or qid in owner.answer_qids:
|
| 648 |
+
raise ValueError("missing or repeated original answer qID")
|
| 649 |
+
owner.answer_qids.append(qid)
|
| 650 |
+
owner.current_answer_qid = qid
|
| 651 |
+
admitted_answer = True
|
| 652 |
owner.counts["answers_track_checked"] += 1
|
| 653 |
result = original(instance, *args, **kwargs)
|
| 654 |
if result_check is not None and not result_check(instance, result):
|
|
|
|
| 657 |
owner.counts[counter] += 1
|
| 658 |
if name == "__init__":
|
| 659 |
owner.containers.append(instance)
|
| 660 |
+
owner.observe_layer_exceptions(instance)
|
| 661 |
return result
|
| 662 |
except BaseException as error:
|
| 663 |
+
if name == "answer" and admitted_answer and isinstance(error, Exception) and not integrity_exception(error):
|
| 664 |
+
owner.answer_calls_failed += 1
|
| 665 |
+
owner.record_question_failure(exception_event(error, owner.current_answer_qid, "container_answer_exception"))
|
| 666 |
+
else:
|
| 667 |
+
owner.events.append(f"{name} raised {type(error).__name__}")
|
| 668 |
raise
|
| 669 |
+
finally:
|
| 670 |
+
if admitted_answer:
|
| 671 |
+
owner.current_answer_qid = None
|
| 672 |
setattr(cls, name, observed)
|
| 673 |
|
| 674 |
loaded = lambda instance, result: getattr(instance, "_model", None) is not None and getattr(instance, "_processor", None) is not None
|
|
|
|
| 741 |
key = id(instance)
|
| 742 |
owner.allowed_deferrals_by_layer[key] = owner.allowed_deferrals_by_layer.get(key, 0) + 1
|
| 743 |
active["sample"] = None
|
| 744 |
+
elif instance is active["layer"] and active["qid"] is not None:
|
| 745 |
+
event = None
|
| 746 |
+
if message == expected_message and error is None:
|
| 747 |
+
event = {"qID": active["qid"], "stage": "main_pre_generation_media_resolution",
|
| 748 |
+
"reason": "Original main path could not resolve a usable clip before sampling",
|
| 749 |
+
"producer": {"path": "/app/solution/layers/vlm_cholec.py", "function": "predict", "exception": None}}
|
| 750 |
+
elif error is not None:
|
| 751 |
+
event = classified_question_failure(error, active["qid"], "main")
|
| 752 |
+
if event is not None:
|
| 753 |
+
owner.record_question_failure(event)
|
| 754 |
+
key = id(instance)
|
| 755 |
+
owner.allowed_deferrals_by_layer[key] = owner.allowed_deferrals_by_layer.get(key, 0) + 1
|
| 756 |
return original_failure(instance, message, error)
|
| 757 |
|
| 758 |
layer_class.predict = observed_predict
|
|
|
|
| 845 |
"actual_grid_t": local.get("grid_t"),
|
| 846 |
"expected_grid_t": local["expected_t"]})
|
| 847 |
return
|
| 848 |
+
if exact_producer and ((record.lineno == 175 and record.msg == "segment_temporal_seam: %s R1/build defer (%s)") or (record.lineno == 179 and record.msg == "segment_temporal_seam: %s inference FAILED (%r); defer to rules")) and isinstance(args[1], Exception):
|
| 849 |
+
event = classified_question_failure(args[1], active["qid"], "temporal")
|
| 850 |
+
if event is not None:
|
| 851 |
+
owner.record_question_failure(event)
|
| 852 |
+
return
|
| 853 |
if any(marker in message for marker in ("inference FAILED", "R1/build defer", "frames (<2)", "model load FAILED")):
|
| 854 |
owner.events.append("temporal seam reported failure: " + message[:500])
|
| 855 |
self.temporal_handler = TemporalHandler()
|
| 856 |
temporal.log.addHandler(self.temporal_handler)
|
| 857 |
|
| 858 |
+
def observe_layer_exceptions(self, container):
|
| 859 |
+
for layer in getattr(container, "layers", []):
|
| 860 |
+
for method_name in ("applies", "predict"):
|
| 861 |
+
original = getattr(layer, method_name, None)
|
| 862 |
+
if original is None or not hasattr(layer, "name"):
|
| 863 |
+
continue
|
| 864 |
+
self.originals.append((layer, method_name, original))
|
| 865 |
+
def observed(*args, _original=original, _layer=layer, _method=method_name, **kwargs):
|
| 866 |
+
try:
|
| 867 |
+
return _original(*args, **kwargs)
|
| 868 |
+
except Exception as error:
|
| 869 |
+
if self.current_answer_qid is not None and not integrity_exception(error):
|
| 870 |
+
key = (id(container), _layer.name)
|
| 871 |
+
self.allowed_layer_exceptions[key] = self.allowed_layer_exceptions.get(key, 0) + 1
|
| 872 |
+
self.record_question_failure(exception_event(error, self.current_answer_qid, "layer_" + _method + "_exception", layer=_layer.name))
|
| 873 |
+
else:
|
| 874 |
+
self.events.append("unattributed or integrity layer exception")
|
| 875 |
+
raise
|
| 876 |
+
setattr(layer, method_name, observed)
|
| 877 |
+
|
| 878 |
+
def record_question_failure(self, event):
|
| 879 |
+
self.question_failures.append({**event, "output_policy": "empty_content", "scientific_coverage": "failed question; no correctness or healthy-neural-coverage claim"})
|
| 880 |
+
|
| 881 |
+
def reconcile_skipped(self, expected_qids, events, question_events=(), fence_events=()):
|
| 882 |
+
expected = set(expected_qids)
|
| 883 |
+
seen = set()
|
| 884 |
+
for event in events:
|
| 885 |
+
validate_staging_event(event, expected)
|
| 886 |
+
seen.add(event["qID"])
|
| 887 |
+
self.record_question_failure({"qID": event["qID"], "stage": event["stage"], "reason": event["message"], "producer": dict(event)})
|
| 888 |
+
question_ids = set()
|
| 889 |
+
for event in question_events:
|
| 890 |
+
validate_question_event(event, expected)
|
| 891 |
+
if event["qID"] in question_ids:
|
| 892 |
+
raise ValueError("repeated answer-loop failure event")
|
| 893 |
+
question_ids.add(event["qID"])
|
| 894 |
+
self.record_question_failure({"qID": event["qID"], "stage": event["stage"], "reason": event["message"], "producer": dict(event)})
|
| 895 |
+
for event in fence_events:
|
| 896 |
+
if not isinstance(event, dict) or set(event) != {"qID", "reason", "stage"} or event["qID"] not in expected or event["reason"] not in ("WEDGE", "CRASH") or event["stage"] != "startup_decoder_fence":
|
| 897 |
+
raise ValueError("unrecognized original decoder fence")
|
| 898 |
+
self.record_question_failure({"qID": event["qID"], "stage": event["stage"], "reason": event["reason"], "producer": dict(event)})
|
| 899 |
+
if set(self.answer_qids) & seen:
|
| 900 |
+
raise ValueError("staging-skipped question unexpectedly reached original answer")
|
| 901 |
+
seen |= question_ids - set(self.answer_qids)
|
| 902 |
+
if set(self.answer_qids) | seen != expected:
|
| 903 |
+
raise ValueError("original answer plus attributed staging-skip qID census mismatch")
|
| 904 |
+
self.skipped_qids = seen
|
| 905 |
+
|
| 906 |
def finish(self, requests):
|
| 907 |
+
completed_expected = requests - len(self.skipped_qids)
|
| 908 |
+
if self.counts["container_initialized"] != 1 or self.counts["answers_completed"] + self.answer_calls_failed != completed_expected or self.counts["answers_track_checked"] != completed_expected:
|
| 909 |
self.events.append("engine initialization/answer census mismatch")
|
| 910 |
if self.counts["detector_load_completed"] < 1 or self.counts["vlm_eager_load_completed"] < 1:
|
| 911 |
self.events.append("required model initialization was not completed")
|
| 912 |
if self.temporal is not None and (self.counts["temporal_load_completed"] < 1 or self.temporal._LOAD_FAILED):
|
| 913 |
self.events.append("required temporal initialization was not completed")
|
| 914 |
for container in self.containers:
|
| 915 |
+
for name, count in getattr(container, "exceptions", {}).items():
|
| 916 |
+
if count != self.allowed_layer_exceptions.get((id(container), name), 0):
|
| 917 |
+
self.events.append("container recorded an unattributed layer exception")
|
| 918 |
for layer in getattr(container, "layers", []):
|
| 919 |
allowed = self.allowed_deferrals_by_layer.get(id(layer), 0)
|
| 920 |
if getattr(layer, "_load_failed", False) or getattr(layer, "_n_infer_fail", 0) != allowed:
|
| 921 |
self.events.append("layer recorded model load/inference failure")
|
| 922 |
return {"counts": dict(self.counts), "failures": list(dict.fromkeys(self.events)),
|
| 923 |
"observed_input_deferrals": list(self.input_deferrals),
|
| 924 |
+
"observed_question_failures": list(self.question_failures),
|
| 925 |
+
"answer_qIDs": list(self.answer_qids), "staging_skipped_qIDs": sorted(self.skipped_qids),
|
| 926 |
+
"answer_calls_failed": self.answer_calls_failed,
|
| 927 |
+
"strict_model_and_guard_health_passed": not self.events and not self.input_deferrals and not self.question_failures,
|
| 928 |
"input_deferral_summary": {
|
| 929 |
"events": len(self.input_deferrals),
|
| 930 |
"affected_qIDs": sorted({item["qID"] for item in self.input_deferrals}),
|
|
|
|
| 969 |
"scope": "original structured shell counters; no correctness or timing conclusion"}
|
| 970 |
|
| 971 |
|
| 972 |
+
def integrity_exception(error):
|
| 973 |
+
module = sys.modules.get("solution.entrypoint.segment_zip_input")
|
| 974 |
+
cls = getattr(module, "ZipIntegrityError", None)
|
| 975 |
+
return isinstance(cls, type) and isinstance(error, cls)
|
| 976 |
+
|
| 977 |
+
|
| 978 |
+
def exception_event(error, qid, stage, **metadata):
|
| 979 |
+
trace, frames = error.__traceback__, []
|
| 980 |
+
while trace is not None:
|
| 981 |
+
frames.append({"path": trace.tb_frame.f_code.co_filename, "function": trace.tb_frame.f_code.co_name, "line": trace.tb_lineno})
|
| 982 |
+
trace = trace.tb_next
|
| 983 |
+
return {"qID": qid, "stage": stage, "reason": str(error), "producer": {"exception_type": type(error).__name__, "exception_module": type(error).__module__, "traceback": frames, **metadata}}
|
| 984 |
+
|
| 985 |
+
|
| 986 |
+
def classified_question_failure(error, qid, route):
|
| 987 |
+
"""Attribute ordinary per-question exceptions; preserve exact known stages and label unknown stages honestly."""
|
| 988 |
+
trace, frames = error.__traceback__, []
|
| 989 |
+
while trace is not None:
|
| 990 |
+
frames.append({"path": trace.tb_frame.f_code.co_filename,
|
| 991 |
+
"function": trace.tb_frame.f_code.co_name, "line": trace.tb_lineno})
|
| 992 |
+
trace = trace.tb_next
|
| 993 |
+
if not frames or not isinstance(error, Exception) or integrity_exception(error):
|
| 994 |
+
return None
|
| 995 |
+
leaf = frames[-1]
|
| 996 |
+
rendering = "/app/solution/entrypoint/segment_final_rendering.py"
|
| 997 |
+
temporal = "/app/solution/entrypoint/segment_temporal_seam.py"
|
| 998 |
+
clock = "/app/solution/entrypoint/segment_main_source_clock.py"
|
| 999 |
+
stage = None
|
| 1000 |
+
if route == "temporal" and type(error) is ValueError:
|
| 1001 |
+
if leaf["path"] == temporal and leaf["function"] == "segment_temporal_predict" and leaf["line"] in (151, 155):
|
| 1002 |
+
stage = "temporal_pre_generation_frame_census" if leaf["line"] == 151 else "temporal_pre_generation_source_clock"
|
| 1003 |
+
elif leaf["path"] == rendering and leaf["function"] in ("emit", "seconds", "hms") and any(x["path"] == rendering and x["function"] == "emit" for x in frames):
|
| 1004 |
+
stage = "temporal_post_generation_invalid_format"
|
| 1005 |
+
if route == "main":
|
| 1006 |
+
if type(error) is ValueError and leaf["path"] == clock and leaf["function"] in ("sample", "validate_metadata", "content"):
|
| 1007 |
+
stage = "main_pre_generation_source_clock"
|
| 1008 |
+
cv = sys.modules.get("cv2")
|
| 1009 |
+
cv_error = getattr(cv, "error", None)
|
| 1010 |
+
decode_error = isinstance(error, OSError) or (isinstance(cv_error, type) and isinstance(error, cv_error))
|
| 1011 |
+
if decode_error and any(x["path"] == "/app/solution/frame_sampler.py" for x in frames):
|
| 1012 |
+
stage = "main_pre_generation_decoder_exception"
|
| 1013 |
+
if stage is None:
|
| 1014 |
+
stage = route + "_inference_exception_unclassified_stage"
|
| 1015 |
+
return {"qID": qid, "stage": stage, "reason": str(error),
|
| 1016 |
+
"producer": {"exception_type": type(error).__name__, "traceback": frames,
|
| 1017 |
+
"model_generation_reached": True if stage == "temporal_post_generation_invalid_format" else None}}
|
| 1018 |
+
|
| 1019 |
+
|
| 1020 |
+
def validate_staging_event(event, expected_qids):
|
| 1021 |
+
fields = {"qID", "phase", "stage", "archive", "member", "declared_bytes", "written_bytes",
|
| 1022 |
+
"exception_type", "errno", "message", "output_policy"}
|
| 1023 |
+
if not isinstance(event, dict) or set(event) != fields:
|
| 1024 |
+
raise ValueError("unexpected trusted staging event schema")
|
| 1025 |
+
if event["qID"] not in expected_qids or event["phase"] not in ("startup_probe", "answer") or event["output_policy"] != "empty_content":
|
| 1026 |
+
raise ValueError("staging event qID/phase/output policy differs")
|
| 1027 |
+
if event["stage"] not in ("zip_member_staging", "input_media_resolution"):
|
| 1028 |
+
raise ValueError("unknown staging failure class")
|
| 1029 |
+
if not isinstance(event["message"], str) or not isinstance(event["exception_type"], str):
|
| 1030 |
+
raise ValueError("invalid staging exception evidence")
|
| 1031 |
+
if type(event["written_bytes"]) is not int or event["written_bytes"] < 0 or (event["errno"] is not None and type(event["errno"]) is not int):
|
| 1032 |
+
raise ValueError("invalid staging byte/errno evidence")
|
| 1033 |
+
if event["stage"] == "input_media_resolution":
|
| 1034 |
+
if (event["archive"], event["member"], event["declared_bytes"], event["written_bytes"], event["exception_type"]) != (None, None, None, 0, "MissingMedia"):
|
| 1035 |
+
raise ValueError("missing-media event identity differs")
|
| 1036 |
+
elif (not isinstance(event["archive"], str) or not isinstance(event["member"], str)
|
| 1037 |
+
or type(event["declared_bytes"]) is not int or not 0 <= event["written_bytes"] <= event["declared_bytes"]):
|
| 1038 |
+
raise ValueError("ZIP failure member/byte evidence differs")
|
| 1039 |
+
|
| 1040 |
+
|
| 1041 |
+
def blank_failed_outputs(output_root, input_root, events):
|
| 1042 |
+
"""Only after the original loop: preserve healthy strings and all latencies."""
|
| 1043 |
+
failed = {event["qID"] for event in events}
|
| 1044 |
+
before = validate_outputs(output_root, input_root, empty_qids=failed)
|
| 1045 |
+
answers, original_sha = read_json(output_root / "answer.json")
|
| 1046 |
+
qids = {answer["qID"] for answer in answers}
|
| 1047 |
+
if not failed <= qids:
|
| 1048 |
+
raise ValueError("question-failure ledger contains an unknown qID")
|
| 1049 |
+
if not failed:
|
| 1050 |
+
return {"blanked_qIDs": [], "answer_sha256_before": original_sha, "answer_sha256_after": original_sha,
|
| 1051 |
+
"successful_answer_contents_unchanged": True, "latencies_unchanged": True}
|
| 1052 |
+
kept = {answer["qID"]: dict(answer) for answer in answers}
|
| 1053 |
+
for answer in answers:
|
| 1054 |
+
if answer["qID"] in failed:
|
| 1055 |
+
answer["content"] = ""
|
| 1056 |
+
target = output_root / "answer.json"
|
| 1057 |
+
temp = output_root / ".r4-answer.json.tmp"
|
| 1058 |
+
try:
|
| 1059 |
+
fd = os.open(temp, os.O_CREAT | os.O_EXCL | os.O_WRONLY | os.O_NOFOLLOW, 0o600)
|
| 1060 |
+
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
| 1061 |
+
json.dump(answers, stream, indent=1, allow_nan=False)
|
| 1062 |
+
stream.write("\n")
|
| 1063 |
+
_, current_sha = read_json(target)
|
| 1064 |
+
if current_sha != original_sha:
|
| 1065 |
+
raise ValueError("answer changed after original loop")
|
| 1066 |
+
os.replace(temp, target)
|
| 1067 |
+
finally:
|
| 1068 |
+
if temp.exists():
|
| 1069 |
+
temp.unlink()
|
| 1070 |
+
after = validate_outputs(output_root, input_root, empty_qids=failed)
|
| 1071 |
+
actual, _ = read_json(target)
|
| 1072 |
+
for answer in actual:
|
| 1073 |
+
old = kept[answer["qID"]]
|
| 1074 |
+
if answer["latency"] != old["latency"] or (answer["qID"] not in failed and answer != old):
|
| 1075 |
+
raise ValueError("healthy answer or measured latency changed")
|
| 1076 |
+
return {"blanked_qIDs": sorted(failed), "answer_sha256_before": original_sha,
|
| 1077 |
+
"answer_sha256_after": after["answer_sha256"],
|
| 1078 |
+
"successful_answer_contents_unchanged": True, "latencies_unchanged": True}
|
| 1079 |
+
|
| 1080 |
+
|
| 1081 |
+
def validate_question_event(event, expected_qids):
|
| 1082 |
+
if not isinstance(event, dict) or set(event) != {"qID", "stage", "exception_type", "exception_module", "message", "output_policy", "traceback"}:
|
| 1083 |
+
raise ValueError("unexpected trusted question event schema")
|
| 1084 |
+
if event["qID"] not in expected_qids or event["stage"] != "answer_loop_exception" or event["output_policy"] != "empty_content":
|
| 1085 |
+
raise ValueError("question event qID/stage/output policy differs")
|
| 1086 |
+
if any(not isinstance(event[k], str) for k in ("exception_type", "exception_module", "message")):
|
| 1087 |
+
raise ValueError("invalid per-question exception text")
|
| 1088 |
+
frames = event["traceback"]
|
| 1089 |
+
if not isinstance(frames, list) or not frames or any(not isinstance(t, dict) or set(t) != {"path", "function", "line"} or not isinstance(t["path"], str) or not isinstance(t["function"], str) or type(t["line"]) is not int for t in frames):
|
| 1090 |
+
raise ValueError("invalid per-question exception traceback")
|
| 1091 |
+
if frames[0]["path"] != "/app/solution/entrypoint/probe_entrypoint.py" or frames[0]["function"] != "answer_all":
|
| 1092 |
+
raise ValueError("question exception did not originate at the trusted answer-loop boundary")
|
| 1093 |
+
if event["exception_type"] == "ZipIntegrityError":
|
| 1094 |
+
raise ValueError("archive integrity error cannot be contained as a question failure")
|
| 1095 |
+
|
| 1096 |
+
|
| 1097 |
def execute(contract, *, input_root: Path, output_root: Path, evidence_root: Path | None = None, backend=None, setup_imports=False, production_invocation=False):
|
| 1098 |
"""Synthetic-fixture seam; production supplies only baked paths and launcher."""
|
| 1099 |
receipt = {"track": TRACK, "status": "failed", "source_validation": None,
|
|
|
|
| 1195 |
guard.violations.append({"key": "environment API", "operation": "guard replaced"})
|
| 1196 |
guard.final_check()
|
| 1197 |
receipt["configuration_violations"] = list(guard.violations)
|
| 1198 |
+
question_events = []
|
| 1199 |
if observations is not None:
|
| 1200 |
+
rows, _ = read_json(input_root / "request.json")
|
| 1201 |
+
rows = [rows] if isinstance(rows, dict) else rows
|
| 1202 |
+
expected_qids = {row["qID"] for row in rows}
|
| 1203 |
+
shell = sys.modules.get("solution.entrypoint.probe_entrypoint")
|
| 1204 |
+
staging = getattr(shell, "_ZIP_STAGING_EVENTS", None)
|
| 1205 |
+
if not isinstance(staging, list):
|
| 1206 |
+
raise ValueError("trusted shell staging ledger missing")
|
| 1207 |
+
ordinary = getattr(shell, "_QUESTION_FAILURE_EVENTS", None)
|
| 1208 |
+
fences = getattr(shell, "_MEDIA_FENCE_EVENTS", None)
|
| 1209 |
+
if not isinstance(ordinary, list) or not isinstance(fences, list):
|
| 1210 |
+
raise ValueError("trusted question/fence ledger missing")
|
| 1211 |
+
observations.reconcile_skipped(expected_qids, staging, ordinary, fences)
|
| 1212 |
+
for qid in receipt["input_validation"].get("unresolved_qIDs", []):
|
| 1213 |
+
if not any(event["qID"] == qid for event in observations.question_failures):
|
| 1214 |
+
observations.record_question_failure({"qID": qid, "stage": "input_media_resolution", "reason": "No unambiguous media candidate for request qID", "producer": {"function": "validate_inputs", "request_sha256": receipt["input_validation"]["request_sha256"]}})
|
| 1215 |
+
question_events = list(observations.question_failures)
|
| 1216 |
receipt["engine_observations"] = observations.finish(receipt["input_validation"]["requests"])
|
| 1217 |
if receipt["engine_observations"]["failures"]:
|
| 1218 |
raise ValueError("original engine initialization or answer failure")
|
| 1219 |
verify_sources(contract)
|
| 1220 |
if validate_inputs(input_root) != receipt["input_validation"]:
|
| 1221 |
raise ValueError("input metadata or layout changed during backend execution")
|
| 1222 |
+
receipt["question_failure_output"] = blank_failed_outputs(output_root, input_root, question_events)
|
| 1223 |
+
receipt["output_validation"] = validate_outputs(output_root, input_root, empty_qids={event["qID"] for event in question_events})
|
| 1224 |
check_input_binding(binding, policy)
|
| 1225 |
if guard.violations or receipt["degraded_markers"]:
|
| 1226 |
raise ValueError("backend degraded or attempted fixed configuration mutation")
|
platform_wrapper/probe_entrypoint.py
ADDED
|
@@ -0,0 +1,1118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Submission I/O shell — the live Grand-Challenge PROCEDURE contract.
|
| 2 |
+
|
| 3 |
+
The contract (live GC socket table, evidence-packet-sourced, never the template):
|
| 4 |
+
|
| 5 |
+
read /input/request.json Request rows (cardinality: see U2)
|
| 6 |
+
read /input/FO_definitions.json organizer FO taxonomy text (encoding: see ASM-4)
|
| 7 |
+
read /input/batch-videos.zip per-question clips, ZIP kind (layout: U1; delivery: U3)
|
| 8 |
+
write /output/answer.json list of {qID, content, latency} — the ONLY file
|
| 9 |
+
this shell ever leaves in /output
|
| 10 |
+
|
| 11 |
+
ARMED MODES, REFUSAL OTHERWISE. This shell runs only under a SANCTIONED arming
|
| 12 |
+
form and refuses everything else with exit code 2 and zero writes:
|
| 13 |
+
|
| 14 |
+
submission (production): `--submission` or SURGFIELD_SUBMISSION_MODE=1
|
| 15 |
+
— the container CMD's form; THIS is the upload path.
|
| 16 |
+
probe (rehearsal): `--probe` or SURGFIELD_PROBE_MODE=1
|
| 17 |
+
— interface rehearsals and gate fixtures; never
|
| 18 |
+
uploaded as a production CMD.
|
| 19 |
+
|
| 20 |
+
The two modes execute the identical contract; they differ only in the stderr
|
| 21 |
+
arming stamp. Any invocation carrying NO sanctioned form — bare, wrong flags,
|
| 22 |
+
misconfigured CMD — exits 2 before any I/O.
|
| 23 |
+
|
| 24 |
+
Behavioral guarantees:
|
| 25 |
+
1. `/output/answer.json` exists with FULL qID coverage from the moment the
|
| 26 |
+
request set is readable: a fallback answer per qID is PRE-SEEDED and each
|
| 27 |
+
computed answer atomically replaces its entry (progressive flush). A job
|
| 28 |
+
killed mid-run still leaves a complete, monotonically-improved answer file.
|
| 29 |
+
2. `/output` contains exactly `answer.json` after a clean exit; scratch lives
|
| 30 |
+
in a private work directory (the atomic-replace temp is consumed by every
|
| 31 |
+
flush).
|
| 32 |
+
3. One `G1DECODE qID=<q> frames=<n> first_frame_sha256=<h> res=<WxH>` line per
|
| 33 |
+
qID on stderr — the decode-proof channel (external parse contract,
|
| 34 |
+
byte-stable). Hash = sha256 of the first decoded frame's raw bytes.
|
| 35 |
+
Decoding runs in a KILLABLE SUBPROCESS with a hard timeout so a wedged
|
| 36 |
+
video can never hang the job.
|
| 37 |
+
4. Time budget: the platform declares a 30 s/question unit rate with per-JOB
|
| 38 |
+
enforcement only (evidence pack, snapshot 2026-07-23T164642Z) —
|
| 39 |
+
operationally POOLED within a job. The shell budgets TOTAL job time and
|
| 40 |
+
logs pacing; it NEVER self-kills (the platform owns enforcement; a
|
| 41 |
+
self-kill could only forfeit answers the platform would have accepted).
|
| 42 |
+
5. No network. This shell downloads nothing; model weights resolve from the
|
| 43 |
+
image's local cache (revision pinning is a build-recipe property).
|
| 44 |
+
|
| 45 |
+
Interface unknowns, handled defensively where marked in-code:
|
| 46 |
+
U1 zip internal tree (flat / plain/ / nested / .MP4 / videoID-keyed)
|
| 47 |
+
U2 request.json cardinality (list vs single object)
|
| 48 |
+
U3 whether GC auto-extracts ZIP-kind sockets
|
| 49 |
+
ASM-4 FO_definitions.json encoding (JSON string vs structured JSON)
|
| 50 |
+
|
| 51 |
+
Supported invocation forms:
|
| 52 |
+
|
| 53 |
+
surgfield-entrypoint --submission ... (dev/rehearsal)
|
| 54 |
+
surgfield-probe --probe ... (rehearsal)
|
| 55 |
+
python -m solution.entrypoint.probe_entrypoint --submission|--probe ...
|
| 56 |
+
|
| 57 |
+
P9 (2026-07-31): the PRODUCTION container CMD is NOT this console script — it is the
|
| 58 |
+
stdlib-only import-death-proof launcher `python -m surgfield_shell.launch --submission`
|
| 59 |
+
(docker/Dockerfile.p4 ENTRYPOINT), which writes the never-empty floor BEFORE importing
|
| 60 |
+
this engine. The forms above eager-import the engine (F1 death risk) and are for
|
| 61 |
+
dev/rehearsal only.
|
| 62 |
+
|
| 63 |
+
Direct file-path invocation (`python .../probe_entrypoint.py`) is NOT a
|
| 64 |
+
supported form — the engine package is not importable from that sys.path.
|
| 65 |
+
Once armed, the outermost coverage net in `main()` guarantees a full-coverage
|
| 66 |
+
`answer.json` (loudly-degraded fallbacks, never a bare /output) under that
|
| 67 |
+
misuse or any other post-arming failure.
|
| 68 |
+
"""
|
| 69 |
+
from __future__ import annotations
|
| 70 |
+
|
| 71 |
+
import argparse
|
| 72 |
+
from contextlib import ExitStack
|
| 73 |
+
import json
|
| 74 |
+
import traceback
|
| 75 |
+
import math
|
| 76 |
+
import os
|
| 77 |
+
import re
|
| 78 |
+
import subprocess
|
| 79 |
+
import sys
|
| 80 |
+
import tempfile
|
| 81 |
+
import time
|
| 82 |
+
import zipfile
|
| 83 |
+
from pathlib import Path
|
| 84 |
+
|
| 85 |
+
# The DECLARED unit rate (track page: "30 seconds per question"), applied as a
|
| 86 |
+
# POOLED per-job budget per the evidence pack's operational verdict. Monitoring
|
| 87 |
+
# only — see guarantee 4.
|
| 88 |
+
POOLED_SEC_PER_QUESTION = 30.0
|
| 89 |
+
|
| 90 |
+
# Hard ceiling for one first-frame decode subprocess. Generous against real
|
| 91 |
+
# clips (a first-frame read is sub-second); tight against a wedged demuxer.
|
| 92 |
+
DECODE_TIMEOUT_SEC = 20.0
|
| 93 |
+
|
| 94 |
+
# P1-04: total wall budget for the pre-answer decode-proof phase (cold cv2
|
| 95 |
+
# subprocesses). Bounds the O(N) probe cost inside the platform's setup window;
|
| 96 |
+
# past it, remaining clips skip the probe (the in-process decode is still
|
| 97 |
+
# SIGALRM-bounded per P1-03, so coverage is not a safety dependency).
|
| 98 |
+
PROBE_PHASE_BUDGET_SEC = 60.0
|
| 99 |
+
|
| 100 |
+
# Trusted structured events; only raw-ZIP materialization failures populate it.
|
| 101 |
+
_ZIP_STAGING_EVENTS: list[dict] = []
|
| 102 |
+
_QUESTION_FAILURE_EVENTS: list[dict] = []
|
| 103 |
+
_MEDIA_FENCE_EVENTS: list[dict] = []
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def _log(msg: str) -> None:
|
| 107 |
+
"""All shell logging goes to stderr: contract-neutral (GC ignores it,
|
| 108 |
+
harnesses parse it), and it keeps stdout free of accidental output."""
|
| 109 |
+
print(msg, file=sys.stderr, flush=True)
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def _safe_float(value, default: float = 0.0) -> float:
|
| 113 |
+
"""Never-raise numeric coercion for request-row fields. A malformed field
|
| 114 |
+
costs AT MOST its own row's fidelity — never the run (the B2-19-1 class:
|
| 115 |
+
field access outside a containment must not be able to raise). inf/nan
|
| 116 |
+
clamp to the default: a non-finite window boundary is malformed, not big."""
|
| 117 |
+
try:
|
| 118 |
+
f = float(value)
|
| 119 |
+
return f if math.isfinite(f) else default
|
| 120 |
+
except (TypeError, ValueError):
|
| 121 |
+
return default
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# Whole-video time-question marker (the Leg-B observability population): the
|
| 125 |
+
# boundary SEMANTICS stay held to the platform observer — the shell only
|
| 126 |
+
# refuses to hide the rows.
|
| 127 |
+
_TIME_Q = re.compile(
|
| 128 |
+
r"how long|how much time|at what time|when (was|is|did|does)|hh:mm:ss", re.I)
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
# ── request.json ─────────────────────────────────────────────────────────
|
| 132 |
+
|
| 133 |
+
def read_requests(path: Path) -> list[dict]:
|
| 134 |
+
"""Parse request.json into a list of row dicts.
|
| 135 |
+
|
| 136 |
+
U2 (cardinality UNKNOWN): the helptext suggests a single object, the
|
| 137 |
+
template a list. Accept BOTH: a dict is wrapped into a one-row list, a
|
| 138 |
+
list is taken as-is. The answer file is always emitted as a LIST — the
|
| 139 |
+
single-object output mirror is unconfirmed, and a list is the shape the
|
| 140 |
+
grader's `load_items` reads (observed tolerant, not contract-asserted).
|
| 141 |
+
"""
|
| 142 |
+
raw = json.loads(path.read_text(encoding="utf-8"))
|
| 143 |
+
if isinstance(raw, dict):
|
| 144 |
+
return [raw]
|
| 145 |
+
if isinstance(raw, list):
|
| 146 |
+
rows = [r for r in raw if isinstance(r, dict)]
|
| 147 |
+
if len(rows) != len(raw):
|
| 148 |
+
_log(f"PROBESHELL: skipping {len(raw) - len(rows)} non-object "
|
| 149 |
+
f"request row(s)")
|
| 150 |
+
return rows
|
| 151 |
+
raise ValueError(f"request.json: expected object or list, got {type(raw).__name__}")
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
# ── FO_definitions.json ──────────────────────────────────────────────────
|
| 155 |
+
|
| 156 |
+
def stage_fo_definitions(path: Path, workdir: Path) -> Path | None:
|
| 157 |
+
"""Decode the organizer FO-definitions socket to a plain-text file and
|
| 158 |
+
return its path (for SURGFIELD_FO_DEFINITIONS_FILE), or None if absent.
|
| 159 |
+
|
| 160 |
+
ASM-4 (string-primary, template-observed): the value is expected to be a
|
| 161 |
+
JSON-encoded plain-text STRING. Tolerate structured JSON too — serialized
|
| 162 |
+
back to text so the taxonomy's heading parser sees SOMETHING rather than
|
| 163 |
+
the shell failing. Never fatal: on any error the engine falls back to its
|
| 164 |
+
built-in class floor.
|
| 165 |
+
"""
|
| 166 |
+
if not path.is_file():
|
| 167 |
+
return None
|
| 168 |
+
try:
|
| 169 |
+
raw = json.loads(path.read_text(encoding="utf-8"))
|
| 170 |
+
text = raw if isinstance(raw, str) else json.dumps(raw, indent=1)
|
| 171 |
+
out = workdir / "FO_definitions.txt"
|
| 172 |
+
out.write_text(text, encoding="utf-8")
|
| 173 |
+
return out
|
| 174 |
+
except (ValueError, OSError) as e:
|
| 175 |
+
_log(f"PROBESHELL: FO_definitions unusable ({type(e).__name__}: {e}); "
|
| 176 |
+
f"engine uses its built-in taxonomy floor")
|
| 177 |
+
return None
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
# ── batch videos ─────────────────────────────────────────────────────────
|
| 181 |
+
|
| 182 |
+
# Video containers we recognize as clips. Lower-cased suffix match; the
|
| 183 |
+
# resolvers below also honor .MP4 (upper) via the engine's resolve_clip.
|
| 184 |
+
_VIDEO_EXTS = (".mp4", ".avi", ".mov", ".mkv", ".webm", ".m4v", ".mpg", ".mpeg")
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
def _log_input_tree(input_dir: Path) -> None:
|
| 188 |
+
"""DIAGNOSTIC (log-only, no behavior change): dump the full /input tree so
|
| 189 |
+
the NEXT live try-out definitively reveals GC's real batch-videos layout.
|
| 190 |
+
One `PROBESHELL: /input tree: ...` line per file (path + size). Total by
|
| 191 |
+
construction — any failure here degrades to a single note, never a raise."""
|
| 192 |
+
try:
|
| 193 |
+
files = sorted((p for p in input_dir.rglob("*") if p.is_file()),
|
| 194 |
+
key=lambda p: str(p))
|
| 195 |
+
_log(f"PROBESHELL: /input tree: {len(files)} file(s) under {input_dir}")
|
| 196 |
+
for p in files:
|
| 197 |
+
try:
|
| 198 |
+
size = p.stat().st_size
|
| 199 |
+
except OSError:
|
| 200 |
+
size = -1
|
| 201 |
+
_log(f"PROBESHELL: /input tree: {p} ({size}B)")
|
| 202 |
+
except Exception as e:
|
| 203 |
+
_log(f"PROBESHELL: /input tree unavailable ({type(e).__name__}: {e})")
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
def _extract_zip(zpath: Path, dest: Path) -> None:
|
| 207 |
+
"""Extract a ZIP into `dest` under the zip-slip guard.
|
| 208 |
+
|
| 209 |
+
SYMLINK-SAFETY IS LOAD-BEARING HERE: stdlib zipfile.extract writes every
|
| 210 |
+
member as a REGULAR FILE — it never honors a symlink attribute — so a
|
| 211 |
+
malicious link member cannot escape `dest`. If this is ever swapped for
|
| 212 |
+
another extractor (shutil.unpack_archive, an external tool), that guarantee
|
| 213 |
+
disappears and the escape reopens; re-prove before swapping."""
|
| 214 |
+
dest.mkdir(parents=True, exist_ok=True)
|
| 215 |
+
skipped = 0
|
| 216 |
+
with zipfile.ZipFile(zpath) as zf:
|
| 217 |
+
for info in zf.infolist():
|
| 218 |
+
name = info.filename
|
| 219 |
+
# zip-slip guard: refuse absolute paths and parent escapes.
|
| 220 |
+
if name.startswith(("/", "\\")) or ".." in Path(name).parts:
|
| 221 |
+
_log(f"PROBESHELL: skipping unsafe zip member {name!r}")
|
| 222 |
+
continue
|
| 223 |
+
# P1-19: PER-MEMBER isolation — a single bad member (BadZipFile / CRC
|
| 224 |
+
# / ENOSPC) must NOT propagate to the batch-wide except and zero the
|
| 225 |
+
# video signal for EVERY question. Log+skip the failing member; keep
|
| 226 |
+
# the rest of the archive.
|
| 227 |
+
try:
|
| 228 |
+
zf.extract(info, dest)
|
| 229 |
+
except Exception as e:
|
| 230 |
+
skipped += 1
|
| 231 |
+
_log(f"PROBESHELL: skipping unextractable zip member {name!r} "
|
| 232 |
+
f"({type(e).__name__}: {e}); other members still staged")
|
| 233 |
+
if skipped:
|
| 234 |
+
_log(f"PROBESHELL: WARN {skipped} zip member(s) failed to extract "
|
| 235 |
+
f"from {zpath.name}; the rest were staged (per-member isolation)")
|
| 236 |
+
|
| 237 |
+
|
| 238 |
+
def _is_zip(path: Path) -> bool:
|
| 239 |
+
"""Content-sniff a regular file for the ZIP signature. Extension-agnostic —
|
| 240 |
+
GC has delivered the archive as `batch-videos` (no `.zip`), so we must NOT
|
| 241 |
+
rely on the name. `zipfile.is_zipfile` reads only the end-of-central-dir
|
| 242 |
+
record, so it is cheap even on a large non-zip (a real .mp4 => False)."""
|
| 243 |
+
try:
|
| 244 |
+
return zipfile.is_zipfile(path)
|
| 245 |
+
except OSError:
|
| 246 |
+
return False
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
def stage_videos_root(input_dir: Path, workdir: Path) -> Path | None:
|
| 250 |
+
"""Locate the clips ROBUSTLY across GC's actual placement: return a single
|
| 251 |
+
directory to search, or None if genuinely no videos are present.
|
| 252 |
+
|
| 253 |
+
Round-1/try-out RCA (2026-07-25): GC put the `batch-videos` input at a path
|
| 254 |
+
OTHER than `/input/batch-videos.zip` and OTHER than `/input/batch-videos/`,
|
| 255 |
+
so the old exact-path resolver deferred every question (frames=0). This
|
| 256 |
+
resolver instead scans `/input` RECURSIVELY and accepts, order-deterministic:
|
| 257 |
+
|
| 258 |
+
(a) ANY ZIP found anywhere under /input — matched by CONTENT, not name
|
| 259 |
+
(handles `batch-videos` with no extension, or a nested
|
| 260 |
+
`.../<uuid>/batch-videos.zip`) — extracted into the work dir; AND
|
| 261 |
+
(b) ANY loose video file (`*.mp4`/`*.avi`/…) anywhere under /input —
|
| 262 |
+
symlinked into the same work dir PRESERVING its path relative to
|
| 263 |
+
/input, so a `plain/` sub-tree is kept intact for the qID resolver.
|
| 264 |
+
|
| 265 |
+
Everything lands under one `<work>/videos` root (never /output); the
|
| 266 |
+
existing qID→clip resolver (`normalize_clips`) then runs over it unchanged.
|
| 267 |
+
The zip-slip guard is preserved (`_extract_zip`). If NOTHING resolves we
|
| 268 |
+
keep the prior defer-to-rules behavior — never crash, never a bare output.
|
| 269 |
+
|
| 270 |
+
Regression: the known-good contract (`/input/batch-videos.zip` with
|
| 271 |
+
`plain/<qID>.mp4`, the P6 harness staging) is a strict subset of case (a)
|
| 272 |
+
and stages to `<work>/videos/plain/<qID>.mp4` exactly as before.
|
| 273 |
+
"""
|
| 274 |
+
dest = workdir / "videos"
|
| 275 |
+
files = sorted((p for p in input_dir.rglob("*") if p.is_file()),
|
| 276 |
+
key=lambda p: str(p))
|
| 277 |
+
zips = []
|
| 278 |
+
loose_videos = []
|
| 279 |
+
for f in files:
|
| 280 |
+
if f.suffix.lower() in _VIDEO_EXTS:
|
| 281 |
+
loose_videos.append(f)
|
| 282 |
+
elif _is_zip(f):
|
| 283 |
+
zips.append(f)
|
| 284 |
+
|
| 285 |
+
staged_any = False
|
| 286 |
+
for z in zips: # deterministic (files is sorted)
|
| 287 |
+
_log(f"PROBESHELL: extracting archive {z} ({z.stat().st_size}B)")
|
| 288 |
+
_extract_zip(z, dest)
|
| 289 |
+
staged_any = True
|
| 290 |
+
for v in loose_videos: # deterministic (files is sorted)
|
| 291 |
+
rel = v.relative_to(input_dir)
|
| 292 |
+
link = dest / rel
|
| 293 |
+
link.parent.mkdir(parents=True, exist_ok=True)
|
| 294 |
+
if not link.exists():
|
| 295 |
+
link.symlink_to(v.resolve())
|
| 296 |
+
staged_any = True
|
| 297 |
+
|
| 298 |
+
if staged_any:
|
| 299 |
+
_log(f"PROBESHELL: staged videos root {dest} "
|
| 300 |
+
f"({len(zips)} archive(s), {len(loose_videos)} loose clip(s))")
|
| 301 |
+
return dest
|
| 302 |
+
_log("PROBESHELL: no batch-videos archive or video files found anywhere "
|
| 303 |
+
"under /input; video-dependent layers will defer on every question")
|
| 304 |
+
return None
|
| 305 |
+
|
| 306 |
+
|
| 307 |
+
def _resolve_deterministic(root: Path, key: str):
|
| 308 |
+
"""Recursive, ORDER-DETERMINISTIC clip lookup by stem == `key`, preferring a
|
| 309 |
+
`plain/` path over `overlayed/` (we tuned on plain), then the shortest path,
|
| 310 |
+
then lexicographic. Complements the engine's tolerant resolvers for the
|
| 311 |
+
arbitrarily-nested trees GC may hand us. Returns a Path or None."""
|
| 312 |
+
hits = [p for p in root.rglob("*")
|
| 313 |
+
if p.is_file() and p.stem == key and p.suffix.lower() in _VIDEO_EXTS]
|
| 314 |
+
|
| 315 |
+
def sort_key(p: Path):
|
| 316 |
+
parts = [seg.lower() for seg in p.parts]
|
| 317 |
+
return (0 if "plain" in parts else 1,
|
| 318 |
+
1 if "overlayed" in parts else 0,
|
| 319 |
+
len(p.parts), str(p))
|
| 320 |
+
|
| 321 |
+
hits.sort(key=sort_key)
|
| 322 |
+
return hits[0] if hits else None
|
| 323 |
+
|
| 324 |
+
|
| 325 |
+
# qID→clip resolution ladder — DOCUMENTED layout first, then bounded fallbacks.
|
| 326 |
+
# Each rung is labeled so the run log shows exactly which layout GC delivered
|
| 327 |
+
# (observe-don't-assume: the try-out enumeration + these lines finally pin the
|
| 328 |
+
# real tree). The template's reference `inference.py` reads `/input/plain/<qID>.mp4`
|
| 329 |
+
# (+ `overlayed/` with a burned-in HH:MM:SS clock).
|
| 330 |
+
# P9 (2026-07-31) — RCA drift corrected: `plain/` is the PRIMARY rung because WE
|
| 331 |
+
# unzip the `batch-videos` ZIP into a plain layout (stage_videos_root), NOT because
|
| 332 |
+
# the platform auto-extracts it. The team's OBSERVED_ON_PLATFORM note recorded the
|
| 333 |
+
# OPPOSITE (the container sees the ZIP as a FILE; no auto-extraction), so the earlier
|
| 334 |
+
# "platform auto-extracts to plain/" premise here was FALSE and is struck. The real
|
| 335 |
+
# /input tree is still un-captured; `_log_input_tree` (see main) emits it verbatim on
|
| 336 |
+
# the next live run so the ladder can finally be verified against ground truth. The
|
| 337 |
+
# rest are known-plausible fallbacks in priority order.
|
| 338 |
+
_RUNG_PRIMARY = "plain" # documented primary: <root>/plain/<qID>.mp4
|
| 339 |
+
_RUNG_OVERLAYED = "overlayed" # documented alt: <root>/overlayed/<qID>.mp4
|
| 340 |
+
_RUNG_FLAT = "flat" # bare root: <root>/<qID>.mp4
|
| 341 |
+
_RUNG_NESTED = "nested-stem" # recursive stem==qID (plain-preferring)
|
| 342 |
+
_RUNG_VIDEOID = "videoID" # keyed by row.videoID (qIDs may share a clip)
|
| 343 |
+
_RUNG_UNRESOLVED = "UNRESOLVED"
|
| 344 |
+
|
| 345 |
+
|
| 346 |
+
def _resolve_qid(root: Path, qid: str, video_id, resolve_video):
|
| 347 |
+
"""Resolve ONE qID to a clip across the ordered ladder. Returns
|
| 348 |
+
`(Path|None, rung_label)`. Rungs are tried in strict priority order so the
|
| 349 |
+
documented layout always wins and any fallback is visible in the label.
|
| 350 |
+
|
| 351 |
+
The primary/overlayed/flat rungs reimplement the engine's `resolve_clip`
|
| 352 |
+
path checks SPLIT BY RUNG (so each is separately labeled); the P6 known-good
|
| 353 |
+
tree (`plain/<qID>.mp4`) resolves on rung 1 exactly as before — no mapping
|
| 354 |
+
change. Rungs 4–5 reuse the deterministic recursive + engine resolvers.
|
| 355 |
+
"""
|
| 356 |
+
q = str(qid)
|
| 357 |
+
for ext in (".mp4", ".MP4"):
|
| 358 |
+
c = root / "plain" / f"{q}{ext}"
|
| 359 |
+
if c.is_file():
|
| 360 |
+
return c, _RUNG_PRIMARY
|
| 361 |
+
for ext in (".mp4", ".MP4"):
|
| 362 |
+
c = root / "overlayed" / f"{q}{ext}"
|
| 363 |
+
if c.is_file():
|
| 364 |
+
return c, _RUNG_OVERLAYED
|
| 365 |
+
for ext in (".mp4", ".MP4"):
|
| 366 |
+
c = root / f"{q}{ext}"
|
| 367 |
+
if c.is_file():
|
| 368 |
+
return c, _RUNG_FLAT
|
| 369 |
+
c = _resolve_deterministic(root, q)
|
| 370 |
+
if c is not None:
|
| 371 |
+
return c, _RUNG_NESTED
|
| 372 |
+
if video_id:
|
| 373 |
+
vid = str(video_id)
|
| 374 |
+
c = _resolve_deterministic(root, vid)
|
| 375 |
+
if c is not None and Path(c).is_file():
|
| 376 |
+
return c, _RUNG_VIDEOID
|
| 377 |
+
return None, _RUNG_UNRESOLVED
|
| 378 |
+
|
| 379 |
+
|
| 380 |
+
def normalize_clips(rows: list[dict], root: Path | None, workdir: Path) -> Path:
|
| 381 |
+
"""Build `<work>/norm/plain/<qID>.mp4` links so the engine's clip-mode
|
| 382 |
+
lookup is deterministic regardless of the delivered tree, resolving each
|
| 383 |
+
qID through the labeled ladder in `_resolve_qid`.
|
| 384 |
+
|
| 385 |
+
OBSERVABILITY (robust-with-visibility, non-negotiable): one `RESOLVE` line
|
| 386 |
+
per qID naming the rung that fired; a loud `RESOLVE-FALLBACK` when a qID
|
| 387 |
+
resolves past the documented primary; a `RESOLVE-SUMMARY` rung histogram;
|
| 388 |
+
and a loud `RESOLVE-FAIL` naming every qID with NO clip. An unresolved qID
|
| 389 |
+
still gets no link — the engine defers and the rule layers answer it — but
|
| 390 |
+
it is never silent (Round-1 died on silent tolerant-guessing).
|
| 391 |
+
"""
|
| 392 |
+
from solution.frame_sampler import resolve_video
|
| 393 |
+
|
| 394 |
+
norm = workdir / "norm" / "plain"
|
| 395 |
+
norm.mkdir(parents=True, exist_ok=True)
|
| 396 |
+
if root is None:
|
| 397 |
+
_log("PROBESHELL: RESOLVE-SUMMARY no videos root; every qID defers to rules")
|
| 398 |
+
return norm.parent
|
| 399 |
+
|
| 400 |
+
rung_counts: dict[str, int] = {}
|
| 401 |
+
unresolved: list[str] = []
|
| 402 |
+
# P1-18: qIDs resolved via the videoID rung map to a SHARED, un-trimmed
|
| 403 |
+
# video (not a per-qID clip). Clip-mode whole_clips from frame 0 and would
|
| 404 |
+
# give every qID sharing that video identical, wrong-window frames. Record
|
| 405 |
+
# them so the engine SEEKS the row's [start,end] instead of whole-clipping.
|
| 406 |
+
videoid_qids: list[str] = []
|
| 407 |
+
for row in rows:
|
| 408 |
+
qid = str(row.get("qID", ""))
|
| 409 |
+
if not qid:
|
| 410 |
+
continue
|
| 411 |
+
src, rung = _resolve_qid(root, qid, row.get("videoID"), resolve_video)
|
| 412 |
+
rung_counts[rung] = rung_counts.get(rung, 0) + 1
|
| 413 |
+
if src is None:
|
| 414 |
+
unresolved.append(qid)
|
| 415 |
+
_log(f"PROBESHELL: RESOLVE qID={qid} rung={_RUNG_UNRESOLVED} "
|
| 416 |
+
f"(no clip anywhere under videos root; defers to rules)")
|
| 417 |
+
continue
|
| 418 |
+
if rung == _RUNG_VIDEOID:
|
| 419 |
+
videoid_qids.append(qid)
|
| 420 |
+
if rung != _RUNG_PRIMARY:
|
| 421 |
+
_log(f"PROBESHELL: RESOLVE-FALLBACK qID={qid} rung={rung} src={src} "
|
| 422 |
+
f"(documented plain/{qid}.mp4 absent — resolved via fallback)")
|
| 423 |
+
else:
|
| 424 |
+
_log(f"PROBESHELL: RESOLVE qID={qid} rung={rung} src={src}")
|
| 425 |
+
link = norm / f"{qid}.mp4"
|
| 426 |
+
if not link.exists():
|
| 427 |
+
link.symlink_to(Path(src).resolve())
|
| 428 |
+
|
| 429 |
+
# P1-18: persist the videoID-rung set next to the normalized tree so the
|
| 430 |
+
# clip-mode engine can pick SEEK-with-window for exactly these qIDs.
|
| 431 |
+
marker = norm.parent / "videoid_rung.json"
|
| 432 |
+
try:
|
| 433 |
+
marker.write_text(json.dumps(sorted(videoid_qids)))
|
| 434 |
+
except OSError as e:
|
| 435 |
+
_log(f"PROBESHELL: could not write videoid_rung marker ({e})")
|
| 436 |
+
if videoid_qids:
|
| 437 |
+
_log(f"PROBESHELL: RESOLVE-VIDEOID {len(videoid_qids)} qID(s) share an "
|
| 438 |
+
f"un-trimmed video — engine will SEEK [start,end], not whole-clip")
|
| 439 |
+
|
| 440 |
+
_log(f"PROBESHELL: RESOLVE-SUMMARY rungs={rung_counts} "
|
| 441 |
+
f"resolved={sum(rung_counts.values()) - len(unresolved)} "
|
| 442 |
+
f"unresolved={len(unresolved)}")
|
| 443 |
+
if unresolved:
|
| 444 |
+
shown = unresolved[:20]
|
| 445 |
+
_log(f"PROBESHELL: RESOLVE-FAIL {len(unresolved)} qID(s) had NO clip: "
|
| 446 |
+
f"{shown}{' …' if len(unresolved) > 20 else ''} — video-blind on these")
|
| 447 |
+
return norm.parent
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
# ── decode-proof channel ─────────────────────────────────────────────────
|
| 451 |
+
|
| 452 |
+
# Self-contained first-frame prober run in a KILLABLE subprocess (production
|
| 453 |
+
# hardening): cv2 can wedge inside C on a malformed container, beyond any
|
| 454 |
+
# in-process timeout. The snippet prints one JSON line; anything else (crash,
|
| 455 |
+
# hang->kill, no frame) degrades to the frames=0 record. The emitted G1DECODE
|
| 456 |
+
# line stays byte-stable.
|
| 457 |
+
#
|
| 458 |
+
# P1-07: the child prints the sentinel ``CV2_INIT_OK`` on stderr AFTER cv2
|
| 459 |
+
# imports+constructs successfully. That lets the parent tell a clip-specific
|
| 460 |
+
# decoder crash (sentinel present, then rc!=0) from a SYSTEMIC child-env fault
|
| 461 |
+
# (no sentinel: missing libGL, cv2 import failure, OOM-kill of the child) — the
|
| 462 |
+
# latter fires for EVERY clip and must NOT unlink every clip (the Round-1
|
| 463 |
+
# frames=0 collapse).
|
| 464 |
+
_DECODE_SNIPPET = (
|
| 465 |
+
"import hashlib, json, sys\n"
|
| 466 |
+
"try:\n"
|
| 467 |
+
" import cv2\n"
|
| 468 |
+
" cap = cv2.VideoCapture(sys.argv[1])\n"
|
| 469 |
+
"except Exception:\n"
|
| 470 |
+
" sys.exit(3)\n"
|
| 471 |
+
"sys.stderr.write('CV2_INIT_OK')\n"
|
| 472 |
+
"ok, frame = cap.read()\n"
|
| 473 |
+
"cap.release()\n"
|
| 474 |
+
"if ok and frame is not None:\n"
|
| 475 |
+
" print(json.dumps({'sha': hashlib.sha256(frame.tobytes()).hexdigest(),\n"
|
| 476 |
+
" 'res': str(frame.shape[1]) + 'x' + str(frame.shape[0])}))\n"
|
| 477 |
+
)
|
| 478 |
+
|
| 479 |
+
|
| 480 |
+
def _probe_first_frame(clip: Path) -> tuple[str, int, str, str | None]:
|
| 481 |
+
"""(sha, frames, res, danger) for one clip via the killable subprocess prober.
|
| 482 |
+
|
| 483 |
+
`danger` classifies the prober outcome for the H6 in-process fence:
|
| 484 |
+
- "WEDGE" → subprocess TimeoutExpired (the child hung on decode); the
|
| 485 |
+
in-process engine would hang on the same clip.
|
| 486 |
+
- "CRASH" → cv2 INITIALIZED (sentinel seen) then the child exited
|
| 487 |
+
non-zero on THIS clip (segfault/OOM/abort in the decoder).
|
| 488 |
+
- "ENVFAULT" → rc!=0 with NO sentinel: the child could not even init cv2
|
| 489 |
+
(P1-07 — a SYSTEMIC env fault, not a clip crash). Reported
|
| 490 |
+
so the caller can ABORT fencing instead of unlinking every
|
| 491 |
+
clip and going whole-pipeline video-blind.
|
| 492 |
+
- None → SAFE: clean rc==0 (frame-0 decoded OR cleanly undecodable),
|
| 493 |
+
an unparseable-stdout rc==0, or a subprocess spawn error.
|
| 494 |
+
A clean frame-0 miss is NOT dangerous — frame_sampler skips
|
| 495 |
+
an unreadable frame and answers from the rest.
|
| 496 |
+
"""
|
| 497 |
+
try:
|
| 498 |
+
r = subprocess.run([sys.executable, "-c", _DECODE_SNIPPET, str(clip)],
|
| 499 |
+
capture_output=True, text=True,
|
| 500 |
+
timeout=DECODE_TIMEOUT_SEC)
|
| 501 |
+
if r.returncode != 0:
|
| 502 |
+
if "CV2_INIT_OK" not in (r.stderr or ""):
|
| 503 |
+
# P1-07: child never initialized cv2 → systemic env fault, not a
|
| 504 |
+
# clip-specific crash. Do NOT fence (would blind-unlink all clips).
|
| 505 |
+
_log(f"PROBESHELL: decode probe ENV-FAULT for {clip.name} "
|
| 506 |
+
f"(rc={r.returncode}, cv2 never initialized); NOT fencing this clip")
|
| 507 |
+
return "none", 0, "0x0", "ENVFAULT"
|
| 508 |
+
_log(f"PROBESHELL: decode probe crashed for {clip.name} "
|
| 509 |
+
f"(returncode={r.returncode}); recording frames=0 and fencing")
|
| 510 |
+
return "none", 0, "0x0", "CRASH"
|
| 511 |
+
if r.stdout.strip():
|
| 512 |
+
d = json.loads(r.stdout.strip().splitlines()[-1])
|
| 513 |
+
return d["sha"], 1, d["res"], None
|
| 514 |
+
except subprocess.TimeoutExpired:
|
| 515 |
+
_log(f"PROBESHELL: decode timeout ({DECODE_TIMEOUT_SEC:.0f}s) for "
|
| 516 |
+
f"{clip.name}; recording frames=0 and fencing")
|
| 517 |
+
return "none", 0, "0x0", "WEDGE"
|
| 518 |
+
except Exception as e:
|
| 519 |
+
_log(f"PROBESHELL: decode probe failed for {clip.name} "
|
| 520 |
+
f"({type(e).__name__}: {e})")
|
| 521 |
+
return "none", 0, "0x0", None
|
| 522 |
+
|
| 523 |
+
|
| 524 |
+
def _fence_decision(to_fence: dict[str, str], n_probed: int):
|
| 525 |
+
"""P1-07: decide which qIDs to fence, and whether the pattern is SYSTEMIC.
|
| 526 |
+
|
| 527 |
+
Returns ``(systemic, env_faults, dangers)``:
|
| 528 |
+
- ``env_faults`` — qIDs whose child never initialized cv2 (env fault).
|
| 529 |
+
- ``dangers`` — qIDs with a real per-clip WEDGE/CRASH.
|
| 530 |
+
- ``systemic`` — True if ANY env fault, OR every probed clip came back
|
| 531 |
+
dangerous (>1). When systemic, the caller must NOT
|
| 532 |
+
unlink clips (would blind the whole pipeline to the
|
| 533 |
+
rules floor at exit 0); the in-process SIGALRM decode
|
| 534 |
+
deadline provides containment instead.
|
| 535 |
+
"""
|
| 536 |
+
env_faults = {q for q, r in to_fence.items() if r == "ENVFAULT"}
|
| 537 |
+
dangers = {q: r for q, r in to_fence.items() if r in ("WEDGE", "CRASH")}
|
| 538 |
+
systemic = bool(env_faults) or (n_probed > 1 and len(dangers) == n_probed)
|
| 539 |
+
return systemic, env_faults, dangers
|
| 540 |
+
|
| 541 |
+
|
| 542 |
+
def emit_decode_records(rows: list[dict], norm_root: Path,
|
| 543 |
+
probe_budget_sec: float | None = None, zip_batch=None
|
| 544 |
+
) -> tuple[dict[str, str], int]:
|
| 545 |
+
"""One G1DECODE stderr line per UNIQUE qID — the decode-proof channel.
|
| 546 |
+
|
| 547 |
+
Hash convention (a parse contract with the external rehearsal harness —
|
| 548 |
+
never change it unilaterally): sha256 over the raw bytes of the FIRST
|
| 549 |
+
decoded frame (`frame.tobytes()`). `frames` counts what THIS probe decoded
|
| 550 |
+
(telemetry — the assertion compares the hash). Absent/undecodable/timed-out
|
| 551 |
+
clips emit `frames=0 first_frame_sha256=none res=0x0` rather than nothing,
|
| 552 |
+
so coverage of the channel itself is checkable.
|
| 553 |
+
|
| 554 |
+
P1-04: each cold cv2 subprocess is charged nowhere in the pooled budget, so
|
| 555 |
+
this phase (a) probes each unique qID ONCE (no per-duplicate-row re-decode),
|
| 556 |
+
(b) caches by resolved clip target so qIDs SHARING a clip decode once, and
|
| 557 |
+
(c) stops probing once `probe_budget_sec` is exhausted (remaining clips are
|
| 558 |
+
still safe — the in-process decode is SIGALRM-bounded per P1-03).
|
| 559 |
+
|
| 560 |
+
Returns `(to_fence, n_probed)`:
|
| 561 |
+
- `to_fence` = {qID: "WEDGE"|"CRASH"|"ENVFAULT"} — qIDs whose prober hit a
|
| 562 |
+
signal. The caller fences WEDGE/CRASH; ENVFAULT (systemic, P1-07) is
|
| 563 |
+
surfaced but NOT unlinked.
|
| 564 |
+
- `n_probed` = number of clips actually decoded this phase (for the caller's
|
| 565 |
+
systemic-pattern guard: all-probed-clips-dangerous ⇒ do not fence).
|
| 566 |
+
"""
|
| 567 |
+
if zip_batch is not None:
|
| 568 |
+
return _emit_zip_decode_records(rows, norm_root, zip_batch, probe_budget_sec)
|
| 569 |
+
to_fence: dict[str, str] = {}
|
| 570 |
+
seen_q: set[str] = set()
|
| 571 |
+
by_target: dict[str, tuple] = {}
|
| 572 |
+
n_probed = 0
|
| 573 |
+
t0 = time.perf_counter()
|
| 574 |
+
budget_logged = False
|
| 575 |
+
for row in rows:
|
| 576 |
+
qid = str(row.get("qID", ""))
|
| 577 |
+
if not qid or qid in seen_q:
|
| 578 |
+
continue
|
| 579 |
+
seen_q.add(qid)
|
| 580 |
+
clip = norm_root / "plain" / f"{qid}.mp4"
|
| 581 |
+
sha, frames, res, danger = ("none", 0, "0x0", None)
|
| 582 |
+
if clip.is_file():
|
| 583 |
+
key = os.path.realpath(clip)
|
| 584 |
+
if key in by_target: # shared clip: decode once
|
| 585 |
+
sha, frames, res, danger = by_target[key]
|
| 586 |
+
elif (probe_budget_sec is not None
|
| 587 |
+
and time.perf_counter() - t0 > probe_budget_sec):
|
| 588 |
+
if not budget_logged:
|
| 589 |
+
budget_logged = True
|
| 590 |
+
_log(f"PROBESHELL: decode-proof budget ({probe_budget_sec:.0f}s) "
|
| 591 |
+
f"exhausted; skipping probe for remaining clips (the "
|
| 592 |
+
f"in-process decode is SIGALRM-bounded, so still safe)")
|
| 593 |
+
else:
|
| 594 |
+
sha, frames, res, danger = _probe_first_frame(clip)
|
| 595 |
+
by_target[key] = (sha, frames, res, danger)
|
| 596 |
+
n_probed += 1
|
| 597 |
+
if danger:
|
| 598 |
+
to_fence[qid] = danger
|
| 599 |
+
_log(f"G1DECODE qID={qid} frames={frames} first_frame_sha256={sha} res={res}")
|
| 600 |
+
return to_fence, n_probed
|
| 601 |
+
|
| 602 |
+
|
| 603 |
+
# ── answers: pre-seed, compute, flush ────────────────────────────────────
|
| 604 |
+
|
| 605 |
+
def _record_zip_failure(zip_batch, qid, phase, error):
|
| 606 |
+
event = zip_batch.record_failure(qid, phase, error)
|
| 607 |
+
_log("ZIP_STAGING_FAILURE " + json.dumps(event, sort_keys=True))
|
| 608 |
+
|
| 609 |
+
|
| 610 |
+
def _emit_zip_decode_records(rows, norm_root, zip_batch, probe_budget_sec):
|
| 611 |
+
from solution.entrypoint.segment_zip_input import MemberStagingError
|
| 612 |
+
to_fence, seen_q, by_target = {}, set(), {}
|
| 613 |
+
n_probed, t0, budget_logged = 0, time.perf_counter(), False
|
| 614 |
+
for row in rows:
|
| 615 |
+
qid = str(row.get("qID", ""))
|
| 616 |
+
if not qid or qid in seen_q:
|
| 617 |
+
continue
|
| 618 |
+
seen_q.add(qid)
|
| 619 |
+
sha, frames, res, danger = "none", 0, "0x0", None
|
| 620 |
+
key = zip_batch.cache_key(qid)
|
| 621 |
+
if key in by_target:
|
| 622 |
+
sha, frames, res, danger = by_target[key]
|
| 623 |
+
elif probe_budget_sec is not None and time.perf_counter() - t0 > probe_budget_sec:
|
| 624 |
+
if not budget_logged:
|
| 625 |
+
_log(f"PROBESHELL: decode-proof budget ({probe_budget_sec:.0f}s) exhausted; remaining ZIP clips unprobed")
|
| 626 |
+
budget_logged = True
|
| 627 |
+
else:
|
| 628 |
+
with ExitStack() as clips:
|
| 629 |
+
try:
|
| 630 |
+
clip = clips.enter_context(zip_batch.member(qid, phase="startup_probe"))
|
| 631 |
+
except MemberStagingError as error:
|
| 632 |
+
_record_zip_failure(zip_batch, qid, "startup_probe", error)
|
| 633 |
+
else:
|
| 634 |
+
# Exception classification ended before invoking the decoder.
|
| 635 |
+
if clip.is_file():
|
| 636 |
+
sha, frames, res, danger = _probe_first_frame(clip)
|
| 637 |
+
by_target[key] = (sha, frames, res, danger)
|
| 638 |
+
n_probed += 1
|
| 639 |
+
if danger:
|
| 640 |
+
to_fence[qid] = danger
|
| 641 |
+
_log(f"G1DECODE qID={qid} frames={frames} first_frame_sha256={sha} res={res}")
|
| 642 |
+
return to_fence, n_probed
|
| 643 |
+
|
| 644 |
+
|
| 645 |
+
def write_answers(output_dir: Path, answers: list[dict]) -> Path:
|
| 646 |
+
"""Atomic (re)write of /output/answer.json: temp-in-/output + os.replace
|
| 647 |
+
(same filesystem => atomic; the temp is consumed by every flush). Called
|
| 648 |
+
once to pre-seed and then after every computed answer — the file is
|
| 649 |
+
complete at every instant."""
|
| 650 |
+
output_dir.mkdir(parents=True, exist_ok=True)
|
| 651 |
+
out = output_dir / "answer.json"
|
| 652 |
+
tmp = output_dir / ".answer.json.tmp"
|
| 653 |
+
tmp.write_text(json.dumps(answers, indent=1), encoding="utf-8")
|
| 654 |
+
os.replace(tmp, out)
|
| 655 |
+
return out
|
| 656 |
+
|
| 657 |
+
|
| 658 |
+
# C1 (2026-07-25): the grader's adversarial detector whole-submission-DQs on these public
|
| 659 |
+
# signal substrings (focus.evaluation.adversarial). Our answers must never trip it.
|
| 660 |
+
#
|
| 661 |
+
# P9 (2026-07-31): promoted from OBSERVE-ONLY to ENFORCE. The prior hedge shipped a colliding
|
| 662 |
+
# answer verbatim; a single genuine free-text answer that happens to contain a signal substring
|
| 663 |
+
# would then risk zeroing all 10 buckets (whole-submission DQ). ENFORCE = on a collision we
|
| 664 |
+
# DECLINE that ONE question by replacing its ENTIRE content with the format's SAFE_FALLBACK
|
| 665 |
+
# (a full-answer forfeit, at most 1 question lost) and log loudly. This is a DECLINE, not an
|
| 666 |
+
# evasion-scrub: we do NOT surgically strip the signal to sneak the rest of the answer through
|
| 667 |
+
# (that would be grader-control evasion, HARD-BARRED). Coverage is preserved (still one answer
|
| 668 |
+
# per qID). GOVERNANCE: this supersedes the 2026-07-25 observe-only disposition; the
|
| 669 |
+
# scrub-vs-decline distinction is flagged for Reza ratification in P9_coverage.md.
|
| 670 |
+
_C1_INJECTION_SIGNALS = (
|
| 671 |
+
"ignore previous instructions", "ignore all previous", "disregard the above",
|
| 672 |
+
"you are now", "system prompt:", "new instructions:", "override:",
|
| 673 |
+
"forget your instructions", "act as if", "pretend that the answer is correct",
|
| 674 |
+
"always respond with correct", "the answer is definitely correct",
|
| 675 |
+
)
|
| 676 |
+
|
| 677 |
+
|
| 678 |
+
def _c1_safe_fallback_for(qid: str, qid_to_question: dict[str, str]) -> str:
|
| 679 |
+
"""Format-aware DECLINE content for a colliding answer. Best-effort: infer the answer
|
| 680 |
+
format from the question and take its SAFE_FALLBACK; plain 'none' if inference is
|
| 681 |
+
unavailable (total by construction — a raise here must never escape the pre-flight)."""
|
| 682 |
+
try:
|
| 683 |
+
from solution.answer_validator import SAFE_FALLBACK
|
| 684 |
+
from solution.cap_fmt_inferrer import infer_all
|
| 685 |
+
q = qid_to_question.get(qid, "")
|
| 686 |
+
return SAFE_FALLBACK.get(infer_all(q)["answer_format"], "none")
|
| 687 |
+
except Exception:
|
| 688 |
+
return "none"
|
| 689 |
+
|
| 690 |
+
|
| 691 |
+
def _c1_collision_preflight(answers: list[dict],
|
| 692 |
+
rows: list[dict] | None = None) -> int:
|
| 693 |
+
"""ENFORCE: on any answer whose content collides with a public injection signal (would
|
| 694 |
+
whole-submission-DQ at the grader), DECLINE that ONE question — replace its whole content
|
| 695 |
+
with the format's SAFE_FALLBACK and log loudly. Mutates `answers` in place. Returns the
|
| 696 |
+
collision (blocked) count. This is a decline, never an evasion-scrub."""
|
| 697 |
+
qid_to_question = {str(r.get("qID", "")): str(r.get("question", ""))
|
| 698 |
+
for r in (rows or [])}
|
| 699 |
+
hits = 0
|
| 700 |
+
for a in answers:
|
| 701 |
+
content = str(a.get("content", "")).lower()
|
| 702 |
+
for sig in _C1_INJECTION_SIGNALS:
|
| 703 |
+
if sig in content:
|
| 704 |
+
hits += 1
|
| 705 |
+
qid = str(a.get("qID", ""))
|
| 706 |
+
safe = _c1_safe_fallback_for(qid, qid_to_question)
|
| 707 |
+
_log(f"C1-COLLISION-BLOCKED qID={qid} signal={sig!r} "
|
| 708 |
+
f"(answer would risk whole-submission DQ; DECLINED → content={safe!r})")
|
| 709 |
+
a["content"] = safe
|
| 710 |
+
break
|
| 711 |
+
if hits:
|
| 712 |
+
_log(f"C1-COLLISION-BLOCKED: {hits} answer(s) collided with the public injection-signal "
|
| 713 |
+
f"list — DECLINED (content replaced with SAFE_FALLBACK; coverage preserved)")
|
| 714 |
+
return hits
|
| 715 |
+
|
| 716 |
+
|
| 717 |
+
def fallback_answers(rows: list[dict]) -> list[dict]:
|
| 718 |
+
"""Full-coverage fallback list that does NOT require the engine package.
|
| 719 |
+
|
| 720 |
+
The PRE-SEED payload and the outermost coverage net's payload: it must work
|
| 721 |
+
even when `solution.*` itself is unimportable (e.g. the unsupported
|
| 722 |
+
file-path invocation), so the format-aware fallback is best-effort and the
|
| 723 |
+
floor is a plain "none" per unique qID. Degradation is loud on stderr and
|
| 724 |
+
fully countable downstream — nothing is masked.
|
| 725 |
+
"""
|
| 726 |
+
try:
|
| 727 |
+
from solution.answer_validator import SAFE_FALLBACK
|
| 728 |
+
from solution.cap_fmt_inferrer import infer_all
|
| 729 |
+
|
| 730 |
+
def content_for(question: str) -> str:
|
| 731 |
+
# Total by construction: this executes INSIDE the outermost except
|
| 732 |
+
# handler, where a raise would escape the net entirely. The
|
| 733 |
+
# format-aware attempt is best-effort; "none" is unconditional.
|
| 734 |
+
try:
|
| 735 |
+
return SAFE_FALLBACK.get(infer_all(question)["answer_format"], "none")
|
| 736 |
+
except Exception:
|
| 737 |
+
return "none"
|
| 738 |
+
except Exception:
|
| 739 |
+
def content_for(question: str) -> str:
|
| 740 |
+
return "none"
|
| 741 |
+
answers, seen = [], set()
|
| 742 |
+
for row in rows:
|
| 743 |
+
qid = str(row.get("qID", ""))
|
| 744 |
+
if not qid or qid in seen:
|
| 745 |
+
continue
|
| 746 |
+
seen.add(qid)
|
| 747 |
+
answers.append({"qID": qid,
|
| 748 |
+
"content": content_for(str(row.get("question", ""))),
|
| 749 |
+
"latency": 0.0})
|
| 750 |
+
return answers
|
| 751 |
+
|
| 752 |
+
|
| 753 |
+
def answer_all(rows: list[dict], videos_root: Path, device: str,
|
| 754 |
+
answers: list[dict], output_dir: Path,
|
| 755 |
+
proc_t0: float | None = None, zip_batch=None) -> list[dict]:
|
| 756 |
+
"""Upgrade the pre-seeded `answers` in place — one engine answer per unique
|
| 757 |
+
qID — flushing atomically after every question.
|
| 758 |
+
|
| 759 |
+
Failure containment is per-question: an ordinary exception inside one
|
| 760 |
+
answer records its origin and writes empty content for only that question. Latency
|
| 761 |
+
is the full per-question wall (infer -> answer -> validate).
|
| 762 |
+
|
| 763 |
+
Pooled-budget monitor (guarantee 4): total = 30 s x N; an overrun is logged
|
| 764 |
+
LOUDLY once and the run CONTINUES — enforcement is the platform's, and the
|
| 765 |
+
progressive flush means a platform kill costs only the not-yet-upgraded
|
| 766 |
+
answers, never the file.
|
| 767 |
+
"""
|
| 768 |
+
from solution.answer_validator import validate
|
| 769 |
+
from solution.cap_fmt_inferrer import infer_all
|
| 770 |
+
from solution.container import SubmissionContainer
|
| 771 |
+
from solution.question_context import Request
|
| 772 |
+
|
| 773 |
+
# P1-20: time the eager model/detector load (the largest setup sink) so a
|
| 774 |
+
# cold-start init overrun is visible and attributable, not silent.
|
| 775 |
+
_init_t0 = time.perf_counter()
|
| 776 |
+
container = SubmissionContainer(videos_dir=videos_root, device=device)
|
| 777 |
+
_init_wall = time.perf_counter() - _init_t0
|
| 778 |
+
_log(f"PROBESHELL: CONTAINER-INIT {_init_wall:.1f}s (eager model/detector load)"
|
| 779 |
+
+ (f"; total setup {time.perf_counter() - proc_t0:.1f}s from process start"
|
| 780 |
+
if proc_t0 is not None else ""))
|
| 781 |
+
slot = {a["qID"]: i for i, a in enumerate(answers)}
|
| 782 |
+
|
| 783 |
+
budget = POOLED_SEC_PER_QUESTION * len(answers)
|
| 784 |
+
job_t0 = time.perf_counter()
|
| 785 |
+
budget_breached = False
|
| 786 |
+
done: set[str] = set()
|
| 787 |
+
for row in rows:
|
| 788 |
+
qid = str(row.get("qID", ""))
|
| 789 |
+
if not qid or qid in done or qid not in slot:
|
| 790 |
+
if qid and qid in done:
|
| 791 |
+
_log(f"PROBESHELL: skipping duplicate qID row ({qid!r})")
|
| 792 |
+
continue
|
| 793 |
+
done.add(qid)
|
| 794 |
+
elapsed = time.perf_counter() - job_t0
|
| 795 |
+
if elapsed > budget and not budget_breached:
|
| 796 |
+
budget_breached = True
|
| 797 |
+
_log(f"PROBESHELL: POOLED BUDGET EXCEEDED ({elapsed:.0f}s > "
|
| 798 |
+
f"{budget:.0f}s for {len(answers)} questions) — continuing; "
|
| 799 |
+
f"the platform owns enforcement and the answer file is "
|
| 800 |
+
f"already complete at every instant")
|
| 801 |
+
t0 = time.perf_counter()
|
| 802 |
+
question = str(row.get("question", ""))
|
| 803 |
+
# Same B2-19-1 class, sibling site: these run OUTSIDE the per-question
|
| 804 |
+
# try — a raising coercion here would degrade the WHOLE run via the
|
| 805 |
+
# outer net instead of costing one row. Never-raise by construction.
|
| 806 |
+
start = _safe_float(row.get("start_time", 0.0) or 0.0)
|
| 807 |
+
end = _safe_float(row.get("end_time", 0.0) or 0.0)
|
| 808 |
+
try:
|
| 809 |
+
scratch = infer_all(question, start_time=start, end_time=end)
|
| 810 |
+
# H5: prefer an answer_format EXPOSED on the request over the inferred
|
| 811 |
+
# one. The observed live contract does NOT carry this field (the
|
| 812 |
+
# Request schema is qID/videoID/start/end/procedure_type/question),
|
| 813 |
+
# so on real inputs this is byte-identical to inference — a defensive
|
| 814 |
+
# guard that honors a true format if a future contract exposes it,
|
| 815 |
+
# avoiding the misroute the H5 validator policy otherwise absorbs.
|
| 816 |
+
exposed_fmt = row.get("answer_format")
|
| 817 |
+
fmt = exposed_fmt if isinstance(exposed_fmt, str) and exposed_fmt \
|
| 818 |
+
else scratch.get("answer_format", "open_ended")
|
| 819 |
+
req = Request(
|
| 820 |
+
qID=qid,
|
| 821 |
+
videoID=str(row.get("videoID", "")),
|
| 822 |
+
start_time=start,
|
| 823 |
+
end_time=end,
|
| 824 |
+
procedure_type=str(row.get("procedure_type", "")),
|
| 825 |
+
question=question,
|
| 826 |
+
)
|
| 827 |
+
if zip_batch is None:
|
| 828 |
+
resp = container.answer(req, scratch_init=scratch)
|
| 829 |
+
else:
|
| 830 |
+
from solution.entrypoint.segment_zip_input import MemberStagingError
|
| 831 |
+
if zip_batch.failed(qid):
|
| 832 |
+
answers[slot[qid]] = {"qID": qid, "content": "", "latency": time.perf_counter() - t0}
|
| 833 |
+
write_answers(output_dir, answers)
|
| 834 |
+
continue
|
| 835 |
+
with ExitStack() as clips:
|
| 836 |
+
try:
|
| 837 |
+
clips.enter_context(zip_batch.member(qid))
|
| 838 |
+
except MemberStagingError as error:
|
| 839 |
+
_record_zip_failure(zip_batch, qid, "answer", error)
|
| 840 |
+
answers[slot[qid]] = {"qID": qid, "content": "", "latency": time.perf_counter() - t0}
|
| 841 |
+
write_answers(output_dir, answers)
|
| 842 |
+
continue
|
| 843 |
+
# Close all loader FDs BEFORE the lease unlinks its clip.
|
| 844 |
+
# This callback is ZIP-only; the original loose path is unchanged.
|
| 845 |
+
if container.video_loader is not None:
|
| 846 |
+
clips.callback(container.video_loader.close_all)
|
| 847 |
+
# Model exceptions occur outside the materialization catch.
|
| 848 |
+
resp = container.answer(req, scratch_init=scratch)
|
| 849 |
+
content = validate(resp.content, fmt)
|
| 850 |
+
answers[slot[qid]] = {"qID": qid, "content": content,
|
| 851 |
+
"latency": time.perf_counter() - t0}
|
| 852 |
+
except Exception as e:
|
| 853 |
+
from solution.entrypoint.segment_zip_input import ZipIntegrityError
|
| 854 |
+
if isinstance(e, ZipIntegrityError):
|
| 855 |
+
raise
|
| 856 |
+
event = {"qID": qid, "stage": "answer_loop_exception", "exception_type": type(e).__name__,
|
| 857 |
+
"exception_module": type(e).__module__, "message": str(e), "output_policy": "empty_content",
|
| 858 |
+
"traceback": [{"path": t.filename, "function": t.name, "line": t.lineno} for t in traceback.extract_tb(e.__traceback__)]}
|
| 859 |
+
_QUESTION_FAILURE_EVENTS.append(event)
|
| 860 |
+
_log("QUESTION_FAILURE " + json.dumps(event, sort_keys=True))
|
| 861 |
+
answers[slot[qid]] = {"qID": qid, "content": "", "latency": time.perf_counter() - t0}
|
| 862 |
+
write_answers(output_dir, answers)
|
| 863 |
+
_log(f"PROBESHELL: job wall={time.perf_counter() - job_t0:.1f}s "
|
| 864 |
+
f"pooled-budget={budget:.0f}s n={len(answers)}")
|
| 865 |
+
# P2-06: emit the per-layer latency/attribution summary at end of run (it was
|
| 866 |
+
# accumulated every question but never reported — the only instrument for
|
| 867 |
+
# diagnosing which layer won and where budget was burned). Never fatal.
|
| 868 |
+
try:
|
| 869 |
+
container.metrics.report()
|
| 870 |
+
except Exception as e:
|
| 871 |
+
_log(f"PROBESHELL: metrics report skipped (non-fatal {type(e).__name__}: {e})")
|
| 872 |
+
return answers
|
| 873 |
+
|
| 874 |
+
|
| 875 |
+
# ── entry ────────────────────────────────────────────────────────────────
|
| 876 |
+
|
| 877 |
+
def main(argv: list[str] | None = None) -> int:
|
| 878 |
+
ap = argparse.ArgumentParser(
|
| 879 |
+
description="solution submission shell (live GC PROCEDURE contract)")
|
| 880 |
+
ap.add_argument("--submission", action="store_true",
|
| 881 |
+
help="arm SUBMISSION mode — the sanctioned production CMD "
|
| 882 |
+
"form (equivalent env: SURGFIELD_SUBMISSION_MODE=1)")
|
| 883 |
+
ap.add_argument("--probe", action="store_true",
|
| 884 |
+
help="arm PROBE mode for interface rehearsal (equivalent "
|
| 885 |
+
"env: SURGFIELD_PROBE_MODE=1)")
|
| 886 |
+
ap.add_argument("--input-dir", default="/input")
|
| 887 |
+
ap.add_argument("--output-dir", default="/output")
|
| 888 |
+
ap.add_argument("--device", default="cuda",
|
| 889 |
+
help="engine device (default: cuda); in a CPU-only context "
|
| 890 |
+
"the detector load fails with one benign WARN and the "
|
| 891 |
+
"detection layers defer — never a crash")
|
| 892 |
+
args = ap.parse_args(argv)
|
| 893 |
+
|
| 894 |
+
# P1-20: capture a process-start monotonic BEFORE any staging/model load so
|
| 895 |
+
# the platform-scored ~120s setup window is measurable end-to-end (the shell
|
| 896 |
+
# previously only timed from inside answer_all, AFTER eager model loading —
|
| 897 |
+
# a cold-start overrun was invisible).
|
| 898 |
+
_proc_t0 = time.perf_counter()
|
| 899 |
+
|
| 900 |
+
# Refusal gate (design unchanged by the promotion): no sanctioned arming
|
| 901 |
+
# form, no run — exit 2 before any I/O. A misbuilt CMD cannot produce
|
| 902 |
+
# output it was never armed to produce.
|
| 903 |
+
submission = args.submission or os.environ.get("SURGFIELD_SUBMISSION_MODE") == "1"
|
| 904 |
+
probe = args.probe or os.environ.get("SURGFIELD_PROBE_MODE") == "1"
|
| 905 |
+
if not (submission or probe):
|
| 906 |
+
_log("PROBE-REFUSED: no sanctioned arming form present "
|
| 907 |
+
"(--submission|--probe, or SURGFIELD_SUBMISSION_MODE=1|"
|
| 908 |
+
"SURGFIELD_PROBE_MODE=1); nothing written. This shell is the "
|
| 909 |
+
"submission entrypoint ONLY when armed via a sanctioned form")
|
| 910 |
+
return 2
|
| 911 |
+
if submission:
|
| 912 |
+
_log("PROBESHELL: submission-mode armed (sanctioned production path)")
|
| 913 |
+
else:
|
| 914 |
+
_log("PROBESHELL: probe-mode armed (solution tree; interface rehearsal only)")
|
| 915 |
+
|
| 916 |
+
input_dir, output_dir = Path(args.input_dir), Path(args.output_dir)
|
| 917 |
+
_ZIP_STAGING_EVENTS.clear()
|
| 918 |
+
_QUESTION_FAILURE_EVENTS.clear()
|
| 919 |
+
_MEDIA_FENCE_EVENTS.clear()
|
| 920 |
+
zip_batch = None
|
| 921 |
+
# P9 (2026-07-31): tempfile.mkdtemp() is deferred to INSIDE the coverage net
|
| 922 |
+
# below (it is only needed for video staging). On a read-only/absent /tmp — the
|
| 923 |
+
# exact failsafe scenario, GC /tmp semantics UNCONFIRMED — the old top-of-main
|
| 924 |
+
# mkdtemp raised BEFORE the format-aware pre-seed, shipping the launcher's floor
|
| 925 |
+
# instead of the strictly-better format-aware fallback that needs no /tmp.
|
| 926 |
+
|
| 927 |
+
try:
|
| 928 |
+
rows = read_requests(input_dir / "request.json")
|
| 929 |
+
except (OSError, ValueError) as e:
|
| 930 |
+
# P9: the launcher (surgfield_shell.launch) already wrote a full-coverage floor
|
| 931 |
+
# to /output BEFORE this shell ran. Without a readable request set there is no
|
| 932 |
+
# qID universe to re-cover, so do NOT clobber that good floor with an empty '[]'
|
| 933 |
+
# (a latent re-introduction of the empty-answer regression the floor exists to
|
| 934 |
+
# prevent, if the launcher's read succeeded where this re-read raised). Only write
|
| 935 |
+
# '[]' when no answer.json exists at all — the socket is still never left bare.
|
| 936 |
+
_log(f"PROBESHELL: request.json unreadable ({type(e).__name__}: {e})")
|
| 937 |
+
ans = output_dir / "answer.json"
|
| 938 |
+
if ans.exists():
|
| 939 |
+
_log("PROBESHELL: preserving the existing launcher floor on disk "
|
| 940 |
+
"(NOT clobbering with [])")
|
| 941 |
+
else:
|
| 942 |
+
write_answers(output_dir, [])
|
| 943 |
+
_log("PROBESHELL: no prior floor on disk — wrote empty [] "
|
| 944 |
+
"(output socket non-bare)")
|
| 945 |
+
return 1
|
| 946 |
+
|
| 947 |
+
# PRE-SEED (guarantee 1): full-coverage fallback answers land in
|
| 948 |
+
# /output/answer.json before any engine work — a kill at any later instant
|
| 949 |
+
# still leaves a complete file. This lands BEFORE mkdtemp so a /tmp failure
|
| 950 |
+
# cannot foreclose the format-aware floor.
|
| 951 |
+
answers = fallback_answers(rows)
|
| 952 |
+
write_answers(output_dir, answers)
|
| 953 |
+
_log(f"PROBESHELL: pre-seeded {len(answers)} fallback answers")
|
| 954 |
+
|
| 955 |
+
# OUTERMOST COVERAGE NET (post-arming): once the request set is readable,
|
| 956 |
+
# NO failure below — engine imports included, i.e. even the unsupported
|
| 957 |
+
# file-path invocation — may leave /output without a full-coverage
|
| 958 |
+
# answer.json. Everything between here and the final write is contained;
|
| 959 |
+
# degradation is loud, never silent.
|
| 960 |
+
try:
|
| 961 |
+
# P9: video-staging workdir, created HERE (inside the coverage net, AFTER the
|
| 962 |
+
# format-aware pre-seed). A read-only/absent /tmp now degrades to the pre-seeded
|
| 963 |
+
# floor via this net, never forecloses it.
|
| 964 |
+
workdir = Path(tempfile.mkdtemp(prefix="surgfield-shell-"))
|
| 965 |
+
|
| 966 |
+
# RUN-SCOPED ENV RESET (in-process isolation): a prior main() call in
|
| 967 |
+
# this interpreter may have left these pointing at a dead workdir —
|
| 968 |
+
# every run starts from a clean slate and sets only what ITS inputs
|
| 969 |
+
# justify.
|
| 970 |
+
os.environ.pop("SURGFIELD_FO_DEFINITIONS_FILE", None)
|
| 971 |
+
os.environ.pop("SURGFIELD_CLIP_INPUT", None)
|
| 972 |
+
|
| 973 |
+
# Zeroed-boundary telemetry (B2-19-1 cure: lives INSIDE the net AND the
|
| 974 |
+
# coercion is never-raise): a row whose end_time is 0.0 — or malformed
|
| 975 |
+
# and therefore coerced to 0.0 — gives every duration-gated layer
|
| 976 |
+
# duration 0; count it loudly, never silently, never fatally.
|
| 977 |
+
zeroed = sum(1 for r in rows
|
| 978 |
+
if _safe_float(r.get("end_time", 0.0) or 0.0) == 0.0)
|
| 979 |
+
if zeroed:
|
| 980 |
+
_log(f"PROBESHELL: WARN {zeroed} request row(s) with end_time==0.0 "
|
| 981 |
+
f"(zeroed boundary)")
|
| 982 |
+
# Whole-video time-questions with start==0 — the Leg-B population:
|
| 983 |
+
# covered by the engine, but never invisibly (semantics HELD to the
|
| 984 |
+
# platform observer; the shell records, it does not resolve).
|
| 985 |
+
whole_video = sum(
|
| 986 |
+
1 for r in rows
|
| 987 |
+
if _safe_float(r.get("start_time", 0.0) or 0.0) == 0.0
|
| 988 |
+
and _safe_float(r.get("end_time", 0.0) or 0.0) > 0.0
|
| 989 |
+
and _TIME_Q.search(str(r.get("question", ""))))
|
| 990 |
+
if whole_video:
|
| 991 |
+
_log(f"PROBESHELL: WARN {whole_video} whole-video time-question "
|
| 992 |
+
f"row(s) with start_time==0.0 (boundary semantics recorded, "
|
| 993 |
+
f"not resolved)")
|
| 994 |
+
|
| 995 |
+
# DIAGNOSTIC (log-only): dump the full /input tree so the next live
|
| 996 |
+
# try-out definitively reveals GC's real batch-videos placement.
|
| 997 |
+
_log_input_tree(input_dir)
|
| 998 |
+
|
| 999 |
+
fo_path = stage_fo_definitions(input_dir / "FO_definitions.json", workdir)
|
| 1000 |
+
if fo_path is not None:
|
| 1001 |
+
os.environ["SURGFIELD_FO_DEFINITIONS_FILE"] = str(fo_path)
|
| 1002 |
+
_log(f"PROBESHELL: FO definitions staged ({fo_path.stat().st_size} bytes)")
|
| 1003 |
+
|
| 1004 |
+
# ADDITIVE track dispatch (STEP-2, 2026-08-04, FRAME/SEGMENT foundation):
|
| 1005 |
+
# SURGFIELD_TRACK selects the input resolver. Default/'procedure' runs the
|
| 1006 |
+
# UNCHANGED video ladder (stage_videos_root + normalize_clips) VERBATIM in
|
| 1007 |
+
# the else — the FRAME/SEGMENT branches are entered ONLY when explicitly
|
| 1008 |
+
# armed, so the PROCEDURE resolved-clips + answer.json are byte-identical
|
| 1009 |
+
# (proven by solution.entrypoint.test_procedure_byte_identity). The env read
|
| 1010 |
+
# is pure (no output side effect); track_inputs is imported LAZILY so a
|
| 1011 |
+
# PROCEDURE run never even imports the FRAME/SEGMENT code.
|
| 1012 |
+
_track = os.environ.get("SURGFIELD_TRACK", "procedure").strip().lower()
|
| 1013 |
+
if _track == "segment":
|
| 1014 |
+
from solution.entrypoint.segment_zip_input import maybe_zip_batch
|
| 1015 |
+
zip_batch = maybe_zip_batch(rows, input_dir, workdir, _ZIP_STAGING_EVENTS,
|
| 1016 |
+
receipt_sink=lambda record: _log("ZIP_MEMBER_RECEIPT " + json.dumps(record, sort_keys=True)))
|
| 1017 |
+
if zip_batch is not None:
|
| 1018 |
+
root, norm_root = None, zip_batch.norm_root
|
| 1019 |
+
_log(f"PROBESHELL: bounded ZIP staging for {len(rows)} questions; one selected clip at a time")
|
| 1020 |
+
elif _track in ("frame", "segment"):
|
| 1021 |
+
from solution.entrypoint.track_inputs import stage_and_normalize_track
|
| 1022 |
+
root, norm_root = stage_and_normalize_track(_track, rows, input_dir, workdir)
|
| 1023 |
+
else:
|
| 1024 |
+
root = stage_videos_root(input_dir, workdir)
|
| 1025 |
+
norm_root = normalize_clips(rows, root, workdir)
|
| 1026 |
+
# Both env writes (SURGFIELD_FO_DEFINITIONS_FILE above, CLIP_INPUT here)
|
| 1027 |
+
# are RUN-SCOPED: reset at the top of this block, set for this run's
|
| 1028 |
+
# duration, read per-call by the engine. In the container nothing
|
| 1029 |
+
# survives the exit; in-process reuse is isolated by the reset.
|
| 1030 |
+
os.environ["SURGFIELD_CLIP_INPUT"] = "1" # per-qID clip semantics engine-side
|
| 1031 |
+
to_fence, n_probed = emit_decode_records(
|
| 1032 |
+
rows, norm_root, probe_budget_sec=PROBE_PHASE_BUDGET_SEC, zip_batch=zip_batch)
|
| 1033 |
+
# H6 fence: a clip whose killable prober hit a WEDGE/CRASH is removed from
|
| 1034 |
+
# the normalized tree so the in-process sampler never touches it; that qID
|
| 1035 |
+
# defers to text/rule like an absent clip. (The in-process decode is ALSO
|
| 1036 |
+
# SIGALRM-bounded now — P1-03 — so a wedge that slips past the probe still
|
| 1037 |
+
# can't hang the job.) NEVER fenced on a clean frame-0 miss.
|
| 1038 |
+
systemic, env_faults, dangers = _fence_decision(to_fence, n_probed)
|
| 1039 |
+
if systemic:
|
| 1040 |
+
_log("PROBESHELL: H6-FENCE ABORT — systemic decode fault "
|
| 1041 |
+
f"(env_faults={sorted(env_faults)[:10]}, "
|
| 1042 |
+
f"dangerous={len(dangers)}/{n_probed} probed); NOT unlinking clips "
|
| 1043 |
+
"(would blind the whole pipeline); relying on the in-process "
|
| 1044 |
+
"SIGALRM decode deadline for containment")
|
| 1045 |
+
elif dangers:
|
| 1046 |
+
_log("PROBESHELL: H6-FENCE unlinking "
|
| 1047 |
+
+ ", ".join(f"{q}({r})" for q, r in sorted(dangers.items()))
|
| 1048 |
+
+ " — in-process decode fenced; affected qIDs recorded for empty responses")
|
| 1049 |
+
for qid in dangers:
|
| 1050 |
+
_MEDIA_FENCE_EVENTS.append({"qID": qid, "stage": "startup_decoder_fence", "reason": dangers[qid]})
|
| 1051 |
+
link = norm_root / "plain" / f"{qid}.mp4"
|
| 1052 |
+
try:
|
| 1053 |
+
if link.is_symlink() or link.exists():
|
| 1054 |
+
link.unlink()
|
| 1055 |
+
except OSError as error:
|
| 1056 |
+
from solution.entrypoint.segment_zip_input import ZipIntegrityError
|
| 1057 |
+
raise ZipIntegrityError("failed to fence dangerous decoder input") from error
|
| 1058 |
+
_setup_wall = time.perf_counter() - _proc_t0
|
| 1059 |
+
_log(f"PROBESHELL: SETUP-WALL {_setup_wall:.1f}s (arm→staging→decode-proof, "
|
| 1060 |
+
f"before model load); platform setup window ~120s")
|
| 1061 |
+
if _setup_wall > 90.0:
|
| 1062 |
+
_log(f"PROBESHELL: WARN setup wall {_setup_wall:.0f}s is within 30s of "
|
| 1063 |
+
f"the ~120s platform setup window — cold-start overrun risk")
|
| 1064 |
+
answers = answer_all(rows, norm_root, args.device, answers, output_dir,
|
| 1065 |
+
proc_t0=_proc_t0, zip_batch=zip_batch)
|
| 1066 |
+
except Exception as e:
|
| 1067 |
+
_log(f"PROBESHELL DEGRADED: shell/engine failure after arming "
|
| 1068 |
+
f"({type(e).__name__}: {e}); the pre-seeded full-coverage "
|
| 1069 |
+
f"answers stand for {len(rows)} rows")
|
| 1070 |
+
finally:
|
| 1071 |
+
if zip_batch is not None:
|
| 1072 |
+
zip_batch.close()
|
| 1073 |
+
|
| 1074 |
+
try:
|
| 1075 |
+
_c1_collision_preflight(answers, rows) # C1 ENFORCE: DECLINE colliding answers (P9)
|
| 1076 |
+
except Exception as e: # belt: the pre-flight must never crash the run
|
| 1077 |
+
_log(f"C1 pre-flight skipped (non-fatal {type(e).__name__}: {e})")
|
| 1078 |
+
|
| 1079 |
+
# P2-14/P2-18: OUTPUT-HYGIENE self-DQ preflight (F0/DUP/COV/FMT/P1/P2/P3).
|
| 1080 |
+
# OBSERVE-ONLY in-container (like C1): it LOGS a self-DQ risk so it is visible
|
| 1081 |
+
# in the container log; it NEVER mutates or blocks (the never-empty coverage
|
| 1082 |
+
# guarantee stands). The blocking pre-upload gate is the CLI wrapper
|
| 1083 |
+
# `run_preflight_on_file` (exit-nonzero on findings) that Reza runs before an
|
| 1084 |
+
# upload. Coverage is checked against the request qID universe; the FMT check
|
| 1085 |
+
# uses the answer_format inferred/exposed per row. Never fatal here.
|
| 1086 |
+
try:
|
| 1087 |
+
from solution.cap_fmt_inferrer import infer_all
|
| 1088 |
+
from solution.answer_validator import normalize_answer_format
|
| 1089 |
+
from solution.entrypoint.output_hygiene_preflight import run_preflight
|
| 1090 |
+
expected_qids = [str(r.get("qID", "")) for r in rows if r.get("qID", "")]
|
| 1091 |
+
formats_by_qid: dict[str, str] = {}
|
| 1092 |
+
for r in rows:
|
| 1093 |
+
qid = str(r.get("qID", ""))
|
| 1094 |
+
if not qid:
|
| 1095 |
+
continue
|
| 1096 |
+
exposed = r.get("answer_format")
|
| 1097 |
+
try:
|
| 1098 |
+
fmt = (normalize_answer_format(str(exposed)) if exposed
|
| 1099 |
+
else infer_all(str(r.get("question", "")))["answer_format"])
|
| 1100 |
+
except Exception:
|
| 1101 |
+
fmt = None
|
| 1102 |
+
if fmt:
|
| 1103 |
+
formats_by_qid[qid] = fmt
|
| 1104 |
+
rep = run_preflight(answers, expected_qids=expected_qids,
|
| 1105 |
+
formats_by_qid=formats_by_qid)
|
| 1106 |
+
_log(f"PROBESHELL: {rep.summary()}")
|
| 1107 |
+
for f in rep.findings:
|
| 1108 |
+
_log(f" HYGIENE [{f.check}] qID={f.qID}: {f.reason} :: {f.detail}")
|
| 1109 |
+
except Exception as e: # observe-only: never affect output
|
| 1110 |
+
_log(f"output-hygiene pre-flight skipped (non-fatal {type(e).__name__}: {e})")
|
| 1111 |
+
|
| 1112 |
+
out = write_answers(output_dir, answers)
|
| 1113 |
+
_log(f"PROBESHELL: wrote {out} ({len(answers)} answers)")
|
| 1114 |
+
return 0
|
| 1115 |
+
|
| 1116 |
+
|
| 1117 |
+
if __name__ == "__main__":
|
| 1118 |
+
sys.exit(main())
|
platform_wrapper/segment_zip_input.py
ADDED
|
@@ -0,0 +1,345 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Bounded SEGMENT ZIP staging. The bootstrap validates request/root custody first.
|
| 2 |
+
Only the selected clip is materialized; the source ZIP remains on read-only input.
|
| 3 |
+
"""
|
| 4 |
+
from contextlib import contextmanager
|
| 5 |
+
import errno
|
| 6 |
+
import hashlib
|
| 7 |
+
import json
|
| 8 |
+
import os
|
| 9 |
+
from pathlib import Path, PurePosixPath
|
| 10 |
+
import stat
|
| 11 |
+
import zipfile
|
| 12 |
+
|
| 13 |
+
class ZipIntegrityError(RuntimeError):
|
| 14 |
+
"""Archive/path/owned-state corruption must remain fatal for the invocation."""
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
BLOCK = 1024 * 1024
|
| 18 |
+
MEDIA = {".mp4", ".avi", ".mov", ".mkv", ".webm", ".m4v", ".mpg", ".mpeg"}
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def stamp(s):
|
| 22 |
+
return tuple(getattr(s, k) for k in ("st_dev", "st_ino", "st_mode", "st_uid", "st_gid", "st_size", "st_mtime_ns", "st_ctime_ns", "st_nlink"))
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def validate_members(infos):
|
| 26 |
+
files, directories = {}, set()
|
| 27 |
+
if len(infos) > 100000:
|
| 28 |
+
raise ZipIntegrityError("ZIP member-count bound")
|
| 29 |
+
for info in infos:
|
| 30 |
+
name = info.filename
|
| 31 |
+
trimmed = name[:-1] if info.is_dir() else name
|
| 32 |
+
parts = trimmed.split("/")
|
| 33 |
+
if not trimmed or "\\" in name or any(x in ("", ".", "..") or ":" in x or any(ord(c) < 32 for c in x) for x in parts):
|
| 34 |
+
raise ZipIntegrityError("unsafe ZIP name")
|
| 35 |
+
kind = stat.S_IFMT(info.external_attr >> 16)
|
| 36 |
+
if kind not in (0, stat.S_IFREG, stat.S_IFDIR) or info.flag_bits & 1:
|
| 37 |
+
raise ZipIntegrityError("special or encrypted ZIP member")
|
| 38 |
+
key = trimmed.casefold()
|
| 39 |
+
if info.is_dir():
|
| 40 |
+
if key in files:
|
| 41 |
+
raise ZipIntegrityError("file-directory collision")
|
| 42 |
+
directories.add(key)
|
| 43 |
+
else:
|
| 44 |
+
if key in files or key in directories:
|
| 45 |
+
raise ZipIntegrityError("duplicate or case-colliding ZIP member")
|
| 46 |
+
files[key] = info
|
| 47 |
+
for info in files.values():
|
| 48 |
+
if any(str(a).casefold() in files for a in PurePosixPath(info.filename).parents if str(a) != "."):
|
| 49 |
+
raise ZipIntegrityError("ancestor file collision")
|
| 50 |
+
return {info.filename: info for info in files.values()}
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def plan_rows(rows, members):
|
| 54 |
+
"""Same strict selection groups as current R3 input validator.
|
| 55 |
+
It never rewrites request clocks, qIDs, question text or videoID.
|
| 56 |
+
"""
|
| 57 |
+
media = [PurePosixPath(n) for n in members if PurePosixPath(n).suffix.lower() in MEDIA]
|
| 58 |
+
result, seen = [], set()
|
| 59 |
+
for row in rows:
|
| 60 |
+
qid = row["qID"]
|
| 61 |
+
if (not isinstance(qid, str) or not qid or qid in seen or qid in (".", "..")
|
| 62 |
+
or "/" in qid or "\\" in qid or any(ord(c) < 32 for c in qid)):
|
| 63 |
+
raise ZipIntegrityError("invalid or duplicate request qID")
|
| 64 |
+
seen.add(qid)
|
| 65 |
+
hits = [p for p in media if p.stem == qid]
|
| 66 |
+
groups = [[p for p in hits if p.parent == PurePosixPath(sub)] for sub in ("plain", "overlayed", ".")]
|
| 67 |
+
groups.append(hits)
|
| 68 |
+
groups.append([p for p in media if p.stem == row.get("videoID")])
|
| 69 |
+
index, selected = next(((i, g) for i, g in enumerate(groups) if g), (None, []))
|
| 70 |
+
if len(selected) > 1:
|
| 71 |
+
raise ZipIntegrityError("ambiguous selected media")
|
| 72 |
+
result.append({"qID": qid, "member": str(selected[0]) if selected else None, "videoid_rung": index == 4})
|
| 73 |
+
return result
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def _open_exclusive(path):
|
| 77 |
+
return os.fdopen(os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY | os.O_NOFOLLOW, 0o600), "wb")
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
class MemberStagingError(Exception):
|
| 81 |
+
"""Raised only by lease materialization, never by the model consumer."""
|
| 82 |
+
def __init__(self, details):
|
| 83 |
+
self.details = details
|
| 84 |
+
super().__init__(details["message"])
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def member_error(path, info, count, cause):
|
| 88 |
+
return MemberStagingError({"stage": "zip_member_staging", "archive": str(path),
|
| 89 |
+
"member": info.filename, "declared_bytes": info.file_size,
|
| 90 |
+
"written_bytes": count, "exception_type": type(cause).__name__,
|
| 91 |
+
"errno": getattr(cause, "errno", None), "message": str(cause)})
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
class ZipLease:
|
| 95 |
+
def __init__(self, archive_path, root, *, max_working_bytes, reserve_bytes=64 * 1024 * 1024, receipt_sink=None):
|
| 96 |
+
self.path, self.root = Path(archive_path), Path(root)
|
| 97 |
+
self.limit, self.reserve = max_working_bytes, reserve_bytes
|
| 98 |
+
self.receipt_sink = receipt_sink
|
| 99 |
+
if type(self.limit) is not int or type(self.reserve) is not int or self.limit <= self.reserve or self.reserve < 0:
|
| 100 |
+
raise ZipIntegrityError("invalid working budget")
|
| 101 |
+
self.root.mkdir(mode=0o700, parents=False, exist_ok=False)
|
| 102 |
+
self.root_id = (self.root.stat().st_dev, self.root.stat().st_ino)
|
| 103 |
+
self.raw = None
|
| 104 |
+
self.zf = None
|
| 105 |
+
self.active = False
|
| 106 |
+
self.receipts = []
|
| 107 |
+
try:
|
| 108 |
+
fd = os.open(self.path, os.O_RDONLY | os.O_NOFOLLOW)
|
| 109 |
+
self.raw = os.fdopen(fd, "rb")
|
| 110 |
+
s = os.fstat(fd)
|
| 111 |
+
if not stat.S_ISREG(s.st_mode):
|
| 112 |
+
raise ZipIntegrityError("archive is not a regular file")
|
| 113 |
+
self.source = stamp(s)
|
| 114 |
+
self.check_source()
|
| 115 |
+
self.zf = zipfile.ZipFile(self.raw)
|
| 116 |
+
self.members = validate_members(self.zf.infolist())
|
| 117 |
+
except BaseException:
|
| 118 |
+
self.close()
|
| 119 |
+
raise
|
| 120 |
+
|
| 121 |
+
def check_source(self):
|
| 122 |
+
try:
|
| 123 |
+
return self._check_source()
|
| 124 |
+
except OSError as error:
|
| 125 |
+
raise ZipIntegrityError("archive/lease identity cannot be checked") from error
|
| 126 |
+
|
| 127 |
+
def _check_source(self):
|
| 128 |
+
if stamp(os.fstat(self.raw.fileno())) != self.source or stamp(self.path.lstat()) != self.source:
|
| 129 |
+
raise ZipIntegrityError("archive identity changed")
|
| 130 |
+
r = self.root.lstat()
|
| 131 |
+
if not stat.S_ISDIR(r.st_mode) or (r.st_dev, r.st_ino) != self.root_id:
|
| 132 |
+
raise ZipIntegrityError("owned lease directory replaced")
|
| 133 |
+
|
| 134 |
+
@contextmanager
|
| 135 |
+
def member(self, name, *, qid=None, phase=None):
|
| 136 |
+
if self.active:
|
| 137 |
+
raise ZipIntegrityError("only one media lease may be live")
|
| 138 |
+
self.check_source()
|
| 139 |
+
info = self.members[name]
|
| 140 |
+
available = os.statvfs(self.root)
|
| 141 |
+
free_bytes = available.f_bavail * available.f_frsize
|
| 142 |
+
if info.file_size > min(self.limit, free_bytes) - self.reserve:
|
| 143 |
+
cause = OSError(errno.ENOSPC, "member does not fit bounded scratch reserve")
|
| 144 |
+
raise member_error(self.path, info, 0, cause)
|
| 145 |
+
self.active = True
|
| 146 |
+
token = hashlib.sha256(name.encode("utf-8")).hexdigest()
|
| 147 |
+
part, final = self.root / (token + ".part"), self.root / (token + ".mp4")
|
| 148 |
+
owned = {}
|
| 149 |
+
count, digest, record = 0, hashlib.sha256(), None
|
| 150 |
+
try:
|
| 151 |
+
try:
|
| 152 |
+
with _open_exclusive(part) as out:
|
| 153 |
+
s = part.lstat(); owned[part] = (s.st_dev, s.st_ino)
|
| 154 |
+
with self.zf.open(info, "r") as src:
|
| 155 |
+
while True:
|
| 156 |
+
block = src.read(BLOCK)
|
| 157 |
+
if not block:
|
| 158 |
+
break
|
| 159 |
+
if count + len(block) > info.file_size or count + len(block) + self.reserve > self.limit:
|
| 160 |
+
raise OSError(errno.ENOSPC, "stream exceeds member or working budget")
|
| 161 |
+
if out.write(block) != len(block):
|
| 162 |
+
raise OSError(errno.EIO, "short write while staging media")
|
| 163 |
+
digest.update(block)
|
| 164 |
+
count += len(block)
|
| 165 |
+
if count != info.file_size:
|
| 166 |
+
raise zipfile.BadZipFile("member size differs at full EOF")
|
| 167 |
+
self.check_source()
|
| 168 |
+
# Full ZipExtFile EOF validates CRC before the clip becomes visible.
|
| 169 |
+
part.chmod(0o400)
|
| 170 |
+
part.rename(final)
|
| 171 |
+
owned[final] = owned.pop(part)
|
| 172 |
+
name_bytes = name.encode("utf-8")
|
| 173 |
+
record = {"qID": qid, "phase": phase, "stage": "zip_member_staging",
|
| 174 |
+
"member": name if len(name_bytes) <= 1024 else None,
|
| 175 |
+
"member_name_bytes": len(name_bytes), "member_name_sha256": hashlib.sha256(name_bytes).hexdigest(),
|
| 176 |
+
"archive_identity": list(self.source), "bytes": count,
|
| 177 |
+
"sha256": digest.hexdigest(), "crc32": info.CRC, "full_member_eof": True,
|
| 178 |
+
"file_allocated_bytes": final.stat().st_blocks * 512,
|
| 179 |
+
"working_budget_bytes": min(self.limit, free_bytes), "free_bytes_before": free_bytes,
|
| 180 |
+
"reserve_bytes": self.reserve, "cleanup_completed": False}
|
| 181 |
+
self.receipts.append(record)
|
| 182 |
+
except (zipfile.BadZipFile, EOFError, OSError) as cause:
|
| 183 |
+
if isinstance(cause, OSError) and cause.errno not in (errno.ENOSPC, errno.EIO):
|
| 184 |
+
raise ZipIntegrityError("unclassified staging filesystem error") from cause
|
| 185 |
+
raise member_error(self.path, info, count, cause) from cause
|
| 186 |
+
yield final
|
| 187 |
+
finally:
|
| 188 |
+
self.active = False
|
| 189 |
+
for path, identity in owned.items():
|
| 190 |
+
try:
|
| 191 |
+
s = path.lstat()
|
| 192 |
+
except FileNotFoundError:
|
| 193 |
+
continue
|
| 194 |
+
except OSError as error:
|
| 195 |
+
raise ZipIntegrityError("owned media cleanup identity unavailable") from error
|
| 196 |
+
if not stat.S_ISREG(s.st_mode) or (s.st_dev, s.st_ino) != identity:
|
| 197 |
+
raise ZipIntegrityError("refusing cleanup of replaced media path")
|
| 198 |
+
try:
|
| 199 |
+
path.unlink()
|
| 200 |
+
except OSError as error:
|
| 201 |
+
raise ZipIntegrityError("owned media cleanup failed") from error
|
| 202 |
+
self.check_source()
|
| 203 |
+
if record is not None:
|
| 204 |
+
try:
|
| 205 |
+
after = os.statvfs(self.root)
|
| 206 |
+
except OSError as error:
|
| 207 |
+
raise ZipIntegrityError("post-cleanup filesystem observation failed") from error
|
| 208 |
+
record["free_bytes_after_unlink"] = after.f_bavail * after.f_frsize
|
| 209 |
+
record["cleanup_completed"] = True
|
| 210 |
+
record["source_identity_unchanged_after_use"] = True
|
| 211 |
+
if self.receipt_sink is not None:
|
| 212 |
+
self.receipt_sink(dict(record))
|
| 213 |
+
|
| 214 |
+
def close(self):
|
| 215 |
+
if self.active:
|
| 216 |
+
raise ZipIntegrityError("cannot close with active lease")
|
| 217 |
+
try:
|
| 218 |
+
if self.raw is not None and not self.raw.closed and hasattr(self, "source"):
|
| 219 |
+
self.check_source()
|
| 220 |
+
finally:
|
| 221 |
+
if self.zf is not None:
|
| 222 |
+
self.zf.close()
|
| 223 |
+
if self.raw is not None:
|
| 224 |
+
self.raw.close()
|
| 225 |
+
|
| 226 |
+
def __enter__(self):
|
| 227 |
+
return self
|
| 228 |
+
|
| 229 |
+
def __exit__(self, *unused):
|
| 230 |
+
self.close()
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
class ZipBatch:
|
| 234 |
+
"""Invocation-owned union of admitted ZIP and loose paths.
|
| 235 |
+
Only one sequential question is leased at a time. No answer/model code here.
|
| 236 |
+
"""
|
| 237 |
+
def __init__(self, rows, input_root, workdir, archives, loose, events, receipt_sink=None):
|
| 238 |
+
self.events, self.failures, self.stores = events, {}, []
|
| 239 |
+
self.active = False
|
| 240 |
+
self.norm_root = Path(workdir) / "norm"
|
| 241 |
+
self.plain = self.norm_root / "plain"
|
| 242 |
+
self.plain.mkdir(parents=True, exist_ok=False)
|
| 243 |
+
self.entries, union, case_names = {}, {}, set()
|
| 244 |
+
try:
|
| 245 |
+
for i, archive in enumerate(archives):
|
| 246 |
+
store = ZipLease(archive, Path(workdir) / ("zip-lease-" + str(i)),
|
| 247 |
+
max_working_bytes=(1 << 63) - 1, receipt_sink=receipt_sink)
|
| 248 |
+
self.stores.append(store)
|
| 249 |
+
for name in store.members:
|
| 250 |
+
if name.casefold() in case_names:
|
| 251 |
+
raise ZipIntegrityError("ambiguous ZIP/loose staging path")
|
| 252 |
+
case_names.add(name.casefold());union[name] = (store, name)
|
| 253 |
+
for file in loose:
|
| 254 |
+
name = file.relative_to(input_root).as_posix()
|
| 255 |
+
if name.casefold() in case_names:
|
| 256 |
+
raise ZipIntegrityError("ambiguous ZIP/loose staging path")
|
| 257 |
+
case_names.add(name.casefold());union[name] = (None, file)
|
| 258 |
+
for name in union:
|
| 259 |
+
if any(str(a).casefold() in case_names for a in PurePosixPath(name).parents if str(a) != "."):
|
| 260 |
+
raise ZipIntegrityError("cross-input ancestor file collision")
|
| 261 |
+
plan = plan_rows(rows, union)
|
| 262 |
+
for item in plan:
|
| 263 |
+
if item["member"] is None:
|
| 264 |
+
link = self.plain / (item["qID"] + ".mp4")
|
| 265 |
+
self.entries[item["qID"]] = (None, None, link, None)
|
| 266 |
+
continue
|
| 267 |
+
store, member = union[item["member"]]
|
| 268 |
+
target = (store.root / (hashlib.sha256(member.encode("utf-8")).hexdigest() + ".mp4")
|
| 269 |
+
if store is not None else Path(member).resolve())
|
| 270 |
+
link = self.plain / (item["qID"] + ".mp4")
|
| 271 |
+
link.symlink_to(target)
|
| 272 |
+
self.entries[item["qID"]] = (store, member, link, target)
|
| 273 |
+
(self.norm_root / "videoid_rung.json").write_text(json.dumps(sorted(x["qID"] for x in plan if x["videoid_rung"])))
|
| 274 |
+
self.plan = plan
|
| 275 |
+
except BaseException:
|
| 276 |
+
self.close()
|
| 277 |
+
raise
|
| 278 |
+
|
| 279 |
+
def cache_key(self, qid):
|
| 280 |
+
store, member, _, _ = self.entries[qid]
|
| 281 |
+
if member is None:
|
| 282 |
+
return ("missing", qid)
|
| 283 |
+
return (str(store.path), member) if store is not None else ("loose", str(member))
|
| 284 |
+
|
| 285 |
+
def failed(self, qid):
|
| 286 |
+
return qid in self.failures
|
| 287 |
+
|
| 288 |
+
def record_failure(self, qid, phase, error):
|
| 289 |
+
if type(error) is not MemberStagingError:
|
| 290 |
+
raise TypeError("only materialization failures may be recorded here")
|
| 291 |
+
event = {"qID": qid, "phase": phase, **error.details, "output_policy": "empty_content"}
|
| 292 |
+
self.events.append(event);self.failures[qid] = event
|
| 293 |
+
return event
|
| 294 |
+
|
| 295 |
+
@contextmanager
|
| 296 |
+
def member(self, qid, *, phase="answer"):
|
| 297 |
+
if self.active:
|
| 298 |
+
raise ZipIntegrityError("only one question media lease may be live")
|
| 299 |
+
self.active = True
|
| 300 |
+
try:
|
| 301 |
+
store, member, link, target = self.entries[qid]
|
| 302 |
+
if target is None:
|
| 303 |
+
raise MemberStagingError({"stage": "input_media_resolution", "archive": None,
|
| 304 |
+
"member": None, "declared_bytes": None, "written_bytes": 0,
|
| 305 |
+
"exception_type": "MissingMedia", "errno": None,
|
| 306 |
+
"message": "no selected media for this request"})
|
| 307 |
+
# A startup H6 fence removes the normalized link. Never resurrect it.
|
| 308 |
+
if not link.is_symlink():
|
| 309 |
+
if link.exists():
|
| 310 |
+
raise ZipIntegrityError("normalized lease link replaced by non-symlink")
|
| 311 |
+
yield link
|
| 312 |
+
return
|
| 313 |
+
if Path(os.readlink(link)) != target:
|
| 314 |
+
raise ZipIntegrityError("normalized lease target changed")
|
| 315 |
+
if store is None:
|
| 316 |
+
yield link
|
| 317 |
+
else:
|
| 318 |
+
with store.member(member, qid=qid, phase=phase):
|
| 319 |
+
yield link
|
| 320 |
+
finally:
|
| 321 |
+
self.active = False
|
| 322 |
+
|
| 323 |
+
def close(self):
|
| 324 |
+
errors = []
|
| 325 |
+
for store in self.stores:
|
| 326 |
+
try:
|
| 327 |
+
store.close()
|
| 328 |
+
except Exception as error:
|
| 329 |
+
errors.append(error)
|
| 330 |
+
if errors:
|
| 331 |
+
raise errors[0]
|
| 332 |
+
|
| 333 |
+
|
| 334 |
+
def maybe_zip_batch(rows, input_root, workdir, events, receipt_sink=None):
|
| 335 |
+
"""Return None for loose-only input so its original staging path is untouched."""
|
| 336 |
+
files = sorted((x for x in Path(input_root).rglob("*") if x.is_file()), key=str)
|
| 337 |
+
archives, loose = [], []
|
| 338 |
+
for file in files:
|
| 339 |
+
if file.suffix.lower() in MEDIA:
|
| 340 |
+
loose.append(file)
|
| 341 |
+
elif zipfile.is_zipfile(file):
|
| 342 |
+
archives.append(file)
|
| 343 |
+
if not archives:
|
| 344 |
+
return None
|
| 345 |
+
return ZipBatch(rows, Path(input_root), Path(workdir), archives, loose, events, receipt_sink=receipt_sink)
|