# Deterministic Final-DWG Scorer Contract This scorer is the official reproducible benchmark score. The checksum-bound vision judge described in [`vision-judge.md`](./vision-judge.md) remains an independent calibration and audit signal; it is not blended into the deterministic result. The correctness reward is computed only from the final `attempt.dwg`. Action traces, screenshots, action count, elapsed time, and whether the model independently saved are diagnostic data and are not scorer inputs. ## Inputs - Opaque `task_id` - Candidate DWG bytes - Active evaluator version - Matching gold-cache store - Versioned scorer configuration The scorer rejects missing candidates and files that do not have the benchmark's pinned `AC1032` DWG signature before calling the trusted evaluator. ## Trusted evaluator result The private evaluator returns: - Validity and a structured failure reason - AutoCAD and evaluator versions - Render-configuration checksum - Candidate source checksum - Entity count and entity-type counts - Prohibited entity types, if any - Model-space extents in drawing units - Task drawing type (`2d` or `3d`), true-3D entity count, and XYZ model extents - Paper-layout entity/annotation inventories and floating viewport camera data - The rendered space/layout and deterministic selection reason - A fixed-size normalized monochrome PNG render - Render checksum - Sampled world-coordinate paths for model-space curves, independent of entity segmentation - Dimension family, measured/displayed value, text position, rotation, and anchor points - Standalone text annotation content, position, height, and rotation The following failures force a zero reward: - `missing_candidate` - `invalid_dwg_signature` - `unreadable_dwg` - `empty_model_space` - `prohibited_entity` - `external_reference` - `render_failed` - `evaluator_unavailable` - `evaluator_error` - `gold_cache_missing` - `gold_cache_invalid` ## Deterministic comparison (score schema v3) The scorer has explicit 2D and 3D paths. It never selects a method based on whether an incidental curve happens to exist in a DWG. ### 2D vector-semantic comparison (v4) ```text raw = weighted_mean(applicable components only) geometry_gate = 0.35 + 0.65 * geometry_similarity score = min(raw, geometry_gate) components: 0.60 * geometry_similarity 0.25 * dimension_similarity 0.10 * standalone_text_annotation_similarity 0.05 * registered_visual_similarity ``` If the gold contains no dimensions or no standalone text, that component is marked non-applicable and the remaining weights are normalized. It does not silently award a perfect score. The evaluator samples `LINE`, polyline, arc, circle, ellipse, spline, and equivalent model-space curves. Python resamples them at a gold-derived physical interval, translates bounding-box centers together, and computes a symmetric soft Chamfer/F1 score. `LEADER`/`MULTILEADER` shafts are annotation rather than part geometry. Isolated line paths that cross the object without terminating on another edge are treated as centerlines or center marks and excluded from the base-geometry cloud. Entity segmentation does not matter; rotation, reflection, missing paths, and extra paths are not aligned away. Uniform scale remains significant unless the gold explicitly says that no dimensions are required or that the drawing is not to scale. Dimensions are matched by semantic family, visibly displayed value, and translated text position. Correct `TEXT`/`MTEXT` radius, diameter, angular, and plain numeric callouts are promoted to dimensions, so manual drafting is equivalent to native `DIMENSION` entities. AutoCAD symbols, decimal commas, and multiplicity prefixes such as `2xR1.4` are normalized. A native text override is authoritative for this component because it is what the submitted sheet shows; the geometry component and gate independently prevent a label from concealing wrong part geometry. Linear tolerance is the larger of 0.05 drawing units or 0.5%; angular tolerance is 0.25 degrees. Measurement correctness contributes 80% and placement 20%. Standalone `TEXT`, `MTEXT`, and text-bearing `MULTILEADER` annotations are matched separately by normalized content and translated placement. AutoCAD formatting codes, case, and standard diameter/degree encodings are normalized. Candidate and gold monochrome masks are centroid-registered and compared with an eight-pixel tolerance. This visual component carries only 5% for 2D. ### 3D layout-spatial comparison (v2) 3D scoring uses the required `Layout1` presentation and genuine model-space 3D facts. It never treats model-space 2D curve paths as solid topology. ```text raw = weighted_mean(applicable components only) geometry_gate = 0.35 + 0.65 * geometry_similarity score = min(raw, geometry_gate) components: 0.60 * geometry_similarity 0.25 * Layout1_dimension_similarity 0.10 * Layout1_text_annotation_similarity 0.05 * presentation_similarity geometry_similarity = true_3d_presence * (0.60 * normalized_Layout1_visual_similarity + 0.40 * isolated_XYZ_extent_similarity) ``` XYZ size compares sorted axis lengths, so model-space translation and an equivalent model orientation do not matter. A candidate without a recognized solid, surface, body, or mesh receives zero geometry credit even if it copied convincing 2D projections into paper space. Planar `REGION` no longer counts as true 3D geometry. The Layout1 foreground is cropped to its occupied bounds and normalized to a 128-by-128 mask before comparison with a four-pixel tolerance. This removes irrelevant page margins and framing while retaining missing, extra, and wrong view content. Presentation scoring requires a populated `Layout1` and compares normalized viewport directions after de-duplicating support viewports created by native `DRAWINGVIEW` objects. A flattened paper-space presentation containing real paper curves is representation-equivalent: when it has fewer than two viewports, camera matching is non-applicable and the trusted `Layout1` render remains the view evidence. Inspection schema v4 isolates genuine 3D bounds. Historical schema-v3 metadata combined them with hidden AutoCAD drawing-view helpers, frequently producing false extents around −10,000. For schema v3, 3D extent similarity is therefore non-applicable instead of a false zero. Evaluator v5/schema v4 enables the physical-size component after its gold cache is rebuilt. Score JSON records values plus explicit applicability flags, counts, evaluator version, and scorer-config hash. `linework_similarity` and `extent_similarity` remain compatibility aliases for registered visual and geometry-extent similarity. ## Legacy raster comparison (v1) The evaluator renders candidate and gold using the same fixed canvas and fit-to-extents rules. Python converts both images to foreground masks, applies a two-pixel tolerance, and computes a symmetric precision/recall F1 score. This path remains only for 2D entries with no sampled vector geometry. World-space size is scored separately: ```text width_ratio = min(candidate_width / gold_width, gold_width / candidate_width) height_ratio = min(candidate_height / gold_height, gold_height / candidate_height) extent_similarity = sqrt(width_ratio * height_ratio) ``` The legacy combined score was: ```text score = 0.8 * linework_similarity + 0.2 * extent_similarity ``` Translation was ignored because the evaluator fit model-space extents to the canvas. This metric is retained only for loading and reproducing historical v1 results. It is not used when vector features are present because annotation styles could change plot extents and cause nearly corresponding geometry to miss the two-pixel tolerance. The scorer emits a diagnostic PNG: matching foreground is black, gold-only foreground is red, and candidate-only foreground is blue. Binary dilation uses an integral image, while vector nearest-neighbor work is spatially indexed and sample-capped. Dense or radically wrong-scale drawings no longer trigger quadratic stalls. ## Determinism and version gates Candidate and gold results must have identical evaluator versions and render-configuration hashes. Gold cache PNG and metadata checksums are validated on every load. The scorer configuration has its own hash, and repeated scoring of identical inputs produces identical result JSON.