# Architecture

AutoCAD Bench is a filesystem-first benchmark with a trusted Windows execution boundary. Task definitions and configurations are versioned inputs; every rollout writes an immutable trace directory that the evaluator and dashboard consume.

## Execution phases

1. **Discover** — load and validate the task corpus and a model/run configuration.
2. **Preflight** — verify credentials, model availability, backend capacity and
   worker identity, and every required gold-cache entry.
3. **Run** — launch or attach to one isolated AutoCAD sandbox, hand off the exact task, and execute the model's computer actions.
4. **Finalize** — retrieve the final DWG and complete trace evidence.
5. **Evaluate** — inspect the DWG inside trusted AutoCAD, compute deterministic diagnostics, and run the vision judge.
6. **Report** — expose normalized benchmark, task, scoring, usage, and trace data through the local dashboard.

```text
tasks/                  configs/
   │                        │
   └──────────┬─────────────┘
              ▼
   orchestration/batch.py
              │
        bounded workers
              │
              ▼
 infrastructure plugin ─────── selected worker backend
              │                         │
              ▼                         ▼
       harness/run.py  ◄────── Windows broker + AutoCAD
              │                         │
              └──── final DWG ──────────┘
                          │
                          ▼
                 evaluation/run.py
                          │
                          ▼
                    runs/<run-id>/
                          │
                          ▼
                    apps/dashboard/
```

## Task model

`tasks/manifest.jsonl` is the canonical catalog. Each row identifies one reference PNG, one private gold DWG, drawing type, difficulty, split, and execution budgets. The loader verifies:

- exactly 50 entries with contiguous IDs from `task-001` to `task-050`;
- 21 2D and 29 3D tasks;
- valid PNGs and AutoCAD `AC1032` DWGs;
- paths that remain inside the repository;
- a fixed seven-task development split; and
- no reused reference or gold artifact.

`tasks/gold-audit.jsonl` records byte sizes, image dimensions, DWG signatures, and manual-review state. Public task models omit the gold path before any data reaches the model harness.

## Package boundaries

| Package | Responsibility |
|---|---|
| `autocad_bench.tasks` | Corpus validation, public task data, and prompts |
| `autocad_bench.harness` | Provider interaction, agent loop, and trace writing |
| `autocad_bench.infrastructure` | Worker backend registration, launch, cleanup, and optional recovery |
| `autocad_bench.sandbox` | Broker protocol, AWS lifecycle, handoff, and evidence retrieval |
| `autocad_bench.orchestration` | Model catalog checks, batch planning, concurrency, resume, and recovery |
| `autocad_bench.evaluation` | Trusted inspection, gold cache, deterministic scoring, and vision judgment |
| `autocad_bench.common` | Atomic filesystem utilities shared across boundaries |

The Python code uses a `src/` layout so imports always resolve through the installed package rather than the working directory. Commands normally discover the checkout from `tasks/manifest.jsonl`; an installed wheel can use `AUTOCAD_BENCH_ROOT` to point at a benchmark checkout.

## Sandbox boundary

The model has no shell, filesystem, AutoLISP, API, MCP, or network tools. It can send only validated mouse and keyboard actions to the loopback broker. The broker captures every post-action frame for audit while the model sees only the observation returned at the end of its action batch.

The Windows source under `sandbox/windows/` builds three runtime layers:

- the authenticated loopback broker;
- the PyAutoGUI interactive input worker; and
- the trusted AutoCAD evaluator plug-in and runner.

The built-in AWS backend uses SSM and an inbound-free security group. Other
backends must enforce an equivalent isolation and admission contract. Runtime
admission verifies the current boot, deployment manifest, broker identity,
display, input driver, and exact-task reset before the model receives control.

## Evaluation boundary

The final DWG is scored only after model execution. For 2D tasks, evaluator v4 renders Model Space. For 3D tasks, it inspects true geometry and all paper layouts but renders the required `Layout1` presentation. Gold-cache entries fail closed if their source checksum, evaluator version, render configuration, drawing type, or required 3D layout contract differs.

The dashboard-preferred score is the structured vision judgment. Deterministic geometry and rendering comparisons remain audit evidence and regression signals.

## Results layout

```text
runs/benchmarks/<benchmark-id>/
├── batch-state.json
└── rollouts/<task-id>/
    ├── run-state.json
    ├── reference.png
    ├── attempt.dwg
    ├── frames/
    ├── model_turns.jsonl
    ├── model_turn_summaries.jsonl
    ├── input_receipts.jsonl
    ├── events.json
    ├── frame_events.json
    └── evaluation/
        ├── status.json
        ├── metadata.json
        ├── candidate-render.png
        ├── score.json
        └── vision-judge.json
```

Writes that drive live status are atomic. Completed traces are immutable; recovery creates explicit evidence and promotion records rather than silently rewriting outcome history.