# Desktop Broker Contract Implementation status on 2026-07-17: v7 moved click-to-focus and visual recovery to the model; v8 added checksum-bound current-boot handoff; v9 added exactly-once actions. V10 source adds replay-safe lease/reset, all-operation transport recovery, supervised SSM tunneling, and non-ambiguous post-action animation evidence. Operators must build and admit their own v10-compatible worker image. See [`deterministic-sandbox-contract.md`](./deterministic-sandbox-contract.md). The desktop broker is an authenticated private control plane between the benchmark harness and a Windows/AutoCAD session. It is never exposed to the model as MCP, a tool server, or a network endpoint. The model emits one normalized action; harness code validates it and calls the broker. ## Lifecycle ```text lease_session -> reset_session -> get_screenshot/execute_action* -> checkpoint_save -> retrieve_artifact + retrieve_events + retrieve_frame_events -> release_session ``` `get_status` is available throughout orchestration. Release is idempotent. The real broker must also release expired leases and clean attempt directories after its configured timeout. Session states are `leased`, `ready`, `running`, `done`, `released`, and `error`. ## Operations - `lease_session(task_id, attempt_id)`: reserve one clean desktop. The pair is an idempotency identity; an identical replay returns the active lease. - `reset_session(session_id, request_id)`: restore the pinned state, prepare `attempt.dwg`, maximize AutoCAD over the complete pinned Windows work area, dismiss known environmental overlays, wait for a rendered/stable ribbon, initialize both input logs, and return the initial screenshot. An identical request returns the cached response. It does not choose the model's future keyboard focus. - `get_screenshot(session_id)`: return the live desktop PNG, dimensions, sequence, timestamp, observed AutoCAD-maximized flag, and checksum. Post-handoff dialogs are allowed. - `execute_action(session_id, action, request_id, expected_action_index)`: validate and execute exactly one action, wait for bounded visual stability, then return the updated screenshot plus input-target and stability receipt evidence. An identical retry returns the cached response with `idempotent_replay=true` and performs no second injection. If post-commit animation does not settle, return the freshest frame with `post_action_visual_stable=false`; never throw an ambiguous action failure after input commit. - `checkpoint_save(session_id)`: accept an already-saved changed artifact; otherwise issue the documented controller save only when an AutoCAD-owned window is foreground. A foreign save dialog must be completed by the agent first. - `retrieve_artifact(session_id)`: return the saved candidate DWG bytes. - `retrieve_events(session_id)`: return Kuro-compatible raw input NDJSON. - `retrieve_frame_events(session_id)`: return screenshot-frame-aligned input NDJSON. - `get_status(session_id)`: return state, action count, save/artifact state, and last error. - `release_session(session_id)`: close/reset the desktop and delete attempt data. ## Action schema Allowed action variants are: ```text click(x, y, button) double_click(x, y, button) drag(path, button, duration_ms) move(x, y, duration_ms) scroll(dx, dy) key(keys) type(text) wait(milliseconds) done() ``` Coordinates must fall inside the pinned display bounds. Drag paths contain 2–64 points. Text is capped at 2,000 printable ASCII characters and cannot contain control characters, preventing multiline or hidden control payloads. Windows/system keys and combinations including `Win`, `Alt+F4`, `Ctrl+Alt+Delete`, and `Ctrl+Shift+Esc` are rejected. The Windows image remains the primary enforcement layer for shell, network, clipboard, AutoLISP, scripts, macros, and process access. Every action includes a required concise `intent` string describing its immediate purpose. It is trace metadata only: the harness logs and displays it, and the broker removes it before sending the normalized input action to PyAutoGUI. The model-facing harness batches above this one-action broker boundary. A `computer_action` tool call contains any non-empty ordered action batch that fits the remaining 5,000-action rollout budget, with no separate per-call item cap. Before the first broker request, the harness validates every item, remaining action budget, live desktop bounds, and that `done` is final. It then calls `execute_action` sequentially for each item with a distinct deterministic request ID and strictly increasing expected action index. This is open-loop execution, not an atomic broker transaction: a late infrastructure failure may occur after earlier items were injected, so the rollout fails closed instead of blindly retrying the complete batch. The broker still captures and logs a post-action screenshot and receipt for every sub-action. Those intermediate frames remain available to the trace/dashboard for audit, but the model receives no observation between items. After the complete batch, it receives one aggregate delivery message plus only the final screenshot. The prompt tells the model to batch only short deterministic sequences such as known-focus `type` followed by `ENTER`, and to use a one-action batch whenever the next action depends on seeing a focus change, dialog, command state, menu, or render. The broker records every accepted action server-side with its index and timestamp. Each request carries a stable `request_id` plus `expected_action_index`. Reusing the same ID with the same action and index replays the cached result; reusing it for different content or sending an unexpected index returns `action_sequence_conflict`. Mouse/keyboard actions additionally produce two run artifacts: - `events.json`: Kuro Windows-compatible NDJSON using `mouse_move`, `click`, `drag`, `scroll`, `key`, and `modifier_change`. Synthetic model input is identified with `sourcePid=-1`, `actionIndex`, and `provenance="broker_action"`. - `frame_events.json`: NDJSON rows shaped as `{frame, video_t_ms, events}`. One row is written for each broker-captured action screenshot, and it embeds inputs accepted since the prior screenshot with `video_t_ms` and `t_in_frame_ms`. Intermediate batch rows are audit-only and are not shown to the model. These are controller-ground-truth action-intent logs, not semantic proof that AutoCAD performed the intended edit. Broker v7 returns a delivery receipt after the worker verifies interactive session, `WinSta0\\Default`, display size, and final pointer coordinates where applicable. It records foreground, active, and focused HWND/process/thread/title/class before and after the action, but those fields are diagnostic-only. Keyboard input is injected into the current desktop even when no focused control is reported or focus belongs to an embedded/native child process. This matches real computer use: the model must click the intended canvas, command line, field, or dialog control, inspect the final batch screenshot, and recover when the effect is absent. Each post-action receipt also records whether bounded visual stabilization passed, its wait duration, and the final changed-pixel count. The harness rejects non-increasing frame sequences, labels successful controller results `input_injected`, and exposes exact batch-start/final hashes plus `screen_changed` to stateless model turns. The final saved/evaluated DWG remains the end-to-end task proof. Both logs are diagnostic/training artifacts and never contribute to the correctness reward. The trusted input worker uses pinned Python 3.12.10 and PyAutoGUI 0.9.54. It is launched headlessly by the broker in the broker/AutoCAD interactive session and communicates only through a private JSONL stdin/stdout pipe. The model never receives Python or broker access. “Full screen” in this benchmark means the normal AutoCAD top-level window is maximized and covers the Windows work area inside the 1920x1080 desktop. The taskbar may remain visible. It does not mean AutoCAD Clean Screen mode; the ribbon and command bar stay visible. This is a hard requirement only for the initial reset screenshot handed to the agent. After handoff, screenshots and actions follow the live desktop so Save As, File Explorer, and other native dialogs remain usable. ## Authentication and isolation - Use TLS plus a short-lived orchestration credential or mutually authenticated TLS. - Authorize every request against its lease; session IDs are not credentials. - Never place broker credentials in a model prompt or model-visible environment. - Permit broker traffic only between the evaluator/orchestrator and desktop subnet. - Keep gold drawings and the trusted evaluator off the model desktop. - Limit upload/download to the prepared attempt artifact. - Treat a valid DWG signature as necessary but not sufficient checkpoint evidence; the save path must prove that post-reset desktop changes reached the artifact. ## Local fake `FakeDesktopBroker` implements the lifecycle in process. It returns deterministic synthetic desktop PNGs with the maximized-state assertion, enforces state and coordinate validation, records actions, emits both Kuro-compatible input artifacts, requires checkpointing before DWG retrieval, returns a fixture `AC1032` artifact, and supports idempotent release. It is for harness and Verifiers integration tests; it does not claim to emulate AutoCAD behavior.