# Portable benchmark bundles

The Python package and benchmark data have separate release boundaries. The
wheel contains executable code; a benchmark bundle contains scorer-side task
definitions, reference images, gold DWGs, audit metadata, and optionally one or
more trusted evaluator caches.

This lets an operator upgrade orchestration code without silently changing the
evaluation corpus, and lets a published corpus be verified before any model or
infrastructure credentials are loaded.

## Export

From a complete checkout:

```bash
uv run autocad-bench bundle-export \
  --output dist/autocad-bench-corpus-v1
```

The default export includes the active evaluator cache. To publish only source
tasks and build the cache later on operator-owned AutoCAD infrastructure:

```bash
uv run autocad-bench bundle-export \
  --output dist/autocad-bench-corpus-v1-source \
  --source-only
```

`--evaluator-version` is repeatable when a release intentionally carries
multiple evaluator versions.

Export is allowlist-based. It copies:

- `tasks/manifest.jsonl`, the audit inventory, canonical suite, and task README;
- each PNG and gold DWG referenced by the manifest; and
- only the selected evaluator cache metadata/renders.

It never copies `.env` files, benchmark configs, controller units, run outputs,
cloud identifiers, or credentials. The output path must not already exist, so
an export cannot merge with stale or unrelated files.

## Validate and use

```bash
autocad-bench bundle-validate \
  --root /opt/autocad-bench-data

export AUTOCAD_BENCH_ROOT=/opt/autocad-bench-data
autocad-bench validate
autocad-bench validate-audit
```

`bundle-validate` checks:

1. the bundle schema and version;
2. the exact file allowlist, rejecting missing or additional files;
3. every byte count and SHA-256 digest;
4. all 50 manifest and audit invariants, including contiguous task IDs;
5. the complete ordered 50-task suite; and
6. every included gold-cache entry against its source DWG checksum and
   evaluator contract.

Symbolic links are rejected. Manifest paths remain relative to the bundle root
and cannot escape it.

After validation, the normal harness, batch controller, sandbox wrapper,
evaluator, and dashboard resolve data through `AUTOCAD_BENCH_ROOT`. Provider
keys and infrastructure settings remain outside the bundle.

## Release ownership

The bundle manifest is `autocad-bench-bundle.json`. Treat the directory as an
immutable release artifact. The local release builder creates a deterministic
archive and publishes its digest alongside the code artifacts; see
[Manual releases](releases.md). The bundle format does not prescribe a registry
or cloud provider.