# Repro Commands — Auden v1 vs stock Gemma 4 12B

All published numbers: llama.cpp commit `76da2450a4f2cc9ce6c7fc8229e25dc0a4b41e5d`, evaluator `auden-docbench-v1-normalized` (built into the project's `scripts/eval_auden_docbench.py`), OCR-text input, Q4_K_M quants. Commands below assume the project repo root as working directory; the server runs locally (`127.0.0.1`) — substitute `<YOUR_SERVER>` if you serve remotely.

## 1. Serve the model (one server per column)

```bash
# Auden v1 (this repo)
llama-server \
  -m q4_k_m/auden-12b-Q4_K_M.gguf \
  --mmproj q4_k_m/mmproj-BF16.gguf \
  --jinja -ngl 999 --host 127.0.0.1 --port 8091

# Stock Gemma 4 12B baseline (public unsloth/gemma-4-12b-it-GGUF)
llama-server \
  -m gemma-4-12b-it-Q4_K_M.gguf \
  --mmproj mmproj-BF16.gguf \
  --jinja -ngl 999 --host 127.0.0.1 --port 8093
```

Sanity gate before any eval: one `POST /v1/chat/completions` with the Auden tool schemas and a small receipt OCR text must return `finish_reason: tool_calls` with an `extract_fields` call. Wait for `/health` = 200 first; never score against a still-loading server.

## 2. Run the three evals (swap `--server-url`/`--out-dir` per model)

```bash
# a. CORD test 100 (real receipts, OCR text; CORD-v2 test split, CC-BY-4.0)
python scripts/eval_auden_docbench.py \
  --manifest data/real/cord/manifest.jsonl \
  --corpus-root data/real/cord \
  --server-url http://127.0.0.1:<PORT>/v1/chat/completions \
  --model-id <MODEL_ID> \
  --out-dir outputs/eval/<RUN>/<MODEL>_cord_ocr_norm \
  --input-mode ocr --limit 100

# b. Abstention 120
python scripts/eval_auden_docbench.py \
  --manifest data/eval/abstention_v1/manifest.jsonl \
  --corpus-root data/eval/abstention_v1 \
  --server-url http://127.0.0.1:<PORT>/v1/chat/completions \
  --model-id <MODEL_ID> \
  --out-dir outputs/eval/<RUN>/<MODEL>_abstention_norm \
  --input-mode ocr

# c. Synthetic 500
python scripts/eval_auden_docbench.py \
  --manifest data/eval/synthetic_v1_500/manifest.jsonl \
  --corpus-root data/eval/synthetic_v1_500 \
  --server-url http://127.0.0.1:<PORT>/v1/chat/completions \
  --model-id <MODEL_ID> \
  --out-dir outputs/eval/<RUN>/<MODEL>_synthetic_500_norm \
  --input-mode ocr
```

Note: the eval script exits non-zero when scores fall below its pass thresholds (expected for the stock baseline); the report and raw outputs are still written.

## 3. Failure analyzer (CORD output)

```bash
python scripts/analyze_cord_failures.py \
  --raw outputs/eval/<RUN>/<MODEL>_cord_ocr_norm/raw_outputs.jsonl \
  --out-dir outputs/eval/<RUN>/<MODEL>_cord_ocr_norm/failure_analysis
```

## 4. Provenance

- Auden v1 numbers: `phase5_rerun` eval set; stock-Gemma baseline: `base_phase5` eval set. Reports and raw outputs are archived in the project's private artifact store along with `memory.md`/`docs/results.md` (full run history).
- v0 (historical comparison column) is reproducible from its preserved LoRA adapter (`r=16, alpha=32`) by re-merging onto `google/gemma-4-12B-it` and re-quantizing with llama.cpp.