# Phase 5 Baseline Evaluation

This guide explains how the framework currently handles baseline evaluation on
the active corpus and how the team should run the main baseline slices.

## Purpose

Phase 5 is the evaluation stage before mitigation comparison.

It is used to:

- run focused baseline slices
- inspect attack successes, leakage, refusals, and ambiguous outcomes
- strengthen scoring based on real outputs
- prepare clean baseline runs for later defended comparisons

## Main Baseline Slices

The main current baseline slices are:

- `configs/hackaprompt_override_25.yaml`
- `configs/tensortrust_hijacking_25.yaml`

For local validation without a hosted backend:

- `configs/hackaprompt_override_25_mock.yaml`
- `configs/tensortrust_hijacking_25_mock.yaml`

## Matching Defended Slices

These run the same subsets with the current mitigation layer:

- `configs/hackaprompt_override_25_mitigated.yaml`
- `configs/tensortrust_hijacking_25_mitigated.yaml`

Mock equivalents:

- `configs/hackaprompt_override_25_mock_mitigated.yaml`
- `configs/tensortrust_hijacking_25_mock_mitigated.yaml`

## Recommended Workflow

1. Run the baseline config.
2. Review `summary.json` and `case_results.jsonl`.
3. Note successes, refusals, and ambiguous outputs.
4. Run the matching defended config.
5. Compare the two summaries with `scripts/compare_runs.py`.

## What To Look For

- attack success rate
- prompt leakage cases
- refusals
- manual-review volume
- which categories stay strong or weak under attack
- whether mitigation improves the same slice later

## Real-Run Support

Useful supporting docs:

- `docs/GROQ_RUNBOOK.md`
- `docs/RESULTS_TEMPLATE.md`

## Important Note On Tests

Automated test count is not dataset size.

- `pytest` validates framework behavior
- the active corpus contains `2,346` normalized cases
- experiment runs apply the framework to sampled dataset slices

So `33 tests passed` means the code is validated by 33 automated checks. It
does not mean only 33 attack records were evaluated.