File size: 2,017 Bytes
eaae571 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | # Phase 5 Baseline Evaluation
This guide explains how the framework currently handles baseline evaluation on
the active corpus and how the team should run the main baseline slices.
## Purpose
Phase 5 is the evaluation stage before mitigation comparison.
It is used to:
- run focused baseline slices
- inspect attack successes, leakage, refusals, and ambiguous outcomes
- strengthen scoring based on real outputs
- prepare clean baseline runs for later defended comparisons
## Main Baseline Slices
The main current baseline slices are:
- `configs/hackaprompt_override_25.yaml`
- `configs/tensortrust_hijacking_25.yaml`
For local validation without a hosted backend:
- `configs/hackaprompt_override_25_mock.yaml`
- `configs/tensortrust_hijacking_25_mock.yaml`
## Matching Defended Slices
These run the same subsets with the current mitigation layer:
- `configs/hackaprompt_override_25_mitigated.yaml`
- `configs/tensortrust_hijacking_25_mitigated.yaml`
Mock equivalents:
- `configs/hackaprompt_override_25_mock_mitigated.yaml`
- `configs/tensortrust_hijacking_25_mock_mitigated.yaml`
## Recommended Workflow
1. Run the baseline config.
2. Review `summary.json` and `case_results.jsonl`.
3. Note successes, refusals, and ambiguous outputs.
4. Run the matching defended config.
5. Compare the two summaries with `scripts/compare_runs.py`.
## What To Look For
- attack success rate
- prompt leakage cases
- refusals
- manual-review volume
- which categories stay strong or weak under attack
- whether mitigation improves the same slice later
## Real-Run Support
Useful supporting docs:
- `docs/GROQ_RUNBOOK.md`
- `docs/RESULTS_TEMPLATE.md`
## Important Note On Tests
Automated test count is not dataset size.
- `pytest` validates framework behavior
- the active corpus contains `2,346` normalized cases
- experiment runs apply the framework to sampled dataset slices
So `33 tests passed` means the code is validated by 33 automated checks. It
does not mean only 33 attack records were evaluated.
|