docs/PHASE5_BASELINE_EVALUATION.md

Phase 5 Baseline Evaluation

This guide explains how the framework currently handles baseline evaluation on the active corpus and how the team should run the main baseline slices.

Purpose

Phase 5 is the evaluation stage before mitigation comparison.

It is used to:

• run focused baseline slices
• inspect attack successes, leakage, refusals, and ambiguous outcomes
• strengthen scoring based on real outputs
• prepare clean baseline runs for later defended comparisons

Main Baseline Slices

The main current baseline slices are:

• configs/hackaprompt_override_25.yaml
• configs/tensortrust_hijacking_25.yaml

For local validation without a hosted backend:

• configs/hackaprompt_override_25_mock.yaml
• configs/tensortrust_hijacking_25_mock.yaml

Matching Defended Slices

These run the same subsets with the current mitigation layer:

• configs/hackaprompt_override_25_mitigated.yaml
• configs/tensortrust_hijacking_25_mitigated.yaml

Mock equivalents:

• configs/hackaprompt_override_25_mock_mitigated.yaml
• configs/tensortrust_hijacking_25_mock_mitigated.yaml

Recommended Workflow

1. Run the baseline config.
2. Review summary.json and case_results.jsonl.
3. Note successes, refusals, and ambiguous outputs.
4. Run the matching defended config.
5. Compare the two summaries with scripts/compare_runs.py.

What To Look For

• attack success rate
• prompt leakage cases
• refusals
• manual-review volume
• which categories stay strong or weak under attack
• whether mitigation improves the same slice later

Real-Run Support

Useful supporting docs:

• docs/GROQ_RUNBOOK.md
• docs/RESULTS_TEMPLATE.md

Important Note On Tests

Automated test count is not dataset size.

• pytest validates framework behavior
• the active corpus contains 2,346 normalized cases
• experiment runs apply the framework to sampled dataset slices

So 33 tests passed means the code is validated by 33 automated checks. It does not mean only 33 attack records were evaluated.