docs/BENCHMARK_MANIFEST_TEMPLATE.md

Benchmark Manifest Template

This template defines the first manifest for the active intake set. It is meant to stay narrow and practical while the team is working only with:

• AgentDojo
• PINT
• HackAPrompt

Purpose

The manifest gives each source a stable record before case extraction begins. It records where the source came from, what kind of cases it contributes, and how it should be handled in normalization.

Recommended Structure

{
  "manifest_version": "1.0",
  "project": "prompt_injection_security_eval",
  "active_sources": [
    {
      "source_id": "agentdojo",
      "source_name": "AgentDojo",
      "source_type": "benchmark",
      "priority": 1,
      "status": "planned",
      "case_focus": [
        "agentic attack scenarios",
        "tool-mediated prompt injection",
        "security-utility trade-off cases"
      ],
      "expected_outputs": [
        "raw source notes",
        "normalized attack cases",
        "source-specific extraction log"
      ],
      "notes": "Primary benchmark source for the first intake round."
    },
    {
      "source_id": "pint",
      "source_name": "PINT",
      "source_type": "benchmark",
      "priority": 2,
      "status": "planned",
      "case_focus": [
        "structured prompt injection cases",
        "benchmark-style attack prompts",
        "cross-check coverage against AgentDojo"
      ],
      "expected_outputs": [
        "raw source notes",
        "normalized attack cases",
        "source-specific extraction log"
      ],
      "notes": "Second benchmark source to broaden coverage."
    },
    {
      "source_id": "hackaprompt",
      "source_name": "HackAPrompt",
      "source_type": "competition",
      "priority": 3,
      "status": "planned",
      "case_focus": [
        "human-generated attack phrasing",
        "creative jailbreak patterns",
        "diversity of prompt styles"
      ],
      "expected_outputs": [
        "raw source notes",
        "normalized attack cases",
        "source-specific extraction log"
      ],
      "notes": "Use as the first diversity source after benchmark intake."
    }
  ]
}

Fields To Keep Stable

• source_id
• source_name
• source_type
• priority
• status
• case_focus
• expected_outputs
• notes
• source_reference
• source_version
• benchmark_split
• target_model_family
• task_domain
• language
• license_or_usage_note

Suggested Status Values

• planned
• in_extraction
• normalized
• reviewed

Extraction Checklist

1. Confirm the source is in the active intake set.
2. Record a source note before extracting cases.
3. Assign an initial taxonomy label to each case.
4. Preserve the source wording as much as possible.
5. Normalize each case into the shared schema.
6. Mark the source as complete only after the extraction log is filled.