File size: 6,152 Bytes
eaae571 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | <svg xmlns="http://www.w3.org/2000/svg" width="1400" height="860" viewBox="0 0 1400 860">
<defs>
<style>
.title { font: 700 30px Arial, sans-serif; fill: #17324d; }
.subtitle { font: 500 16px Arial, sans-serif; fill: #35516b; }
.section { font: 700 20px Arial, sans-serif; fill: #17324d; }
.label { font: 600 16px Arial, sans-serif; fill: #17324d; }
.body { font: 14px Arial, sans-serif; fill: #243746; }
.small { font: 12px Arial, sans-serif; fill: #4c6377; }
.box { fill: #f8fbff; stroke: #7aa5c8; stroke-width: 2; rx: 14; ry: 14; }
.box2 { fill: #f4faf5; stroke: #75b58a; stroke-width: 2; rx: 14; ry: 14; }
.box3 { fill: #fff8f1; stroke: #d39a56; stroke-width: 2; rx: 14; ry: 14; }
.box4 { fill: #faf5ff; stroke: #9b7ad1; stroke-width: 2; rx: 14; ry: 14; }
.arrow { stroke: #4b6784; stroke-width: 3; fill: none; marker-end: url(#arrowhead); }
</style>
<marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
<polygon points="0 0, 10 3.5, 0 7" fill="#4b6784"/>
</marker>
</defs>
<text x="50" y="48" class="title">Prompt Injection Evaluation Framework - Current MVP Architecture</text>
<text x="50" y="78" class="subtitle">Current state: normalized attack corpus, mock + Groq model adapters, minimal scoring, reproducible run artifacts</text>
<rect x="50" y="120" width="270" height="250" class="box"/>
<text x="75" y="155" class="section">1. Data Sources</text>
<text x="75" y="190" class="label">Benchmarks / corpora</text>
<text x="75" y="215" class="body">- HackAPrompt</text>
<text x="75" y="240" class="body">- TensorTrust</text>
<text x="75" y="275" class="label">Key files</text>
<text x="75" y="300" class="body">data/curated/*</text>
<text x="75" y="325" class="body">data/normalized/attack_cases.jsonl</text>
<rect x="360" y="120" width="300" height="250" class="box2"/>
<text x="385" y="155" class="section">2. Attack Taxonomy in Use</text>
<text x="385" y="190" class="body">- Direct instruction override</text>
<text x="385" y="214" class="body">- Prompt leakage</text>
<text x="385" y="238" class="body">- Indirect prompt injection</text>
<text x="385" y="262" class="body">- Goal hijacking</text>
<text x="385" y="286" class="body">- Role / context manipulation</text>
<text x="385" y="310" class="body">- Adaptive / multi-turn</text>
<text x="385" y="334" class="body">- Tool-mediated attacks</text>
<text x="385" y="358" class="body">- Semantic separation / obfuscation</text>
<rect x="710" y="120" width="300" height="270" class="box3"/>
<text x="735" y="155" class="section">3. Orchestration Layer</text>
<text x="735" y="190" class="label">Main files</text>
<text x="735" y="215" class="body">orchestration/run_experiment.py</text>
<text x="735" y="240" class="body">orchestration/config.py</text>
<text x="735" y="265" class="body">orchestration/runner.py</text>
<text x="735" y="300" class="label">Role</text>
<text x="735" y="325" class="body">Loads config, samples cases,</text>
<text x="735" y="350" class="body">calls model adapter, scores output,</text>
<text x="735" y="375" class="body">writes experiment artifacts</text>
<rect x="1050" y="120" width="300" height="250" class="box4"/>
<text x="1075" y="155" class="section">4. Model Adapters</text>
<text x="1075" y="190" class="label">Main files</text>
<text x="1075" y="216" class="body">models/base.py</text>
<text x="1075" y="240" class="body">models/mock.py</text>
<text x="1075" y="264" class="body">models/groq.py</text>
<text x="1075" y="300" class="label">Backends today</text>
<text x="1075" y="326" class="body">- mock: local pipeline validation</text>
<text x="1075" y="350" class="body">- groq: live hosted inference</text>
<path d="M 320 245 L 360 245" class="arrow"/>
<path d="M 660 245 L 710 245" class="arrow"/>
<path d="M 1010 245 L 1050 245" class="arrow"/>
<rect x="120" y="470" width="320" height="220" class="box"/>
<text x="145" y="505" class="section">5. Evaluation Layer</text>
<text x="145" y="540" class="label">Main file</text>
<text x="145" y="566" class="body">evaluation/scoring.py</text>
<text x="145" y="602" class="label">Current metric</text>
<text x="145" y="628" class="body">- attack_success</text>
<text x="145" y="652" class="small">Current MVP uses a simple expected-target</text>
<text x="145" y="672" class="small">substring check. Stronger category-specific</text>
<text x="145" y="692" class="small">scoring is the next scaling step.</text>
<rect x="540" y="470" width="320" height="220" class="box2"/>
<text x="565" y="505" class="section">6. Artifact / I/O Layer</text>
<text x="565" y="540" class="label">Main file</text>
<text x="565" y="566" class="body">utils/io.py</text>
<text x="565" y="602" class="label">Outputs per run</text>
<text x="565" y="628" class="body">- manifest.json</text>
<text x="565" y="652" class="body">- case_results.jsonl</text>
<text x="565" y="676" class="body">- summary.json</text>
<rect x="960" y="470" width="320" height="220" class="box3"/>
<text x="985" y="505" class="section">7. Configs / Validation</text>
<text x="985" y="540" class="body">configs/baseline.yaml</text>
<text x="985" y="564" class="body">configs/groq_smoke.yaml</text>
<text x="985" y="588" class="body">configs/groq_20.yaml</text>
<text x="985" y="624" class="body">tests/test_smoke.py</text>
<text x="985" y="660" class="small">Validated with mock runs and live Groq</text>
<text x="985" y="680" class="small">smoke / 20-case experiments.</text>
<path d="M 1180 370 L 1180 430 L 280 430 L 280 470" class="arrow"/>
<path d="M 1180 370 L 1180 470 L 700 470" class="arrow"/>
<path d="M 1180 370 L 1180 470" class="arrow"/>
<text x="50" y="790" class="section">Current state summary</text>
<text x="50" y="820" class="body">Today the project has a working MVP: it can load normalized attack data, run experiments through mock or Groq, score outputs with a basic evaluator, and save reproducible artifacts for analysis.</text>
</svg>
|