LLM Prompt Injection Eval Framework

Group 5 Capstone  ·  Branch: juan/integrate-p567-v1  →  merged to main PR #5 ✓ MERGED

At a glance

Total experiments run
16
4 models × 2 datasets × baseline + mitigated
Attack cases evaluated
400
25 cases per experiment run
Extraction mitigation
0%
ASR after mitigation — all 4 models
Scorer bugs fixed
3
mapping_confidence · mitigation_blocked · <think> strips

System Architecture — Evaluation Pipeline

🗂️
Dataset
attack_cases.jsonl
HackAPrompt · TensorTrust
🎲
Selector
Seeded sampling
seed=42, n=25
🛡️
Pre-Mitigation
surface_aligned
blocklist · isolation
🤖
LLM
Groq API
4 models tested
🔍
Post-Mitigation
Response filter
leakage markers
📊
Scorer
evaluate_case()
ASR · risk band
💾
Results
summary.json
case_results.jsonl

Experiment Results — All Models

Model Dataset Mode ASR Successes Refusals Manual Review Mit. Latency
llama-3.1-8b-instant (baseline model)
llama-3.1-8b HackAPrompt override Baseline 44% 11 / 2501
llama-3.1-8b HackAPrompt override Mitigated 28% 7 / 2529
llama-3.1-8b TensorTrust extraction Baseline 76% 19 / 2516
llama-3.1-8b TensorTrust extraction Mitigated 0% 0 / 2510
llama-3.3-70b-versatile
llama-3.3-70b HackAPrompt override Baseline 48% 12 / 25010.008ms
llama-3.3-70b HackAPrompt override Mitigated 32% ↓16% 8 / 25050.126ms
llama-3.3-70b TensorTrust extraction Baseline 68% 17 / 25280.009ms
llama-3.3-70b TensorTrust extraction Mitigated 0% ↓68% 0 / 25020.203ms
qwen/qwen3-32b ⚠️ anomaly detected
qwen3-32b HackAPrompt override Baseline 60% 15 / 25060.010ms
qwen3-32b HackAPrompt override Mitigated 72% ↑12% 18 / 25030.145ms
qwen3-32b TensorTrust extraction Baseline 96% 24 / 25010.008ms
qwen3-32b TensorTrust extraction Mitigated 0% ↓96% 0 / 25010.202ms
meta-llama/llama-4-scout-17b ⚠️ anomaly detected
llama-4-scout HackAPrompt override Baseline 36% 9 / 25050.007ms
llama-4-scout HackAPrompt override Mitigated 48% ↑12% 12 / 25140.165ms
llama-4-scout TensorTrust extraction Baseline 92% 23 / 25320.008ms
llama-4-scout TensorTrust extraction Mitigated 0% ↓92% 0 / 25010.229ms

Key Findings & Insights

✅ Universal Win

Mitigation kills extraction attacks

The surface_aligned mitigation drops extraction ASR to 0% across every model tested — llama-3.1-8b, llama-3.3-70b, qwen3-32b, and llama-4-scout. TensorTrust extraction is our most reliable benchmark: no manual review cases, no false flags.

⚠️ Anomaly

Mitigation backfires on override attacks

qwen3-32b: 60% → 72%. llama-4-scout: 36% → 48%. Adding security instructions to the system prompt created conflicting signals that made these models more compliant with attackers. Documented in Wei et al. (NeurIPS 2023) as "competing objectives" failure.

🔬 Research Finding

Model architecture matters

llama-3.3-70b responded correctly to mitigation on override (48%→32%), while smaller or reasoning-oriented models did not. Mitigation is not one-size-fits-all — the same strategy can help one model and hurt another.

🐛 False Positive

<think> block scorer bug

qwen3-32b emits internal chain-of-thought in <think> tags. The model quoted the attack target while resisting it in its reasoning trace. Scorer found the substring and called it a success. Fixed by stripping think blocks before scoring.

Scoring Fixes — What Was Wrong & What We Fixed

Bug 1 — mapping_confidence false flags

All 1,346 TensorTrust cases carry mapping_confidence='low'. A condition in manual_review logic flagged every unresolved case, causing 23/25 TensorTrust hijacking cases to be sent to manual review even when outcomes were unambiguous. Removed the condition entirely.

Bug 2 — mitigation_blocked not propagated

When the pre-request mitigation blocked an attack, the scorer still flagged it as manual_review because attack_category='prompt_leakage' always triggered that path. Added mitigation_blocked and response_mitigation_blocked kwargs to evaluate_case() so confirmed blocks are excluded from review queue.

Bug 3 — <think> block false positives (qwen3-32b)

Reasoning models output internal scratchpad in <think>…</think> tags. Scorer was scanning the full response including the scratchpad. New _strip_think_blocks() function removes all think content before any match is attempted. Fixes at least 2 of 3 anomalous qwen32b override cases.

Attack Taxonomy Covered

instruction_override
HackAPrompt
Attacker replaces original task instructions with new ones
evasion_obfuscation
HackAPrompt
Misspellings, encoding tricks, indirect phrasing to bypass filters
prompt_leakage / extraction
TensorTrust
Forces model to reveal system prompt or secret context
goal_hijacking
TensorTrust
Redirects model to attacker's goal instead of original task

Tech Stack

Language
Python 3.12
LLM API
Groq (OpenAI-compatible)
Mitigation
surface_aligned strategy
Datasets
HackAPrompt · TensorTrust
Reproducibility
seed=42 · SHA-256 hash
Output format
JSONL + JSON manifests
Version control
GitHub · PR #5 merged
Team
Camilo · Juan · Amal · Tumadhir · Mario · Ratticha