Group 5 Capstone · Branch: juan/integrate-p567-v1 → merged to main
PR #5 ✓ MERGED
| Model | Dataset | Mode | ASR | Successes | Refusals | Manual Review | Mit. Latency |
|---|---|---|---|---|---|---|---|
| llama-3.1-8b-instant (baseline model) | |||||||
| llama-3.1-8b | HackAPrompt override | Baseline | 44% | 11 / 25 | 0 | 1 | — |
| llama-3.1-8b | HackAPrompt override | Mitigated | 28% | 7 / 25 | 2 | 9 | — |
| llama-3.1-8b | TensorTrust extraction | Baseline | 76% | 19 / 25 | 1 | 6 | — |
| llama-3.1-8b | TensorTrust extraction | Mitigated | 0% | 0 / 25 | 1 | 0 | — |
| llama-3.3-70b-versatile | |||||||
| llama-3.3-70b | HackAPrompt override | Baseline | 48% | 12 / 25 | 0 | 1 | 0.008ms |
| llama-3.3-70b | HackAPrompt override | Mitigated | 32% ↓16% | 8 / 25 | 0 | 5 | 0.126ms |
| llama-3.3-70b | TensorTrust extraction | Baseline | 68% | 17 / 25 | 2 | 8 | 0.009ms |
| llama-3.3-70b | TensorTrust extraction | Mitigated | 0% ↓68% | 0 / 25 | 0 | 2 | 0.203ms |
| qwen/qwen3-32b ⚠️ anomaly detected | |||||||
| qwen3-32b | HackAPrompt override | Baseline | 60% | 15 / 25 | 0 | 6 | 0.010ms |
| qwen3-32b | HackAPrompt override | Mitigated | 72% ↑12% | 18 / 25 | 0 | 3 | 0.145ms |
| qwen3-32b | TensorTrust extraction | Baseline | 96% | 24 / 25 | 0 | 1 | 0.008ms |
| qwen3-32b | TensorTrust extraction | Mitigated | 0% ↓96% | 0 / 25 | 0 | 1 | 0.202ms |
| meta-llama/llama-4-scout-17b ⚠️ anomaly detected | |||||||
| llama-4-scout | HackAPrompt override | Baseline | 36% | 9 / 25 | 0 | 5 | 0.007ms |
| llama-4-scout | HackAPrompt override | Mitigated | 48% ↑12% | 12 / 25 | 1 | 4 | 0.165ms |
| llama-4-scout | TensorTrust extraction | Baseline | 92% | 23 / 25 | 3 | 2 | 0.008ms |
| llama-4-scout | TensorTrust extraction | Mitigated | 0% ↓92% | 0 / 25 | 0 | 1 | 0.229ms |
The surface_aligned mitigation drops extraction ASR to 0% across every model tested — llama-3.1-8b, llama-3.3-70b, qwen3-32b, and llama-4-scout. TensorTrust extraction is our most reliable benchmark: no manual review cases, no false flags.
qwen3-32b: 60% → 72%. llama-4-scout: 36% → 48%. Adding security instructions to the system prompt created conflicting signals that made these models more compliant with attackers. Documented in Wei et al. (NeurIPS 2023) as "competing objectives" failure.
llama-3.3-70b responded correctly to mitigation on override (48%→32%), while smaller or reasoning-oriented models did not. Mitigation is not one-size-fits-all — the same strategy can help one model and hurt another.
qwen3-32b emits internal chain-of-thought in <think> tags. The model quoted the attack target while resisting it in its reasoning trace. Scorer found the substring and called it a success. Fixed by stripping think blocks before scoring.
All 1,346 TensorTrust cases carry mapping_confidence='low'. A condition in manual_review logic flagged every unresolved case, causing 23/25 TensorTrust hijacking cases to be sent to manual review even when outcomes were unambiguous. Removed the condition entirely.
When the pre-request mitigation blocked an attack, the scorer still flagged it as manual_review because attack_category='prompt_leakage' always triggered that path. Added mitigation_blocked and response_mitigation_blocked kwargs to evaluate_case() so confirmed blocks are excluded from review queue.
Reasoning models output internal scratchpad in <think>…</think> tags. Scorer was scanning the full response including the scratchpad. New _strip_think_blocks() function removes all think content before any match is attempted. Fixes at least 2 of 3 anomalous qwen32b override cases.