File size: 2,914 Bytes
eaae571 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | # LLM Prompt Injection Security Evaluation Framework
## Project Overview
### Project Type
- University research capstone project focused on LLM security.
### Project Objective
- Design and implement a testing framework that evaluates how vulnerable LLM chatbots are to prompt injection attacks.
- Evaluate how mitigation strategies improve security outcomes.
The system will:
1. Simulate prompt injection attacks.
2. Measure how often attacks succeed.
3. Apply mitigation strategies.
4. Measure how much mitigation reduces successful attacks.
## Scope Clarification
### This Project Is NOT
- Training LLM models
- Building a new AI model
- Building a ChatGPT-like system
- Performing large-scale machine learning training
### This Project IS
- Designing attack scenarios
- Evaluating LLM security vulnerabilities
- Implementing defensive mitigation strategies
- Measuring improvements after mitigation
## Attack Scenarios to Evaluate
- Direct prompt injection through user inputs
- Attempts to extract hidden system prompts or internal instructions
- Indirect prompt injection through retrieved documents (RAG scenarios)
- Adaptive attackers who modify prompts after observing system responses
The framework should allow easily adding new attack strategies.
## Mitigation Strategies
- Prompt hardening
- Input filtering
- Instruction isolation
- Detection mechanisms (e.g., classifier-based detection)
The framework should support comparing multiple mitigation approaches.
The system should also allow analyzing trade-offs such as:
- Security improvement
- Usability impact
- False positives or false negatives
## Evaluation Metrics
The system should measure:
- Attack Success Rate (ASR)
- Reduction in successful attacks after mitigation
- False positive / false negative rates for detection methods
- Impact on normal system performance
## Models
The framework may test different LLMs including:
- API-based models (e.g., GPT-family models)
- Open-source models (e.g., LLaMA, Mistral)
The architecture should support easily adding new models.
## System Architecture (High Level)
The pipeline will conceptually look like this:
LLM Chatbot
↓
Attack Generator
↓
Attack Execution
↓
Mitigation Layer
↓
Evaluation Metrics
↓
Results & Analysis
The framework should support running multiple experiments and logging results.
## Repository Goals
The repository should prioritize:
- Modular architecture
- Reproducible experiments
- Clear documentation
- Experiment logging
- Easy extension for new attacks
- Easy extension for new mitigation strategies
## Research Inspiration
This project is inspired by research on prompt injection attacks and LLM security, including academic papers that:
- Formalize prompt injection attacks
- Benchmark attack strategies
- Compare defense mechanisms
The goal is to build a practical experimental framework inspired by this research.
|