# Proposed Methodology _Prepared for the group proposal report due in Week 5_ ## Methodologies This project adopts an experimental cybersecurity evaluation methodology to assess how vulnerable LLM-based chatbots and lightweight agent workflows are to prompt injection attacks, and how effectively selected mitigations reduce that risk. The project will not train new language models. Instead, it will build a modular evaluation pipeline that runs prompt injection attacks against selected open-source models, records baseline vulnerability, applies mitigations, and compares results before and after defense. The methodology is designed to produce a repository, a report, and a working proof of concept that are reproducible and aligned with current prompt injection research practice (Debenedetti et al., 2024; Greshake et al., 2023; Liu et al., 2024). ## Methods ### Overall method The project will use a controlled experimental pipeline with repeated attack-and-evaluate cycles. The pipeline contains five stages: 1. Define attack taxonomy and benchmark tasks. 2. Collect or adapt prompt injection test cases. 3. Run baseline attacks against selected models. 4. Apply mitigation strategies matched to each attack surface. 5. Re-run the same attacks to measure improvement. This method was selected because it directly supports the project's core objective: quantifying vulnerability and mitigation effectiveness rather than only describing attacks conceptually, which is consistent with benchmark-oriented prompt injection research (Liu et al., 2024). ### Experimental pipeline The technical workflow will follow this sequence: 1. **Target model selection** The system will test open-source models only. The recommended starting models are Llama-family and Qwen-family models because they are the easiest to begin with and are widely used in current prompt injection experiments. Kimi K2 can be explored later as an optional extension if implementation effort remains manageable. 2. **Attack library construction** A repository of prompt injection attacks will be built and organized by taxonomy. The attack set will include direct attacks, indirect attacks, prompt leakage attempts, role or instruction override attacks, and adaptive multi-turn attacks (Greshake et al., 2023; Liu et al., 2023; Liu et al., 2024). 3. **Benchmark integration** The attack library will be supplemented with benchmark tasks and datasets, especially AgentDojo and PINT where feasible, to improve methodological rigor and comparability (Debenedetti et al., 2024). 4. **Baseline execution** Each attack will be executed against the selected model or models without mitigation enabled. Outputs and metadata will be logged for later analysis. 5. **Mitigation layer evaluation** At least one mitigation will be mapped to each major attack surface. Likely examples include prompt hardening for direct attacks, instruction/data separation for indirect attacks, and guardrail or filtering logic for tool or output abuse (Chen et al., 2025; Rebedea et al., 2023; Sahoo et al., 2024). 6. **Post-mitigation re-evaluation** The same experiments will be repeated after the mitigation layer is enabled so that improvements can be measured under comparable conditions. 7. **Reporting and proof of concept** Results will be summarized in the report and in repository documentation. The final deliverable will also include a proof-of-concept pipeline showing the taxonomy, experiment flow, baseline vulnerability, and measured mitigation effects. ### Attack taxonomy to be used To satisfy the project's minimum requirement and to keep the evaluation systematic, the attack repository will be organized around a clear taxonomy rather than a loose set of prompts. The most relevant taxonomy groups for this project are: 1. **Direct instruction override attacks** Simple user-supplied prompts such as "ignore previous instructions" that attempt to replace or dominate the system prompt. 2. **Prompt leakage attacks** Prompts designed to reveal hidden instructions, system prompts, safety policies, or internal workflow details. 3. **Indirect prompt injection attacks** Malicious instructions embedded in retrieved documents, web pages, emails, or tool outputs that the model later consumes as context. 4. **Goal hijacking attacks** Prompts that redirect the model from the user's intended task toward the attacker's objective. 5. **Role-play or context manipulation attacks** Payloads that use personas, fictional settings, fake completions, or instruction framing to bypass normal safety behaviour. 6. **Adaptive or multi-turn attacks** Iterative attacks in which the attacker changes wording based on prior refusals or filters. 7. **Tool-mediated or agent-specific attacks** Attacks targeting systems that can browse, retrieve files, call tools, or perform actions on behalf of the user. 8. **Semantic separation and obfuscation attacks** Payloads that hide intent through separators, formatting tricks, language changes, or context mixing. These taxonomy groups were chosen because they cover the core categories repeatedly identified in the literature and map well to the expected attack surfaces (Greshake et al., 2023; Liu et al., 2023; Liu et al., 2024). The implementation should include at least one proof-of-concept attack from every major taxonomy category, even if early versions remain surface-level. The preferred design is to cover all categories at least once, then deepen the strongest attack paths later. ### Benchmark datasets and scenario sources The attack cases will be grounded in benchmark and literature sources rather than only ad hoc prompt writing. The preferred benchmark sources are: - **AgentDojo** for dynamic agent-style attack scenarios, security-utility trade-off measurement, and realistic tool-mediated tasks (Debenedetti et al., 2024). - **PINT** for benchmark prompt injection cases that can supplement the direct and indirect attack repository. - **Literature-derived prompts** from Liu et al. (2024), Greshake et al. (2023), HouYi-style categories, HackAPrompt, and Tensor Trust (Liu et al., 2024; Greshake et al., 2023; Liu et al., 2023; Schulhoff et al., 2023; Toyer et al., 2023). If time or infrastructure limits prevent full use of both benchmarks, the minimum acceptable design is to use one benchmark source, preferably AgentDojo or PINT, and then supplement it with project-authored prompts tied to the same taxonomy categories. ### Minimum viable experiment design Based on the client notes, the project must at minimum demonstrate: 1. One benchmark-backed attack taxonomy. 2. One mitigation strategy. 3. One attacking or probing model setup. 4. One target model under evaluation. 5. One reproducible benchmark or dataset source such as AgentDojo or PINT. The recommended minimum configuration is: - **Target model under test**: One open-source model such as a Llama or Qwen variant. - **Attack generation / probing setup**: One attack library or attacker-assisted prompting workflow using benchmark cases from AgentDojo or PINT. - **Mitigation**: One concrete defense such as prompt hardening, guardrails, or instruction-data separation. - **Evaluation metric**: ASR before and after mitigation, with at least one supporting metric such as prompt leakage rate or benign utility. This minimum design is deliberately narrow enough to be feasible while still satisfying the project's core requirement. ### Recommended implementation order The recommended development order is: 1. Build a simple chatbot evaluation pipeline first. 2. Implement one proof-of-concept attack for each major taxonomy category. 3. Test those attacks against a primary open-source model such as Qwen or Llama. 4. Add at least one mitigation per attack surface. 5. Measure ASR and supporting metrics before and after mitigation. 6. Extend the same pipeline to a RAG scenario if time and stability permit. This staged approach keeps the first milestone achievable while still leaving room for a stronger final system. ### Why this method was chosen This approach was chosen for four reasons: - It is directly aligned with the literature on prompt injection benchmarking (Debenedetti et al., 2024; Liu et al., 2024). - It satisfies the client request for a pipeline, taxonomy, benchmark use, and measurable evaluation. - It fits the capstone scope because it emphasizes engineering and evaluation rather than model training. - It produces concrete outputs for the rubric: methodology, resources, milestones, expected outcomes, and a structured final report. ## Data Collection ### What data will be collected The project will collect four categories of data: 1. **Attack prompts and attack scenarios** These include manually designed prompts based on literature taxonomies and benchmark-derived attack cases. 2. **Model response data** For each attack, the project will record the model's response, whether the attack succeeded, whether restricted information was exposed, and whether the model followed the malicious instruction. 3. **Mitigation outcome data** The project will record how the same attack behaves once a mitigation has been applied. 4. **Operational experiment data** Metadata such as model name, attack category, attack surface, defense used, execution time, mean time to detect, and mean time to mitigate will be recorded where implementation allows. ### Sources of data Data will be collected from three main sources: - **Benchmark datasets and environments** such as AgentDojo and PINT (Debenedetti et al., 2024). - **Literature-derived attack cases** based on papers such as Liu et al. (2024), Greshake et al. (2023), and the HouYi taxonomy (Greshake et al., 2023; Liu et al., 2023; Liu et al., 2024). - **Project-authored test cases** created to fill gaps for specific attack surfaces or mitigation experiments. ### Data items recorded per run For each experiment run, the project will record: - Benchmark or dataset source. - Attack taxonomy label. - Prompt or scenario identifier. - Model under test. - Attacker setup or prompt source. - Mitigation used. - Raw model response. - Attack success label. - Prompt leakage label if applicable. - Utility outcome for benign tasks if applicable. - Timing and logging metadata. ### Data collection procedure The collection procedure will be: 1. Review the literature and extract canonical attack categories. 2. Adapt benchmark tasks and attack prompts into a unified repository format. 3. Create any missing test prompts needed for project-specific scenarios. 4. Execute prompts through a consistent experiment runner. 5. Store outputs and labels in structured logs. 6. Re-run the same test set after mitigation so the before/after comparison remains fair. ### Why this data collection strategy was chosen This data collection method balances rigor and feasibility. Benchmark data improves credibility, while project-authored prompts allow the team to tailor the evaluation to the client's requested attack surfaces and proof-of-concept design (Debenedetti et al., 2024; Liu et al., 2024). ## Data Analysis ### Primary analysis technique The primary analysis technique will be quantitative comparison of model behaviour before and after mitigation. The main metric will be **Attack Success Rate (ASR)**, because ASR is the most common and accepted metric in the literature for prompt injection evaluation (Debenedetti et al., 2024; Liu et al., 2024). ASR will be calculated as: `ASR = successful attacks / total attacks in a category` This will be measured: - Overall. - Per model. - Per attack type. - Per attack surface. - Per mitigation strategy. ### Additional analysis metrics Because ASR alone is not enough, the project will also analyse: - **Prompt leakage rate**: frequency of successful extraction of hidden instructions or system prompts. - **Benign utility or task completion rate**: whether the model can still perform normal tasks after mitigation. - **False positive rate**: how often a mitigation blocks benign inputs. - **False negative rate**: how often a mitigation fails to block a malicious input. - **Mean time to detect (MTTD)**: average time or number of steps required to identify an injection attempt. - **Mean time to mitigate (MTTM)**: average time or number of steps required to apply a mitigation or neutralize the attack. - **Severity or risk category**: a structured classification such as low, medium, high, or critical, based on factors such as instruction override, harmful output generation, sensitive information leakage, or unauthorized tool behavior. These supporting metrics are useful because recent work evaluates not only attack success, but also operational response and the security-utility trade-off of defensive interventions (Debenedetti et al., 2024; Liu et al., 2024). ### LLM-as-a-judge validation An additional optional validation method is to use a stronger LLM as a secondary evaluator. In this setup, the project first computes its own attack labels and metrics, then uses an external LLM judge to review selected outputs and provide a second opinion on whether the attack succeeded or whether the mitigation was effective. This should not replace the primary evaluation logic, but it can be useful for checking whether manual or rule-based scoring is broadly consistent with a model-based evaluator. ### Qualitative analysis Quantitative metrics will be supplemented by qualitative review of selected outputs. This is necessary because some attacks may partially succeed in ways that a simple binary score does not fully capture. Qualitative analysis will focus on: - How the model interprets malicious instructions. - Whether the mitigation changes model reasoning or refusal patterns. - Which attack types remain effective after defense. - What residual risks remain for deployment. ### Comparative analysis Where resources permit, the project will compare: - Llama versus Qwen style open-source models. - Direct versus indirect attack surfaces. - Single-turn versus adaptive attacks. - Different mitigation strategies for the same attack category. This comparative approach is important because the literature shows that defenses often perform unevenly across models and attack types (Debenedetti et al., 2024; Liu et al., 2024; Pandya et al., 2025). ## Models Chosen The project will prioritize a small but representative set of models rather than a large benchmark sweep. The likely model groups are: - **Open-source models** such as Llama and Qwen variants, because they are accessible, reproducible, and suitable for proof-of-concept experimentation. - **Optional extension models** such as Kimi K2, if setup complexity and time allow additional comparison. These models were chosen to support a useful contrast: - Open-source models allow reproducible testing without depending on API access. - Comparing more than one open-source family makes the evaluation more meaningful than testing a single model family. If compute or setup becomes a constraint, the methodology can still remain valid with one primary open-source model and one benchmark-backed mitigation experiment. ### Recommended model setup for this project To stay aligned with the client notes and likely capstone constraints, the recommended sequence is: 1. **Primary target model** Start with one open-source model, preferably a Llama-family or Qwen-family model, because it is reproducible, practical for a proof of concept, and feasible within project constraints. 2. **Optional comparison model** Add a second open-source model such as Qwen or Llama if feasible. This strengthens the analysis but should be treated as an extension rather than a dependency. 3. **Optional third comparison or novelty model** Add Kimi K2 only if the main pipeline is already stable. It may strengthen the novelty of the evaluation, but it should not be the first dependency. 4. **Attacking model or attack generator** The project may use benchmark prompts directly or use an LLM-assisted attack generation workflow to adapt attack prompts. In either case, the report should distinguish between the model being attacked and any model used to help generate or refine attacks. ### System scope progression The recommended system scope is progressive rather than all at once: 1. Start with a simple chatbot setting. 2. Validate attacks, logging, and metrics in that simple setting. 3. Add a RAG-based scenario once the baseline pipeline is working. This progression is practical because basic RAG is easy to assemble, but high-quality RAG evaluation is much harder to do well. ## Accuracy and Validation of Analysis Accuracy in this project does not mean predictive model accuracy. Instead, it means whether the evaluation correctly identifies successful and unsuccessful attacks and whether the before/after comparison is fair and reproducible. Accuracy will be supported through the following steps: 1. **Consistent scoring criteria** Each attack category will have explicit success conditions, such as instruction override, prompt leakage, or unsafe output generation. 2. **Repeated test execution** The same benchmark cases will be run before and after mitigation under comparable settings. 3. **Category-based reporting** Results will be reported by attack class and attack surface, reducing the risk that one dominant category hides important weaknesses elsewhere. 4. **Cross-checking with literature definitions** Attack labels and evaluation logic will be based on published benchmark and taxonomy papers to improve validity. 5. **Manual review of sampled outputs** A subset of outputs will be reviewed manually to validate automated labels, especially for ambiguous partial-success cases. ## Mitigation Strategy Design The client requested at least one mitigation per attack surface. The project therefore plans to map defenses as follows: - **Direct prompt injection** Prompt hardening, role anchoring, and structured delimiters. - **Prompt leakage / instruction override** Output filtering and explicit refusal rules. - **Indirect injection through retrieved or external content** Instruction-data separation, sanitization, or guardrail logic before the model consumes untrusted content. - **Agent or tool-mediated misuse** Tool access restrictions, validation checks, or policy-based guardrails. This design follows the literature's defense-in-depth approach and avoids depending on one universal mitigation (Chen et al., 2025; Rebedea et al., 2023; Sahoo et al., 2024). ### Mitigation taxonomy to be considered The full set of mitigations relevant to the project includes: 1. **Prompt hardening** Stronger system prompts, explicit instruction hierarchy, refusal framing, and delimiter-based separation. 2. **Instruction-data separation** Structured query design or channel separation so that retrieved or user-provided content is treated as untrusted data rather than executable instruction. 3. **Input sanitization and filtering** Detection of suspicious patterns, filtering of malicious external content, and validation of retrieved context before the model consumes it. 4. **Output filtering and leakage prevention** Rules or classifiers that prevent system prompt disclosure, policy leakage, or unsafe outputs from being returned to the user. 5. **Guardrails and policy enforcement** Framework-level controls such as programmable rails, tool restrictions, response validation, or action approval gates. 6. **Tool-use constraints** Limiting what tools can do, constraining arguments passed to tools, and validating external actions before execution. 7. **Detection-based monitoring** Logging, anomaly detection, and step-level review to support mean time to detect and mean time to mitigate measurements. 8. **Defense-in-depth combinations** Combining prompt hardening with filtering, guardrails, or separation mechanisms rather than relying on one isolated control. For the proposal, the minimum requirement is to implement at least one mitigation. The recommended first mitigation is either prompt hardening for direct attacks or instruction-data separation / guardrails for indirect attacks, because these are both practical and well-supported in the literature (Chen et al., 2025; Rebedea et al., 2023; Sahoo et al., 2024). ## Resources Required The methodology requires the following resources: - Python-based experiment code and configuration files, - access to one or more open-source models, - benchmark datasets or environments such as AgentDojo and PINT, - structured logging for experiment outputs, - documentation through the repository and README, - report-writing support for analysis, results, and discussion. ## Expected Methodological Outputs By following this methodology, the project should produce: - A structured taxonomy of prompt injection attacks. - A reusable attack repository. - A benchmark-informed evaluation pipeline. - Baseline ASR results across selected models. - Post-mitigation comparison results. - Utility and operational metrics such as false positives and time-to-detect/time-to-mitigate where feasible. - A proof-of-concept implementation. - Report-ready figures and tables for the final proposal and report. ## Short Version for the Proposal Report If a shorter version is needed for the final report section, the methodology can be summarized as follows: The project will use an experimental evaluation methodology to test how vulnerable LLM chatbots are to prompt injection attacks and how effectively selected mitigations reduce those vulnerabilities. Attack cases will be collected from benchmark sources such as AgentDojo and PINT, from published literature, and from project-authored prompts aligned with a structured taxonomy. The framework will execute baseline attacks against selected open-source models, measure Attack Success Rate and related metrics, apply one mitigation per major attack surface, and then repeat the same experiments to compare performance before and after defense. Data analysis will combine quantitative measures such as ASR, prompt leakage rate, false positives, mean time to detect, and mean time to mitigate with qualitative review of model outputs. This method was chosen because it is consistent with current research, feasible within capstone constraints, and capable of producing a rigorous repository, proof of concept, and report. ## References Chen, S., Piet, J., Sitawarin, C., & Wagner, D. (2025). *StruQ: Defending against prompt injection with structured queries*. Debenedetti, E., Zhang, J., Balunovic, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). *AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents*. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). *Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection*. Liu, F. W., & Hu, C. (2024). *Exploring vulnerabilities and protections in large language models: A survey*. Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., et al. (2023). *Prompt injection attack against LLM-integrated applications*. Liu, Y., Jia, Y., Geng, R., Jia, J., & Gong, N. Z. (2024). *Formalizing and benchmarking prompt injection attacks and defenses*. Pandya, H., et al. (2025). *Adaptive white-box attacks against prompt injection defenses*. Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023). *NeMo Guardrails: A toolkit for controllable and safe LLM applications with programmable rails*. Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., & Chadha, A. (2024). *A systematic survey of prompt engineering in large language models: Techniques and applications*. Schulhoff, S., Pinto, J., Khan, A., Bouchard, L. F., Si, C., Anati, S., et al. (2023). *Ignore this title and HackAPrompt: Exposing systemic vulnerabilities of LLMs through a global prompt hacking competition*. Toyer, S., Watkins, O., Mendes, E. A., Svegliato, J., Bailey, L., Wang, T., et al. (2023). *Tensor Trust: Interpretable prompt injection attacks from an online game*.