# Literature Review: Improving Prompt Injection Resistance in LLM Chatbots

_Last updated: March 12, 2026_

## Overview

This literature review consolidates the stronger material from the revised literature review document and aligns it with the project scope defined in the repository, the professor's proposal rubric, and the latest client guidance. The project focuses on evaluating prompt injection vulnerabilities in LLM-based chatbots and agent-like systems, then measuring how well mitigation strategies reduce those vulnerabilities. The review therefore prioritizes papers that directly support five project needs:

1. defining prompt injection and its attack surface,
2. building a structured attack taxonomy,
3. selecting benchmark datasets and evaluation environments,
4. choosing practical mitigation strategies, and
5. designing defensible evaluation metrics and reporting.

The literature shows that prompt injection is no longer a narrow prompt engineering issue. It is a broader application security problem affecting chatbots, retrieval-augmented systems, and tool-using agents. Across the literature, the main research pattern is consistent: models remain vulnerable, attack success is measurable, and no single mitigation is sufficient across all attack surfaces.

## 1. Foundational Threat Framing

### 1.1 Early definition of prompt injection

Perez and Ribeiro (2022) established prompt injection as a distinct attack category by showing that a short adversarial instruction can override an application's original prompt and redirect model behaviour. Their work is foundational because it introduces the basic concepts of goal hijacking and prompt leakage that later papers formalize in more detail.

For this project, this paper is important because it provides the starting definition used to explain why user-controlled text should be treated as untrusted input in LLM systems.

### 1.2 Formal benchmark-oriented framing

Liu et al. (2024), _Formalizing and Benchmarking Prompt Injection Attacks and Defenses_, is the single most important paper for the proposal. It provides a formal framework for prompt injection and a benchmark methodology that directly mirrors the project's workflow: define attack categories, execute attacks against models, apply defenses, and compare pre-mitigation and post-mitigation performance using Attack Success Rate (ASR).

This paper is especially useful because it connects theory and implementation. It does not treat prompt injection as anecdotal misuse; it treats it as a measurable security problem with defined threat models, standardized attack categories, and comparable defense evaluations. That makes it the strongest methodological anchor for the project.

## 2. Attack Surface and Taxonomy

### 2.1 Indirect prompt injection in real-world systems

Greshake et al. (2023), _Not What You've Signed Up For_, demonstrated that prompt injection is not limited to direct user messages. They showed that malicious instructions embedded in external content such as web pages, emails, and documents can influence LLM-integrated applications when those applications retrieve or read that content. This work is central because it broadens the threat model from a simple chatbot to realistic systems with retrieval, browsing, and tool access.

For this project, the paper justifies why the methodology must include at least one indirect attack surface rather than evaluating only direct "ignore previous instructions" prompts.

### 2.2 Taxonomy for attack design

Liu et al. (2023), _Prompt Injection Attack Against LLM-Integrated Applications_, introduced a practical attack taxonomy and the HouYi framework. Their work categorizes attacks by objective and strategy, including goal hijacking, prompt leakage, context ignoring, semantic separation, escape-character style payloads, and iterative refinement. They also showed that real applications are widely vulnerable.

This paper is directly transferable to the project because it provides terminology for structuring the attack repository and the test suite. It supports the client's request to develop a taxonomy of attacks and to make the attack set more systematic than a list of ad hoc prompts.

### 2.3 Human-generated attack diversity

Schulhoff et al. (2023), _HackAPrompt_, and Toyer et al. (2023), _Tensor Trust_, add a different kind of value. Instead of focusing only on researcher-designed attacks, they show how diverse and creative human-generated attacks can be when framed as competitions or games. These sources are useful for broadening prompt diversity in the attack library and reducing the risk that evaluation is biased toward a small set of overly familiar attack templates.

For this project, these works support the idea that benchmark coverage should include both canonical academic attacks and realistic, human-generated attack styles.

## 3. Benchmarking Environments and Datasets

### 3.1 AgentDojo as a benchmark environment

Debenedetti et al. (2024), _AgentDojo_, is one of the most relevant recent papers because it moves from isolated prompt tests to dynamic agent evaluation. AgentDojo includes realistic user tasks, tool interactions, environment state, and automated scoring for both normal task completion and attack success. Its design is highly relevant to the client's feedback about using benchmark datasets such as AgentDojo and structuring the project around a pipeline rather than a one-off proof of concept.

AgentDojo is also valuable because it explicitly measures the security-utility trade-off. A system can reduce attack success but also degrade normal task performance. This is a critical idea for the project because the methodology should not report lower ASR alone; it should also show whether the mitigation damages useful behaviour.

### 3.2 PINT and benchmark-driven evaluation

The client specifically requested benchmark-based evaluation using datasets such as AgentDojo and PINT. Within the project, these benchmarks can be used as structured sources of attack cases and scenarios rather than relying only on self-authored prompts. This improves credibility, reproducibility, and comparability with published work.

Benchmark-driven evaluation is especially important for the proposal because it strengthens methodological justification under the rubric. It demonstrates that the project is not merely building a demo; it is aligning its experiments with established evaluation resources.

### 3.3 Formal benchmark methodology

Liu et al. (2024) remains central here as well, because the paper provides the benchmark logic for comparing attacks and defenses across categories. Together, Liu et al. (2024) and AgentDojo provide the core structure for the project's attack suite, experiment orchestration, and reporting.

## 4. Why Current Models Remain Vulnerable

### 4.1 Failure of safety alignment under adversarial prompting

Wei et al. (2023), _Jailbroken: How Does LLM Safety Training Fail?_, provides the theoretical explanation for why aligned models remain vulnerable. The paper argues that safety training often fails because of competing objectives and weak generalization to adversarial prompt distributions. This is important for the proposal because it explains why prompt injection remains relevant even for strong modern models.

### 4.2 Adaptive attacks on safety-aligned models

Andriushchenko, Croce, and Flammarion (2024) showed that simple adaptive attacks can achieve very high success rates against safety-aligned commercial models. Their findings are important because they justify the inclusion of adaptive or iterative attacks in the methodology, especially where an attacker modifies prompts after observing refusals or filters.

This supports the client's feedback that the project should cover different injection attacks, not only single-turn direct attacks.

### 4.3 Optimization-based attacks

Zou et al. (2023), _Universal and Transferable Adversarial Attacks on Aligned Language Models_, introduced GCG-style adversarial suffixes. Although the project is not primarily about harmful-content jailbreaks, this work is still important because it shows that sophisticated optimized attacks can transfer across models and can outperform naive prompt manipulations.

Including this literature strengthens the proposal by showing awareness of the broader attack spectrum from simple prompt override to optimized adversarial attacks.

### 4.4 Stealthy attacks against naive filters

Liu et al. (2023), _AutoDAN_, showed that fluent and stealthy attack prompts can bypass simplistic filtering approaches. This matters for the methodology because it cautions against selecting only weak mitigations such as naive keyword blocking. It also strengthens the rationale for evaluating multiple mitigation strategies across different attack surfaces.

## 5. Mitigation Strategies in Current Literature

### 5.1 Structured separation of instructions and data

Chen et al. (2025), _StruQ_, is the strongest mitigation paper for this project. It proposes a structured query design that clearly separates trusted instructions from untrusted data. This approach directly addresses one of the root causes of prompt injection: the model receives all text in a single channel and may not reliably distinguish between instructions and data.

For this project, StruQ is useful as both a conceptual mitigation and an architectural principle. Even if the exact implementation is simplified, the project can still adopt the same defensive idea: isolate system instructions from user and retrieved content wherever possible.

### 5.2 Guardrail-based defenses

Rebedea et al. (2023), _NeMo Guardrails_, provides a practical implementation-oriented defense approach. Guardrails are relevant because they can be integrated into systems without retraining a base model, which fits the scope of the capstone. This paper is useful for the project's mitigation layer because it shows how programmable input/output constraints can be applied to real applications.

### 5.3 Prompt hardening and defensive prompting

Sahoo et al. (2024), in their prompt engineering survey, show that prompt hardening can improve resilience against basic attacks but is insufficient against sophisticated attacks when used alone. This is consistent with broader literature: prompt hardening should be treated as one layer in a defense-in-depth strategy rather than a complete solution.

### 5.4 Filtering and authentication approaches

Recent work such as FATH and related test-time defenses shows that filtering, tagging, and authentication mechanisms can reduce some forms of indirect injection without requiring full retraining. These papers are useful for discussing practical mitigation options in the proposal, especially when the project needs to compare one mitigation per attack surface as suggested by the client.

### 5.5 Limits of current defenses

Pandya et al. (2025) showed that some recent fine-tuned defenses can still be broken by adaptive white-box attacks. This is an important caution for the project. It means the evaluation should not assume any defense is final or universally robust. Instead, the methodology should frame mitigation results as relative improvements under tested conditions.

## 6. Standards and Practical Security Context

The OWASP GenAI Top 10, especially LLM01:2025 Prompt Injection, provides a practical security framing that complements the academic papers. OWASP is useful for translating the research into engineering language: attack surface, trusted versus untrusted data, impact, and layered mitigations.

This is relevant to the proposal because it helps justify why the project should report not only whether attacks work, but also what kind of impact they enable, such as prompt leakage, policy override, unsafe output, or tool misuse.

## 7. Synthesis for the Proposed Project

Across the literature, several conclusions are consistent and directly shape the project:

### 7.1 No single mitigation is enough

The literature repeatedly shows that single-layer defenses reduce ASR in some scenarios but fail in others. This justifies a methodology that compares multiple mitigation strategies rather than claiming one universal solution.

### 7.2 Evaluation must cover multiple attack surfaces

Direct prompt injection alone is too narrow. The project should test at least:

- direct user-message attacks,
- prompt leakage attempts,
- indirect injection through external or retrieved content, and
- adaptive or iterative attack refinement.

This aligns both with the literature and with the client's emphasis on taxonomy and attack-surface-specific mitigations.

### 7.3 Benchmark datasets improve rigor

Benchmarks such as AgentDojo and PINT improve reproducibility and make results easier to defend in an academic report. They also reduce the risk that the project's conclusions are based only on self-selected attack prompts.

### 7.4 Security must be balanced with utility

The best recent evaluation work measures both security and normal task performance. This means the project should not rely on ASR alone. It should also track utility-related outcomes and operational metrics such as mean time to detect and mean time to mitigate where feasible.

## 8. Implications for the Methodology Section

The literature directly supports the following project methodology decisions:

- Use a modular experiment pipeline with attack generation, execution, mitigation, evaluation, and reporting.
- Build a structured attack taxonomy derived from foundational and benchmark literature.
- Use benchmark datasets and environments, especially AgentDojo and PINT, where feasible.
- Evaluate open-source models across more than one model family where feasible, such as Llama, Qwen, or Kimi K2 style variants.
- Include at least one mitigation per major attack surface.
- Measure ASR before and after mitigation, and complement it with additional metrics such as benign task utility, false positives, mean time to detect, and mean time to mitigate.
- Present findings in a repository, README-based documentation, and a written report with reproducible experiment settings.

## 9. Ranked Literature Most Relevant to the Proposal

### Highest priority

1. Liu et al. (2024), _Formalizing and Benchmarking Prompt Injection Attacks and Defenses_
2. Greshake et al. (2023), _Not What You've Signed Up For_
3. Liu et al. (2023), _Prompt Injection Attack Against LLM-Integrated Applications_
4. Debenedetti et al. (2024), _AgentDojo_
5. Chen et al. (2025), _StruQ_

### Essential support

6. Liu and Hu (2024), _Exploring Vulnerabilities and Protections in Large Language Models: A Survey_
7. Schulhoff et al. (2023), _HackAPrompt_
8. Toyer et al. (2023), _Tensor Trust_
9. Perez and Ribeiro (2022), _Ignore Previous Prompt_
10. Zou et al. (2023), _Universal and Transferable Adversarial Attacks on Aligned Language Models_

### Supporting literature

11. Wei et al. (2023), _Jailbroken_
12. Andriushchenko et al. (2024), _Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks_
13. Liu et al. (2023), _AutoDAN_
14. Rebedea et al. (2023), _NeMo Guardrails_
15. Sahoo et al. (2024), _A Systematic Survey of Prompt Engineering in Large Language Models_

## 10. Conclusion

The literature strongly supports the project's proposed direction. Prompt injection remains an open security problem across chatbots and agentic LLM systems. Existing work provides the conceptual tools, benchmark structures, and mitigation candidates needed to build a rigorous capstone project. The strongest methodological pattern is clear: define a realistic taxonomy of attacks, evaluate them systematically across models, apply layered mitigations, and compare security gains against utility trade-offs using reproducible metrics.

This directly supports the capstone's intended contribution: a practical evaluation framework, a structured attack repository, comparative mitigation testing, and a report that explains both what improved and where residual risk remains.

## 11. Selected References

- Andriushchenko, M., Croce, F., & Flammarion, N. (2024). _Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks_.
- Chen, S., Piet, J., Sitawarin, C., & Wagner, D. (2025). _StruQ: Defending Against Prompt Injection with Structured Queries_.
- Debenedetti, E., Zhang, J., Balunovic, M., Beurer-Kellner, L., Fischer, M., & Tramèr, F. (2024). _AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents_.
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). _Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection_.
- Liu, F. W., & Hu, C. (2024). _Exploring Vulnerabilities and Protections in Large Language Models: A Survey_. Local copy: `docs/2406.00240v1.pdf`.
- Liu, X., Xu, N., Chen, M., & Xiao, C. (2023). _AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models_.
- Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., et al. (2023). _Prompt Injection Attack Against LLM-Integrated Applications_.
- Liu, Y., Jia, Y., Geng, R., Jia, J., & Gong, N. Z. (2024). _Formalizing and Benchmarking Prompt Injection Attacks and Defenses_. Local copy: `docs/2310.12815v5.pdf`.
- Pandya, H., et al. (2025). _Adaptive white-box attacks against prompt injection defenses_.
- Perez, F., & Ribeiro, I. (2022). _Ignore Previous Prompt: Attack Techniques for Language Models_.
- Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023). _NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails_.
- Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., & Chadha, A. (2024). _A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications_.
- Schulhoff, S., Pinto, J., Khan, A., Bouchard, L. F., Si, C., Anati, S., et al. (2023). _Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition_.
- Toyer, S., Watkins, O., Mendes, E. A., Svegliato, J., Bailey, L., Wang, T., et al. (2023). _Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game_.
- Wei, A., Haghtalab, N., & Steinhardt, J. (2023). _Jailbroken: How Does LLM Safety Training Fail?_ 
- Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). _Universal and Transferable Adversarial Attacks on Aligned Language Models_.