docs/PROPOSED_METHODOLOGY.txt

Proposed Methodology

Prepared for the group proposal report due in Week 5

Methodologies

This project adopts an experimental cybersecurity evaluation methodology
to assess how vulnerable LLM-based chatbots and lightweight agent
workflows are to prompt injection attacks, and how effectively selected
mitigations reduce that risk. The project will not train new language
models. Instead, it will build a modular evaluation pipeline that runs
prompt injection attacks against selected open-source models, records
baseline vulnerability, applies mitigations, and compares results before
and after defense. The methodology is designed to produce a repository,
a report, and a working proof of concept that are reproducible and
aligned with current prompt injection research practice (Debenedetti et
al., 2024; Greshake et al., 2023; Liu et al., 2024).

Methods

Overall method

The project will use a controlled experimental pipeline with repeated
attack-and-evaluate cycles. The pipeline contains five stages:

1.  Define attack taxonomy and benchmark tasks.
2.  Collect or adapt prompt injection test cases.
3.  Run baseline attacks against selected models.
4.  Apply mitigation strategies matched to each attack surface.
5.  Re-run the same attacks to measure improvement.

This method was selected because it directly supports the project’s core
objective: quantifying vulnerability and mitigation effectiveness rather
than only describing attacks conceptually, which is consistent with
benchmark-oriented prompt injection research (Liu et al., 2024).

Experimental pipeline

The technical workflow will follow this sequence:

1.  Target model selection
    The system will test open-source models only. The recommended
    starting models are Llama-family and Qwen-family models because they
    are the easiest to begin with and are widely used in current prompt
    injection experiments. Kimi K2 can be explored later as an optional
    extension if implementation effort remains manageable.

2.  Attack library construction
    A repository of prompt injection attacks will be built and organized
    by taxonomy. The attack set will include direct attacks, indirect
    attacks, prompt leakage attempts, role or instruction override
    attacks, and adaptive multi-turn attacks (Greshake et al., 2023; Liu
    et al., 2023; Liu et al., 2024).

3.  Benchmark integration
    The attack library will be supplemented with benchmark tasks and
    datasets, especially AgentDojo and PINT where feasible, to improve
    methodological rigor and comparability (Debenedetti et al., 2024).

4.  Baseline execution
    Each attack will be executed against the selected model or models
    without mitigation enabled. Outputs and metadata will be logged for
    later analysis.

5.  Mitigation layer evaluation
    At least one mitigation will be mapped to each major attack surface.
    Likely examples include prompt hardening for direct attacks,
    instruction/data separation for indirect attacks, and guardrail or
    filtering logic for tool or output abuse (Chen et al., 2025; Rebedea
    et al., 2023; Sahoo et al., 2024).

6.  Post-mitigation re-evaluation
    The same experiments will be repeated after the mitigation layer is
    enabled so that improvements can be measured under comparable
    conditions.

7.  Reporting and proof of concept
    Results will be summarized in the report and in repository
    documentation. The final deliverable will also include a
    proof-of-concept pipeline showing the taxonomy, experiment flow,
    baseline vulnerability, and measured mitigation effects.

Attack taxonomy to be used

To satisfy the project’s minimum requirement and to keep the evaluation
systematic, the attack repository will be organized around a clear
taxonomy rather than a loose set of prompts. The most relevant taxonomy
groups for this project are:

1.  Direct instruction override attacks
    Simple user-supplied prompts such as “ignore previous instructions”
    that attempt to replace or dominate the system prompt.

2.  Prompt leakage attacks
    Prompts designed to reveal hidden instructions, system prompts,
    safety policies, or internal workflow details.

3.  Indirect prompt injection attacks
    Malicious instructions embedded in retrieved documents, web pages,
    emails, or tool outputs that the model later consumes as context.

4.  Goal hijacking attacks
    Prompts that redirect the model from the user’s intended task toward
    the attacker’s objective.

5.  Role-play or context manipulation attacks
    Payloads that use personas, fictional settings, fake completions, or
    instruction framing to bypass normal safety behaviour.

6.  Adaptive or multi-turn attacks
    Iterative attacks in which the attacker changes wording based on
    prior refusals or filters.

7.  Tool-mediated or agent-specific attacks
    Attacks targeting systems that can browse, retrieve files, call
    tools, or perform actions on behalf of the user.

8.  Semantic separation and obfuscation attacks
    Payloads that hide intent through separators, formatting tricks,
    language changes, or context mixing.

These taxonomy groups were chosen because they cover the core categories
repeatedly identified in the literature and map well to the expected
attack surfaces (Greshake et al., 2023; Liu et al., 2023; Liu et al.,
2024). The implementation should include at least one proof-of-concept
attack from every major taxonomy category, even if early versions remain
surface-level. The preferred design is to cover all categories at least
once, then deepen the strongest attack paths later.

Benchmark datasets and scenario sources

The attack cases will be grounded in benchmark and literature sources
rather than only ad hoc prompt writing. The preferred benchmark sources
are:

-   AgentDojo for dynamic agent-style attack scenarios, security-utility
    trade-off measurement, and realistic tool-mediated tasks
    (Debenedetti et al., 2024).
-   PINT for benchmark prompt injection cases that can supplement the
    direct and indirect attack repository.
-   Literature-derived prompts from Liu et al. (2024), Greshake et
    al. (2023), HouYi-style categories, HackAPrompt, and Tensor Trust
    (Liu et al., 2024; Greshake et al., 2023; Liu et al., 2023;
    Schulhoff et al., 2023; Toyer et al., 2023).

If time or infrastructure limits prevent full use of both benchmarks,
the minimum acceptable design is to use one benchmark source, preferably
AgentDojo or PINT, and then supplement it with project-authored prompts
tied to the same taxonomy categories.

Minimum viable experiment design

Based on the client notes, the project must at minimum demonstrate:

1.  One benchmark-backed attack taxonomy.
2.  One mitigation strategy.
3.  One attacking or probing model setup.
4.  One target model under evaluation.
5.  One reproducible benchmark or dataset source such as AgentDojo or
    PINT.

The recommended minimum configuration is:

-   Target model under test: One open-source model such as a Llama or
    Qwen variant.
-   Attack generation / probing setup: One attack library or
    attacker-assisted prompting workflow using benchmark cases from
    AgentDojo or PINT.
-   Mitigation: One concrete defense such as prompt hardening,
    guardrails, or instruction-data separation.
-   Evaluation metric: ASR before and after mitigation, with at least
    one supporting metric such as prompt leakage rate or benign utility.

This minimum design is deliberately narrow enough to be feasible while
still satisfying the project’s core requirement.

Recommended implementation order

The recommended development order is:

1.  Build a simple chatbot evaluation pipeline first.
2.  Implement one proof-of-concept attack for each major taxonomy
    category.
3.  Test those attacks against a primary open-source model such as Qwen
    or Llama.
4.  Add at least one mitigation per attack surface.
5.  Measure ASR and supporting metrics before and after mitigation.
6.  Extend the same pipeline to a RAG scenario if time and stability
    permit.

This staged approach keeps the first milestone achievable while still
leaving room for a stronger final system.

Why this method was chosen

This approach was chosen for four reasons:

-   It is directly aligned with the literature on prompt injection
    benchmarking (Debenedetti et al., 2024; Liu et al., 2024).
-   It satisfies the client request for a pipeline, taxonomy, benchmark
    use, and measurable evaluation.
-   It fits the capstone scope because it emphasizes engineering and
    evaluation rather than model training.
-   It produces concrete outputs for the rubric: methodology, resources,
    milestones, expected outcomes, and a structured final report.

Data Collection

What data will be collected

The project will collect four categories of data:

1.  Attack prompts and attack scenarios
    These include manually designed prompts based on literature
    taxonomies and benchmark-derived attack cases.

2.  Model response data
    For each attack, the project will record the model’s response,
    whether the attack succeeded, whether restricted information was
    exposed, and whether the model followed the malicious instruction.

3.  Mitigation outcome data
    The project will record how the same attack behaves once a
    mitigation has been applied.

4.  Operational experiment data
    Metadata such as model name, attack category, attack surface,
    defense used, execution time, mean time to detect, and mean time to
    mitigate will be recorded where implementation allows.

Sources of data

Data will be collected from three main sources:

-   Benchmark datasets and environments such as AgentDojo and PINT
    (Debenedetti et al., 2024).
-   Literature-derived attack cases based on papers such as Liu et
    al. (2024), Greshake et al. (2023), and the HouYi taxonomy (Greshake
    et al., 2023; Liu et al., 2023; Liu et al., 2024).
-   Project-authored test cases created to fill gaps for specific attack
    surfaces or mitigation experiments.

Data items recorded per run

For each experiment run, the project will record:

-   Benchmark or dataset source.
-   Attack taxonomy label.
-   Prompt or scenario identifier.
-   Model under test.
-   Attacker setup or prompt source.
-   Mitigation used.
-   Raw model response.
-   Attack success label.
-   Prompt leakage label if applicable.
-   Utility outcome for benign tasks if applicable.
-   Timing and logging metadata.

Data collection procedure

The collection procedure will be:

1.  Review the literature and extract canonical attack categories.
2.  Adapt benchmark tasks and attack prompts into a unified repository
    format.
3.  Create any missing test prompts needed for project-specific
    scenarios.
4.  Execute prompts through a consistent experiment runner.
5.  Store outputs and labels in structured logs.
6.  Re-run the same test set after mitigation so the before/after
    comparison remains fair.

Why this data collection strategy was chosen

This data collection method balances rigor and feasibility. Benchmark
data improves credibility, while project-authored prompts allow the team
to tailor the evaluation to the client’s requested attack surfaces and
proof-of-concept design (Debenedetti et al., 2024; Liu et al., 2024).

Data Analysis

Primary analysis technique

The primary analysis technique will be quantitative comparison of model
behaviour before and after mitigation. The main metric will be Attack
Success Rate (ASR), because ASR is the most common and accepted metric
in the literature for prompt injection evaluation (Debenedetti et al.,
2024; Liu et al., 2024).

ASR will be calculated as:

ASR = successful attacks / total attacks in a category

This will be measured:

-   Overall.
-   Per model.
-   Per attack type.
-   Per attack surface.
-   Per mitigation strategy.

Additional analysis metrics

Because ASR alone is not enough, the project will also analyse:

-   Prompt leakage rate: frequency of successful extraction of hidden
    instructions or system prompts.
-   Benign utility or task completion rate: whether the model can still
    perform normal tasks after mitigation.
-   False positive rate: how often a mitigation blocks benign inputs.
-   False negative rate: how often a mitigation fails to block a
    malicious input.
-   Mean time to detect (MTTD): average time or number of steps required
    to identify an injection attempt.
-   Mean time to mitigate (MTTM): average time or number of steps
    required to apply a mitigation or neutralize the attack.
-   Severity or risk category: a structured classification such as low,
    medium, high, or critical, based on factors such as instruction
    override, harmful output generation, sensitive information leakage,
    or unauthorized tool behavior.

These supporting metrics are useful because recent work evaluates not
only attack success, but also operational response and the
security-utility trade-off of defensive interventions (Debenedetti et
al., 2024; Liu et al., 2024).

LLM-as-a-judge validation

An additional optional validation method is to use a stronger LLM as a
secondary evaluator. In this setup, the project first computes its own
attack labels and metrics, then uses an external LLM judge to review
selected outputs and provide a second opinion on whether the attack
succeeded or whether the mitigation was effective. This should not
replace the primary evaluation logic, but it can be useful for checking
whether manual or rule-based scoring is broadly consistent with a
model-based evaluator.

Qualitative analysis

Quantitative metrics will be supplemented by qualitative review of
selected outputs. This is necessary because some attacks may partially
succeed in ways that a simple binary score does not fully capture.
Qualitative analysis will focus on:

-   How the model interprets malicious instructions.
-   Whether the mitigation changes model reasoning or refusal patterns.
-   Which attack types remain effective after defense.
-   What residual risks remain for deployment.

Comparative analysis

Where resources permit, the project will compare:

-   Llama versus Qwen style open-source models.
-   Direct versus indirect attack surfaces.
-   Single-turn versus adaptive attacks.
-   Different mitigation strategies for the same attack category.

This comparative approach is important because the literature shows that
defenses often perform unevenly across models and attack types
(Debenedetti et al., 2024; Liu et al., 2024; Pandya et al., 2025).

Models Chosen

The project will prioritize a small but representative set of models
rather than a large benchmark sweep. The likely model groups are:

-   Open-source models such as Llama and Qwen variants, because they are
    accessible, reproducible, and suitable for proof-of-concept
    experimentation.
-   Optional extension models such as Kimi K2, if setup complexity and
    time allow additional comparison.

These models were chosen to support a useful contrast:

-   Open-source models allow reproducible testing without depending on
    API access.
-   Comparing more than one open-source family makes the evaluation more
    meaningful than testing a single model family.

If compute or setup becomes a constraint, the methodology can still
remain valid with one primary open-source model and one benchmark-backed
mitigation experiment.

Recommended model setup for this project

To stay aligned with the client notes and likely capstone constraints,
the recommended sequence is:

1.  Primary target model
    Start with one open-source model, preferably a Llama-family or
    Qwen-family model, because it is reproducible, practical for a proof
    of concept, and feasible within project constraints.

2.  Optional comparison model
    Add a second open-source model such as Qwen or Llama if feasible.
    This strengthens the analysis but should be treated as an extension
    rather than a dependency.

3.  Optional third comparison or novelty model
    Add Kimi K2 only if the main pipeline is already stable. It may
    strengthen the novelty of the evaluation, but it should not be the
    first dependency.

4.  Attacking model or attack generator
    The project may use benchmark prompts directly or use an
    LLM-assisted attack generation workflow to adapt attack prompts. In
    either case, the report should distinguish between the model being
    attacked and any model used to help generate or refine attacks.

System scope progression

The recommended system scope is progressive rather than all at once:

1.  Start with a simple chatbot setting.
2.  Validate attacks, logging, and metrics in that simple setting.
3.  Add a RAG-based scenario once the baseline pipeline is working.

This progression is practical because basic RAG is easy to assemble, but
high-quality RAG evaluation is much harder to do well.

Accuracy and Validation of Analysis

Accuracy in this project does not mean predictive model accuracy.
Instead, it means whether the evaluation correctly identifies successful
and unsuccessful attacks and whether the before/after comparison is fair
and reproducible.

Accuracy will be supported through the following steps:

1.  Consistent scoring criteria
    Each attack category will have explicit success conditions, such as
    instruction override, prompt leakage, or unsafe output generation.

2.  Repeated test execution
    The same benchmark cases will be run before and after mitigation
    under comparable settings.

3.  Category-based reporting
    Results will be reported by attack class and attack surface,
    reducing the risk that one dominant category hides important
    weaknesses elsewhere.

4.  Cross-checking with literature definitions
    Attack labels and evaluation logic will be based on published
    benchmark and taxonomy papers to improve validity.

5.  Manual review of sampled outputs
    A subset of outputs will be reviewed manually to validate automated
    labels, especially for ambiguous partial-success cases.

Mitigation Strategy Design

The client requested at least one mitigation per attack surface. The
project therefore plans to map defenses as follows:

-   Direct prompt injection
    Prompt hardening, role anchoring, and structured delimiters.

-   Prompt leakage / instruction override
    Output filtering and explicit refusal rules.

-   Indirect injection through retrieved or external content
    Instruction-data separation, sanitization, or guardrail logic before
    the model consumes untrusted content.

-   Agent or tool-mediated misuse
    Tool access restrictions, validation checks, or policy-based
    guardrails.

This design follows the literature’s defense-in-depth approach and
avoids depending on one universal mitigation (Chen et al., 2025; Rebedea
et al., 2023; Sahoo et al., 2024).

Mitigation taxonomy to be considered

The full set of mitigations relevant to the project includes:

1.  Prompt hardening
    Stronger system prompts, explicit instruction hierarchy, refusal
    framing, and delimiter-based separation.

2.  Instruction-data separation
    Structured query design or channel separation so that retrieved or
    user-provided content is treated as untrusted data rather than
    executable instruction.

3.  Input sanitization and filtering
    Detection of suspicious patterns, filtering of malicious external
    content, and validation of retrieved context before the model
    consumes it.

4.  Output filtering and leakage prevention
    Rules or classifiers that prevent system prompt disclosure, policy
    leakage, or unsafe outputs from being returned to the user.

5.  Guardrails and policy enforcement
    Framework-level controls such as programmable rails, tool
    restrictions, response validation, or action approval gates.

6.  Tool-use constraints
    Limiting what tools can do, constraining arguments passed to tools,
    and validating external actions before execution.

7.  Detection-based monitoring
    Logging, anomaly detection, and step-level review to support mean
    time to detect and mean time to mitigate measurements.

8.  Defense-in-depth combinations
    Combining prompt hardening with filtering, guardrails, or separation
    mechanisms rather than relying on one isolated control.

For the proposal, the minimum requirement is to implement at least one
mitigation. The recommended first mitigation is either prompt hardening
for direct attacks or instruction-data separation / guardrails for
indirect attacks, because these are both practical and well-supported in
the literature (Chen et al., 2025; Rebedea et al., 2023; Sahoo et al.,
2024).

Resources Required

The methodology requires the following resources:

-   Python-based experiment code and configuration files,
-   access to one or more open-source models,
-   benchmark datasets or environments such as AgentDojo and PINT,
-   structured logging for experiment outputs,
-   documentation through the repository and README,
-   report-writing support for analysis, results, and discussion.

Expected Methodological Outputs

By following this methodology, the project should produce:

-   A structured taxonomy of prompt injection attacks.
-   A reusable attack repository.
-   A benchmark-informed evaluation pipeline.
-   Baseline ASR results across selected models.
-   Post-mitigation comparison results.
-   Utility and operational metrics such as false positives and
    time-to-detect/time-to-mitigate where feasible.
-   A proof-of-concept implementation.
-   Report-ready figures and tables for the final proposal and report.

Short Version for the Proposal Report

If a shorter version is needed for the final report section, the
methodology can be summarized as follows:

The project will use an experimental evaluation methodology to test how
vulnerable LLM chatbots are to prompt injection attacks and how
effectively selected mitigations reduce those vulnerabilities. Attack
cases will be collected from benchmark sources such as AgentDojo and
PINT, from published literature, and from project-authored prompts
aligned with a structured taxonomy. The framework will execute baseline
attacks against selected open-source models, measure Attack Success Rate
and related metrics, apply one mitigation per major attack surface, and
then repeat the same experiments to compare performance before and after
defense. Data analysis will combine quantitative measures such as ASR,
prompt leakage rate, false positives, mean time to detect, and mean time
to mitigate with qualitative review of model outputs. This method was
chosen because it is consistent with current research, feasible within
capstone constraints, and capable of producing a rigorous repository,
proof of concept, and report.

References

Chen, S., Piet, J., Sitawarin, C., & Wagner, D. (2025). StruQ: Defending
against prompt injection with structured queries.

Debenedetti, E., Zhang, J., Balunovic, M., Beurer-Kellner, L., Fischer,
M., & Tramèr, F. (2024). AgentDojo: A dynamic environment to evaluate
prompt injection attacks and defenses for LLM agents.

Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz,
M. (2023). Not what you’ve signed up for: Compromising real-world
LLM-integrated applications with indirect prompt injection.

Liu, F. W., & Hu, C. (2024). Exploring vulnerabilities and protections
in large language models: A survey.

Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., et al. (2023).
Prompt injection attack against LLM-integrated applications.

Liu, Y., Jia, Y., Geng, R., Jia, J., & Gong, N. Z. (2024). Formalizing
and benchmarking prompt injection attacks and defenses.

Pandya, H., et al. (2025). Adaptive white-box attacks against prompt
injection defenses.

Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J.
(2023). NeMo Guardrails: A toolkit for controllable and safe LLM
applications with programmable rails.

Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., & Chadha, A.
(2024). A systematic survey of prompt engineering in large language
models: Techniques and applications.

Schulhoff, S., Pinto, J., Khan, A., Bouchard, L. F., Si, C., Anati, S.,
et al. (2023). Ignore this title and HackAPrompt: Exposing systemic
vulnerabilities of LLMs through a global prompt hacking competition.

Toyer, S., Watkins, O., Mendes, E. A., Svegliato, J., Bailey, L., Wang,
T., et al. (2023). Tensor Trust: Interpretable prompt injection attacks
from an online game.