# Client Transcript Summary _Based on the transcript you provided on March 12, 2026_ ## Purpose This document summarizes the key guidance from the client conversation so the team can check whether the current project direction is aligned with expectations. ## Main Takeaways ### 1. Start with a pipeline and proof of concept The client does not require a highly novel research contribution at the start. The stronger recommendation is to build a practical pipeline first: - Implement prompt injection attacks. - Test them against selected LLMs. - Measure results using published metrics. - Add mitigations after the baseline proof of concept works. The expected approach is: start simple, get a working proof of concept, then extend it. ### 2. Use open-source models first The client explicitly recommended starting with open-source models because API credits are unavailable or inconvenient. The recommended starting point is: - **Qwen** - **Llama** The client also mentioned **Kimi K2** as an interesting optional model because fewer people appear to have tested it, but it was presented as an extension rather than the safest first step. ## 3. Another LLM can help with the attack or evaluation workflow The transcript suggests that a second LLM can be used in two optional ways: - To help generate or refine injection prompts. - To act as an evaluator or "LLM as a judge" for checking how strong the attacks or mitigations are. This does **not** mean the project must depend entirely on automated LLM judging. The client's position was that this should be supplementary, not the only evaluation method. ## 4. Cover the taxonomy broadly before going deep The client strongly emphasized taxonomy coverage. The expectation is: - Cover all major attack categories from the taxonomy paper. - At least implement one proof-of-concept example for each major attack surface. - Do not leave one attack category out. - More depth is good, but full depth across all categories is not required initially. This means breadth first, then depth later. ## 5. Start with a simple chatbot, then extend to RAG if possible The recommended build order is: 1. Start with a simple chatbot evaluation setting. 2. Get attacks and metrics working there. 3. Add RAG later if time permits. The client specifically noted that basic RAG is easy to build, but good RAG is much harder to do well. That means RAG should be treated as an extension after the base pipeline is stable. ## 6. Use benchmark datasets for attack cases The client recommended using benchmark datasets and specifically named: - **AgentDojo** - **PINT** The intent is to use these as benchmark sources for injection cases and as inspiration for additional project-authored attacks. ## 7. Metrics should include more than ASR The client confirmed that **Attack Success Rate (ASR)** is the main starting metric, but also recommended: - **Mean time to detect (MTTD)** - **Mean time to mitigate / respond (MTTM)** The transcript also supports using an LLM judge as a secondary validation step if useful. ## 8. Mitigations should be mapped to attack surfaces The client emphasized that each attack surface should have at least one mitigation. The expectation is not necessarily a deep mitigation study for every category at the start, but each attack type should at least be paired with an initial defense approach. ## 9. Deliverables should stay focused The client's preferred deliverables are: - A clean, modular repository - A report with standard academic structure - A strong README explaining how to run the code and understand the modules The client explicitly said there is no need to overbuild extra deliverables if the repository and report already cover the work properly. ## 10. Practical workflow recommended by the client The client's suggested order of work was: 1. Read the core papers. 2. Find additional related papers. 3. Build the literature review. 4. Develop the methodology from the attack taxonomy and mitigation ideas. 5. Implement a modular codebase in parallel. 6. Produce results and compare models and mitigations. 7. Document limitations and future work clearly. ## What this means for our current direction The current methodology is broadly aligned with the transcript, but a few points are especially important: - The project should prioritize **Qwen** and **Llama** first. - **Kimi K2** should stay optional. - The team should aim to cover **all major taxonomy categories at least once**. - The first milestone should be a **simple chatbot proof of concept**, not full RAG immediately. - **ASR, MTTD, and MTTM** should be treated as key metrics. - **AgentDojo** and **PINT** are appropriate benchmark sources. - A second LLM may help generate attacks or validate results, but it should not replace the main evaluation logic. ## My assessment The current project direction is correct, but it needs to stay disciplined. The main risk is trying to do too much depth too early. The transcript points toward a staged approach: 1. Build the modular baseline. 2. Cover every attack category at proof-of-concept level. 3. Add one mitigation per surface. 4. Measure ASR and timing metrics. 5. Extend to stronger comparisons or RAG only after the baseline works. That is a defensible and realistic capstone strategy.