docs/MEETING_QUESTIONS_TODAY.md

Client Meeting Questions (Today)

1) Research Contribution Expectations

• What level of research contribution is expected?
  • Exploratory implementation
  • Strong evaluation of existing methods
  • Novel work with potential publishability
• Do you expect innovation, or is strong application/evaluation of existing methods sufficient?

2) Project Scope Confirmation

Can we confirm our understanding of the project?

Our understanding

• The project is a security evaluation pipeline for LLM chatbots.

It is NOT

• Training LLMs
• Building ChatGPT
• Building a new AI model
• Doing heavy ML training

It IS

• Designing attacks
• Measuring vulnerability
• Adding protection
• Measuring improvement

Also, can we confirm that the focus is evaluating prompt injection vulnerabilities and mitigation strategies rather than developing/training new models?

3) Model Prioritization

• Which LLMs should we prioritize testing?
  • API-based models
  • Open-source models
  • Both

4) System Complexity in Scope

• Should evaluation focus on simple chatbot systems only, or also include more complex architectures such as:
  • RAG systems
  • Enterprise-like deployments

5) Attacker Capabilities and Scenarios

Which attacker capabilities should be in scope?

1. Standard user-level interaction with the chatbot
2. Attempts to extract hidden system prompts or internal instructions
3. Indirect prompt injection through retrieved content (e.g., RAG)
4. Adaptive attackers who iteratively modify prompts after observing system defenses

6) Evaluation Methodology

• What evaluation methodology should we use?
  • ASR only
  • ASR + severity metrics
  • Statistical analysis
  • Defined scenario-based evaluations
• Are we expected to implement one mitigation, or compare multiple?
• Is there an expected minimum improvement after mitigation?

7) Deliverables and Final Output

• What format should final results be delivered in?
  • Report
  • GitHub repository
  • Live demo
  • Dashboard
  • Prototype

Suggested Priority Order for Live Discussion

1. Scope confirmation
2. Contribution expectations
3. Models + architecture scope (chatbot vs RAG/enterprise)
4. Attack scenario scope
5. Evaluation methodology + expected mitigation gains
6. Final deliverable format