A chatbot that says something wrong is embarrassing. An agent that does something wrong, such as issuing a refund, emailing a customer list or running a query it shouldn't, is an incident. The more tools you give an agent, the more ways an attacker has to misuse it.
Sparring is part of training in any dojo, and it should be part of yours. Here's how to attack your own agent before someone else does.
Know the threat list
The OWASP Top 10 for LLM Applications (2025) is the most widely used starting point. It ranks prompt injection as LLM01, the top risk, followed by sensitive information disclosure. Further down are risks that matter most for agents: excessive agency (LLM06), system prompt leakage (LLM07) and unbounded consumption (LLM10).[1]
Indirect injection is the agent-shaped threat
Direct injection is a user typing “ignore your instructions.” The more dangerous version is indirect. Greshake et al. showed that attackers can plant instructions in data an LLM-integrated application is likely to retrieve, like a web page, an email or a document, and take over the application without ever talking to it directly. They demonstrated data theft and control over which APIs get called against real systems, including Bing's GPT-4-powered chat.[2]
If your agent reads anything it didn't write (support tickets, web search results, uploaded files, tool outputs), it's exposed to this.
Automate the attacker
Hand-written attack prompts run out quickly. Perez et al. at DeepMind showed that you can use one language model to red-team another. Generated test cases uncovered tens of thousands of offensive replies in a 280B-parameter chatbot, and also surfaced leaked training data and the chatbot generating personal and hospital phone numbers as its own contact details.[3] Automated red-teaming won't find everything, but it finds far more than a person with a spreadsheet.
It's also what regulators expect. NIST's Generative AI Profile (AI 600-1) repeatedly recommends AI red-teaming, adversarial role-play and independent evaluation as part of managing generative AI risk.[4]
A red-team checklist for agents
- Prompt injection: direct (“ignore previous instructions”) and indirect (the same text hidden in a document the agent retrieves).
- Jailbreaks: role-play, hypotheticals and multi-turn escalation designed to unlock behaviour the agent should refuse.
- Data exfiltration: attempts to get the agent to reveal other users' records, its system prompt or its tool credentials.
- PII leakage: does it repeat back personal data it saw in context when it shouldn't?
- Harmful content: requests the agent must decline, whatever the framing.
- Excessive agency: can someone talk it into an action outside its remit, like a refund above its limit or a delete without confirmation?
Load is an attack surface too
An agent that's correct but takes 40 seconds under load has still failed. Unbounded consumption is on the OWASP list for a reason.[1] Google's SRE book argues for tracking latency as percentiles rather than averages, because the average hides the slow tail your unhappiest users actually experience.[5] Measure p50, p95 and p99 under realistic concurrency, not one request at a time.
Running it in Kojen
Kojen's Simulations page covers both kinds of sparring:
- Adversarial runs red-team attacks across prompt injection, jailbreak, data exfiltration, harmful content and PII leakage, grades each response and rolls the results into a robustness score.
- Stress test sends real concurrent requests and measures latency, error rate and throughput.
- Scenario runs scenarios you write (or ones Kojen generates) and grades each one pass or fail. It's the place for your agent-specific abuse cases.
Defence is layered
Red-teaming tells you where you're weak. It doesn't make you strong on its own. Pair it with least-privilege tools (an agent that can't delete can't be tricked into deleting), confirmation steps for irreversible actions, and treating everything the agent retrieves as data rather than instructions. The Kojen Agent itself asks before deleting anything, for the same reason.
