Agents in Production
Lesson 3 of 6Advanced 10 min

Red-teaming and agent security

Once an agent reads untrusted content and holds real permissions, every webpage and document is a possible attacker.

In this lesson you'll

  • Explain direct and indirect prompt injection
  • Use automated red-teaming
  • Apply least privilege to agent tools

Prompt injection tops the OWASP Top 10 for LLM Applications [1]. Greshake et al. showed the more dangerous variant: indirect prompt injection, where instructions hidden in retrieved content such as a webpage or email hijack an LLM-integrated application without the user typing anything malicious [2]. Zou et al. found adversarial suffixes, found automatically, that transfer across models [3], so refusal training alone isn't a defence.

Agents raise the stakes because they act. In December 2025 OWASP published a separate, peer-reviewed Top 10 for Agentic Applications for 2026, covering the risks specific to autonomous systems that plan, use tools and act on a user's behalf [4].

Red-team before you ship

Perez et al. used one language model to generate test cases that uncover harmful behaviour in another, finding failures at a scale manual testing can't reach [5]. Combine automated attacks with hand-written ones targeting your tools and data.

Defences

  • Least privilege: give each tool the narrowest scope it needs, and use read-only access where possible.
  • Confirm before acting: require approval for irreversible or outward-facing actions.
  • Treat content as data: never let retrieved text grant new permissions or change goals.
  • Log everything: keep transcripts so incidents can be traced.

In Kojen

Adversarial simulations run prompt-injection, jailbreak, data-exfiltration, harmful-content and PII-leak attacks and grade a robustness score. API keys run with their owner's permissions, so create separate, expiring keys for agents.

Key takeaways

  • Indirect prompt injection turns any content into an attack surface.
  • Automate red-teaming, then add targeted manual attacks.
  • Limit what each tool can do, and confirm before irreversible actions.

Practise in the dojo

Run an Adversarial simulation and read which attack categories got through.

Run an adversarial simulation

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. What makes prompt injection "indirect"?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]OWASP GenAI Security Project (2025). OWASP Top 10 for LLM Applications. genai.owasp.org.
  2. [2]Greshake, K. et al. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec 2023 · arXiv:2302.12173.
  3. [3]Zou, A. et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043.
  4. [4]OWASP GenAI Security Project (2025). OWASP Top 10 for Agentic Applications for 2026. genai.owasp.org, December 2025.
  5. [5]Perez, E. et al. (2022). Red Teaming Language Models with Language Models. EMNLP 2022 · arXiv:2202.03286.