Red teaming
Red teaming is deliberately attacking an AI system, by hand or with automated attackers, to find harmful, insecure or failing behaviour before real users do.
Perez et al. used one language model to generate test cases that make another misbehave; Ganguli et al. released about 39,000 human red-team attacks and found RLHF models got harder to red-team as they grew.
For agents, red teaming covers prompt and indirect injection, tool misuse, data exfiltration and goal hijacking, as catalogued in OWASP's agentic Top 10.
Learn more on Kojen
Related terms
- Prompt injectionPrompt injection is an attack in which text supplied to a language model overrides or subverts its intended instructions, for example "ignore previous instructions and…"..
- Indirect prompt injectionIndirect prompt injection is prompt injection delivered through content an AI system reads, such as a web page, email, document or tool result, rather than typed by the user..
- Agent evalAn agent eval measures an AI agent on multi-step tasks, scoring both the final outcome and how it got there, including tool calls, turns, cost and reliability over repeated runs..
Sources
- Perez et al., 2022. Red Teaming Language Models with Language Models (arXiv:2202.03286)
- Ganguli et al., 2022. Red Teaming Language Models to Reduce Harms (arXiv:2209.07858)
- OWASP. OWASP Top 10 for Agentic Applications for 2026
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free