Skip to content

Glossary · Security

Red teaming

Red teaming is deliberately attacking an AI system, by hand or with automated attackers, to find harmful, insecure or failing behaviour before real users do.

Also called AI red teaming, adversarial testing · Reviewed 15 November 2026

Perez et al. used one language model to generate test cases that make another misbehave; Ganguli et al. released about 39,000 human red-team attacks and found RLHF models got harder to red-team as they grew.

For agents, red teaming covers prompt and indirect injection, tool misuse, data exfiltration and goal hijacking, as catalogued in OWASP's agentic Top 10.

Learn more on Kojen

Sources

  1. [1]Perez et al., 2022. Red Teaming Language Models with Language Models (arXiv:2202.03286)
  2. [2]Ganguli et al., 2022. Red Teaming Language Models to Reduce Harms (arXiv:2209.07858)
  3. [3]OWASP. OWASP Top 10 for Agentic Applications for 2026

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free