Prompt injection
Prompt injection is an attack in which text supplied to a language model overrides or subverts its intended instructions, for example "ignore previous instructions and…".
It ranks first (LLM01) in the OWASP Top 10 for LLM Applications. Perez and Ribeiro showed simple handcrafted inputs could hijack GPT-3's goals or leak its prompt.
There is no complete fix, because instructions and data share one channel. Defences layer least-privilege tools, input and output filtering, human confirmation for risky actions, and red-team testing.
Learn more on Kojen
Related terms
- Indirect prompt injectionIndirect prompt injection is prompt injection delivered through content an AI system reads, such as a web page, email, document or tool result, rather than typed by the user..
- Red teamingRed teaming is deliberately attacking an AI system, by hand or with automated attackers, to find harmful, insecure or failing behaviour before real users do..
- MCPThe Model Context Protocol (MCP) is an open protocol for connecting AI applications to external tools, data and prompts through a standard client–server interface, built on JSON-RPC..
Sources
- OWASP. OWASP Top 10 for LLM Applications
- Perez & Ribeiro, 2022. Ignore Previous Prompt: Attack Techniques For Language Models (arXiv:2211.09527)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free