Red-teaming depth
Promptfoo's red-team mode is the most complete we found: dozens of plugins (harmful content, PII, BOLA and BFLA, SQL and shell injection, excessive agency, MCP attacks) combined with strategies such as iterative jailbreaks, Crescendo, GOAT, tree search, Best-of-N and encodings, and presets mapped to the OWASP LLM and agentic Top 10s, NIST AI RMF, MITRE ATLAS and the EU AI Act. The free Community tier includes up to 10k probes a month.
Kojen's adversarial simulations are narrower: they generate attacks in five categories (prompt injection, jailbreak, data exfiltration, harmful content, PII leak), judge each response with a model from a different family than the target, and report a robustness score with a confidence interval and severity per attack. Stress tests measure latency and error rates under concurrent load, and scenario simulations check generated tasks against success criteria.
After the scan
Promptfoo tells you what broke. Its promptfoo optimize command can improve a single prompt against one provider, but it has no statistics (no intervals or pass@k metric), no training and no serving.
Kojen keeps the attacks it generated, so you can tune the prompt or switch models in the Forge, rerun the same red-team set and see whether the robustness interval actually moved, then deploy the better version as an OpenAI-compatible endpoint.
What the OpenAI acquisition means
Promptfoo says the open-source project stays MIT-licensed and maintained, and its pricing page still lists Community, Enterprise and On-Premise tiers. OpenAI has also named Promptfoo as the migration path for its own hosted Evals, which became read-only on October 31, 2026 and shut down on November 30, 2026. Teams that red-team models from several vendors may want to weigh the change of ownership; the MIT license means the code can always be forked.
Promptfoo at a glance
An open-source CLI and library for testing prompts, agents and RAG apps with YAML test cases and assertions, plus the broadest red-teaming plugin and strategy set of any tool here, with OWASP, NIST, MITRE ATLAS and EU AI Act presets.
- The widest red-team coverage available, with presets for OWASP LLM and agentic Top 10s, NIST AI RMF, MITRE ATLAS and the EU AI Act.
- MIT-licensed config-as-code that lives in your repo and runs in CI.
- OpenAI now names it as the migration path off OpenAI's own hosted Evals, which shut down on November 30, 2026.
- No statistics built in: no confidence intervals and no pass@k metric.
- Now owned by OpenAI, which some teams testing other vendors' models will weigh.
- Doesn't improve the model it finds problems in; there is no tuning or serving.
Kojen at a glance
A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.
- Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
- Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
- Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.
- Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
- No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
- Closed source and hosted only; no self-hosting.