Skip to content

Kojen vs · Red-teaming

Kojen vs Promptfoo

Promptfoo is a free, MIT-licensed command-line tool for testing prompts and agents and for red-teaming them, with the broadest attack library and compliance presets available; OpenAI announced it was acquiring Promptfoo on March 9, 2026, and it remains open source. Kojen is a hosted dojo that red-teams and evaluates with confidence intervals, then tunes, retests and deploys the agent, so the choice is between the deepest testing tool and a loop that also fixes what testing finds.

If you only need security scanning in CI, Promptfoo is the stronger and cheaper choice.

By Will SchulzUpdated 13 sources

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

Kojen vs Promptfoo: features and pricing, checked November 2026
Checked Nov 15, 2026PromptfooKojen
What it isRed-teaming

OpenAI (acquisition announced March 9, 2026); still MIT open source

Training dojo

Kojen (that's us)

PricingCommunity: free forever, including self-hosting and red-teaming up to 10k probes a month. Enterprise and On-Premise: custom pricing. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceYes, MIT RepoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Yes

YAML test cases and assertions, runs in CI

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Yes

Model-graded assertions

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

Partial

eval --repeat n, but no pass@k metric

Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

No

Not in the CLI docs

Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Partial

promptfoo optimize: one prompt against one provider

Yes

Prompt tuning plus search across 11 hosted models

Fine-tuning / LoRA

Trains model weights or adapters.

No
Partial

Text LoRA on Qwen3-8B / Llama-3.1-8B, early access; adapters can't be served yet; full fine-tuning and RLHF are previews

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

Yes

Dozens of plugins; Crescendo, GOAT, tree and Best-of-N strategies; OWASP/NIST/MITRE presets

Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Model endpoints

Serves a model or tuned version behind an API you call in production.

No
Yes

OpenAI-compatible, SSE streaming, tool calls forwarded

Production tracing

Logs and traces live traffic from your app.

Partial

OpenTelemetry traces per test case, not production monitoring

Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Yes

promptfoo mcp, plus MCP red-team testing

Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

Yes

Free in the Community tier

No

Hosted only

Red-teaming depth

Promptfoo's red-team mode is the most complete we found: dozens of plugins (harmful content, PII, BOLA and BFLA, SQL and shell injection, excessive agency, MCP attacks) combined with strategies such as iterative jailbreaks, Crescendo, GOAT, tree search, Best-of-N and encodings, and presets mapped to the OWASP LLM and agentic Top 10s, NIST AI RMF, MITRE ATLAS and the EU AI Act. The free Community tier includes up to 10k probes a month.

Kojen's adversarial simulations are narrower: they generate attacks in five categories (prompt injection, jailbreak, data exfiltration, harmful content, PII leak), judge each response with a model from a different family than the target, and report a robustness score with a confidence interval and severity per attack. Stress tests measure latency and error rates under concurrent load, and scenario simulations check generated tasks against success criteria.

After the scan

Promptfoo tells you what broke. Its promptfoo optimize command can improve a single prompt against one provider, but it has no statistics (no intervals or pass@k metric), no training and no serving.

Kojen keeps the attacks it generated, so you can tune the prompt or switch models in the Forge, rerun the same red-team set and see whether the robustness interval actually moved, then deploy the better version as an OpenAI-compatible endpoint.

What the OpenAI acquisition means

Promptfoo says the open-source project stays MIT-licensed and maintained, and its pricing page still lists Community, Enterprise and On-Premise tiers. OpenAI has also named Promptfoo as the migration path for its own hosted Evals, which became read-only on October 31, 2026 and shut down on November 30, 2026. Teams that red-team models from several vendors may want to weigh the change of ownership; the MIT license means the code can always be forked.

Promptfoo at a glance

An open-source CLI and library for testing prompts, agents and RAG apps with YAML test cases and assertions, plus the broadest red-teaming plugin and strategy set of any tool here, with OWASP, NIST, MITRE ATLAS and EU AI Act presets.

Strengths

  • The widest red-team coverage available, with presets for OWASP LLM and agentic Top 10s, NIST AI RMF, MITRE ATLAS and the EU AI Act.
  • MIT-licensed config-as-code that lives in your repo and runs in CI.
  • OpenAI now names it as the migration path off OpenAI's own hosted Evals, which shut down on November 30, 2026.

Limits

  • No statistics built in: no confidence intervals and no pass@k metric.
  • Now owned by OpenAI, which some teams testing other vendors' models will weigh.
  • Doesn't improve the model it finds problems in; there is no tuning or serving.

Kojen at a glance

A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.

Strengths

  • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
  • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
  • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

Limits

  • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
  • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
  • Closed source and hosted only; no self-hosting.

When to choose Promptfoo, and when to choose Kojen

Choose Promptfoo if…

  • You need the widest attack coverage and compliance presets (OWASP, NIST, MITRE ATLAS, EU AI Act).
  • You want config-as-code tests that run free in CI on your own machines.
  • You are migrating off OpenAI's hosted Evals, which OpenAI points to Promptfoo.
  • You need on-premise deployment with full data isolation.

Choose Kojen if…

  • You want to fix what red-teaming finds and rerun the same attacks to prove it.
  • You need confidence intervals on robustness and pass rates, not just counts.
  • You want evaluations, tuning and a production endpoint in one hosted product.
  • You prefer a tool not owned by a model vendor.

The short version: For pure red-team coverage, Promptfoo wins. Kojen wins when red-teaming is one step in a loop that also tunes, re-measures and ships the agent. Running Promptfoo in CI and Kojen for training is a reasonable combination.

Frequently asked questions

Did OpenAI buy Promptfoo?
Yes. Promptfoo announced on March 9, 2026 that it was joining OpenAI. The project says it remains open source under the MIT license, and existing customers continue to be supported.
Is Promptfoo still free?
Yes. As of November 2026 the Community tier is free forever, including self-hosting and red-teaming up to 10k probes a month. Enterprise and On-Premise tiers are custom-priced.
What replaced OpenAI Evals?
OpenAI's hosted Evals dashboard, API and graders went read-only on October 31, 2026 and shut down on November 30, 2026. OpenAI's deprecations page recommends migrating to Promptfoo, and its cookbook has a migration guide.
Does Promptfoo have an MCP server?
Yes, promptfoo mcp runs one over stdio or HTTP, and Promptfoo can also red-team MCP servers. Kojen has its own MCP server for driving training, evaluation and deployment.
Which attack types does Kojen test?
Prompt injection, jailbreak, data exfiltration, harmful content and PII leak, with a configurable number of attacks per type. Errors count as failures, so a crashed response is never scored as defended.

Keep reading

Sources

Checked November 2026.

  1. [1]Promptfoo is joining OpenAI. Promptfoo blog.
  2. [2]OpenAI acquires Promptfoo to secure its AI agents. TechCrunch.
  3. [3]Pricing. Promptfoo.
  4. [4]promptfoo/promptfoo. GitHub.
  5. [5]Red team configuration. Promptfoo docs.
  6. [6]Command line. Promptfoo docs.
  7. [7]MCP server. Promptfoo docs.
  8. [8]Tracing. Promptfoo docs.
  9. [9]Kojen for Agents: REST API & MCP Server. Kojen.
  10. [10]llms.txt. Kojen.
  11. [11]Pricing. Kojen.
  12. [12]Deprecations: Evals platform. OpenAI docs.
  13. [13]Moving from OpenAI Evals to Promptfoo. OpenAI Cookbook.

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free