Skip to content

Alternatives · Red-teaming

Promptfoo alternatives

The best Promptfoo alternatives are DeepEval with DeepTeam for an open-source eval-plus-red-team stack, NVIDIA's garak or Microsoft's PyRIT for pure security testing, Giskard for EU-hosted red-teaming, the UK AISI's Inspect for rigorous statistics, and Kojen if you want to fix and retest what red-teaming finds. Promptfoo itself is still free and MIT-licensed after OpenAI announced it was acquiring the company on March 9, 2026, so switching is a choice, not a necessity.

Teams usually look for an alternative for one of three reasons: vendor neutrality after the acquisition, statistics Promptfoo doesn't provide, or a loop that also improves the model.

By Will SchulzUpdated 30 sources

The best Promptfoo alternatives

  1. 01DeepEval / Confident AI

    www.confident-ai.com

    Best for: an open-source eval and red-team stack with a hosted platform

    DeepEval's metric library is one of the largest available, and its sibling DeepTeam covers red-teaming mapped to the OWASP LLM Top 10, OWASP agentic risks, NIST AI RMF and MITRE ATLAS. Confident AI adds tracing and a hosted MCP server; hosted red-teaming is Enterprise-only.

    Pricing: Free (2 seats, 1 project); Starter $200/month; Team $2,000/month; Team++ and Enterprise custom. Priced per organization, not per seat. Source · Open source: yes (Apache-2.0)

    Strengths

    • One of the largest metric libraries, including agent, multi-turn and MCP metrics.
    • Prompt optimizers (GEPA, MIPROv2) built in.
    • A hosted MCP server and tracing platform on top of the open-source library.

    Limits

    • Red-teaming is a separate library, and in the hosted product it is Enterprise-only.
    • Steep jump between paid tiers, from $200 to $2,000 a month.
  2. Best for: scanning a model or endpoint for known vulnerabilities

    NVIDIA's scanner runs a large, research-backed probe library against many kinds of targets and tags results by OWASP category. It's free and Apache-2.0, but it tests the model rather than your app's own cases.

    Pricing: Free, Apache-2.0. No paid tier. Source · Open source: yes (Apache-2.0)

    Strengths

    • A large, research-backed probe library.
    • Talks to many targets: Hugging Face, OpenAI, Bedrock, NIM, LiteLLM, REST and local gguf models.
    • OWASP tagging and AVID reporting.

    Limits

    • Scans a model or endpoint rather than your app's own test cases.
    • CLI only, with no team platform.
  3. Best for: expert red teams running multi-turn attacks

    Microsoft's toolkit composes targets, converters and scorers into adaptive multi-turn attacks such as Crescendo, TAP and PAIR, across text, image, audio and video. It takes more code than Promptfoo but goes deeper on conversation-level attacks.

    Pricing: Free, MIT. Source · Open source: yes (MIT)

    Strengths

    • Strong multi-turn, adaptive attacks.
    • Composable building blocks suited to expert, human-in-the-loop red teams.
    • Maintained by Microsoft's AI Red Team (moved to github.com/microsoft/PyRIT in 2026).

    Limits

    • Code-heavy, with more setup than a scanner.
    • No general evaluation workflow and no published compliance mapping that we found.
  4. Best for: red-teaming with EU data residency or on-prem

    Giskard combines quality checks and a vulnerability scan in an open-source library, with Giskard Hub offering continuous red-teaming in the EU, the US, private cloud or on-prem. Enterprise pricing isn't public.

    Pricing: Open-source library free; Hub Enterprise by quote. SaaS in the EU or US, private cloud or on-prem. Source · Open source: yes (Apache-2.0)

    Strengths

    • Quality checks and red-teaming in one tool.
    • EU data residency and on-prem options.
    • A Hub playground non-developers can use.

    Limits

    • Enterprise pricing isn't published.
    • The v2 to v3 rewrite dropped some features (v2 is no longer maintained).
  5. Best for: evaluations with rigorous statistics

    Inspect has the best statistics of any tool we checked (clustered standard errors, bootstrap and Wilson intervals, pass@k and pass^k reducers) and sandboxed agent evals, but no attack generator.

    Pricing: Free, MIT. Source · Open source: yes (MIT)

    Strengths

    • Rigorous statistics: clustered standard errors, bootstrap and Wilson intervals, pass@k and pass^k.
    • Sandboxed agent evaluation (Docker, Kubernetes, Modal and more), including external agents.
    • A credible public-sector maintainer and a large benchmark library.

    Limits

    • Researcher-oriented: no hosted team platform or production tracing.
    • No red-team attack generation and no prompt optimization.
  6. 06Kojenour product

    kojen.ai

    Best for: red-teaming as one step in a train, retest and deploy loop

    Kojen's red-team covers fewer attack types than Promptfoo (five categories), but it reports robustness with a confidence interval, keeps the generated attacks for reruns, and lets you tune the prompt or switch models and retest before deploying. We make Kojen.

    Pricing: Free to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source · Open source: no

    Strengths

    • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
    • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
    • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

    Limits

    • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
    • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
    • Closed source and hosted only; no self-hosting.

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

Promptfoo alternatives: features and pricing, checked November 2026
Checked Nov 15, 2026PromptfooDeepEval / Confident AIgarakPyRITInspectKojen
What it isRed-teaming

OpenAI (acquisition announced March 9, 2026); still MIT open source

Evaluation framework

Confident AI

Red-teaming

NVIDIA

Red-teaming

Microsoft AI Red Team

Evaluation framework

UK AI Security Institute

Training dojo

Kojen (that's us)

PricingCommunity: free forever, including self-hosting and red-teaming up to 10k probes a month. Enterprise and On-Premise: custom pricing. SourceFree (2 seats, 1 project); Starter $200/month; Team $2,000/month; Team++ and Enterprise custom. Priced per organization, not per seat. SourceFree, Apache-2.0. No paid tier. SourceFree, MIT. SourceFree, MIT. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceYes, MIT RepoYes, Apache-2.0 RepoYes, Apache-2.0 RepoYes, MIT RepoYes, MIT RepoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Yes

YAML test cases and assertions, runs in CI

Yes
Partial

Runs its own probe sets, not your test cases

Partial

Scenarios and datasets for repeatable assessments

Yes

200+ pre-built evals

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Yes

Model-graded assertions

Yes

G-Eval, DAG and dozens of built-in metrics

Yes

ModelAsJudge, refusal and jailbreak judges

Yes

True/false, Likert and classification scorers

Yes

model_graded_qa and related scorers

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

Partial

eval --repeat n, but no pass@k metric

Partial

Repeat runs per case; no pass@k metric

Partial

--generations outputs per prompt (default 10); pass counts, no pass@k

Not verified
Yes

Epochs with pass_at(k), pass_k and at_least reducers

Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

No

Not in the CLI docs

No

Not in the official docs

Not verified
Not verified
Yes

stderr (clustered), bootstrap_stderr, ci, ci_wilson

Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Partial

promptfoo optimize: one prompt against one provider

Yes

PromptOptimizer with GEPA and MIPROv2

No
No
No
Yes

Prompt tuning plus search across 11 hosted models

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

Yes

Dozens of plugins; Crescendo, GOAT, tree and Best-of-N strategies; OWASP/NIST/MITRE presets

Yes

DeepTeam library; hosted red-teaming is Enterprise-only

Yes

Probe/detector scanner with OWASP tags and AVID reports

Yes

Crescendo, TAP, PAIR, many-shot, Skeleton Key; text, image, audio and video converters

Partial

Safety and agentic benchmarks; no attack generator

Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Production tracing

Logs and traces live traffic from your app.

Partial

OpenTelemetry traces per test case, not production monitoring

Yes

Tracing in Confident AI

No

JSONL and HTML reports

No

Stores conversations and scores, not live traffic

Partial

Inspect View log viewer, not production traffic

Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Yes

promptfoo mcp, plus MCP red-team testing

Yes

Hosted MCP server

No
Not verified
Partial

Agents under test can use MCP tools; no Inspect MCP server

Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

Yes

Free in the Community tier

Partial

Libraries run locally; platform on-prem is Enterprise

Yes
Yes
Yes
No

Hosted only

Do you need to leave Promptfoo?

Probably not for technical reasons. Promptfoo says the CLI stays MIT-licensed and maintained, the Community tier is still free with up to 10k red-team probes a month, and OpenAI now recommends it as the replacement for OpenAI's own hosted Evals, which shut down on November 30, 2026. Its red-team coverage, with OWASP, NIST, MITRE ATLAS and EU AI Act presets, remains the broadest available.

The reasons to look elsewhere are narrower: a policy against tools owned by a model vendor, a need for confidence intervals or pass@k (Promptfoo has a --repeat flag but no such metrics), or wanting the same tool to improve the model after testing it.

When to choose Promptfoo, and when to choose Kojen

Choose Promptfoo if…

  • You want the broadest red-team plugin and strategy library, free.
  • You need compliance presets for OWASP, NIST, MITRE ATLAS or the EU AI Act.
  • You're moving off OpenAI's hosted Evals, which OpenAI points to Promptfoo.

Choose Kojen if…

  • You want robustness and pass rates with confidence intervals.
  • You want to tune, retest against the same attacks and deploy, in one hosted product.
  • You'd rather use a tool that isn't owned by a model vendor.

Frequently asked questions

Is Promptfoo still open source after the OpenAI acquisition?
Yes. Promptfoo's announcement and README say it remains open source under the MIT license, and the free Community tier is unchanged as of November 2026.
What is the best free alternative to Promptfoo?
For red-teaming, NVIDIA garak (Apache-2.0) or Microsoft PyRIT (MIT); for evaluations, DeepEval (Apache-2.0) or Inspect (MIT). DeepTeam, from the DeepEval team, is the closest open-source match for Promptfoo's combined eval and red-team scope.
Which Promptfoo alternative maps to the OWASP LLM Top 10?
DeepTeam maps to the OWASP LLM Top 10 and agentic risks, NIST AI RMF and MITRE ATLAS; garak tags probes by OWASP category; Giskard's scan covers OWASP LLM Top 10 categories.
Which tool reports confidence intervals on red-team results?
Of the tools here, Kojen reports robustness with a confidence interval, and Inspect provides interval metrics for evaluations generally. Promptfoo, DeepEval and garak report counts and rates without intervals.

Keep reading

Sources

Checked November 2026.

  1. [1]Promptfoo is joining OpenAI. Promptfoo blog.
  2. [2]OpenAI acquires Promptfoo to secure its AI agents. TechCrunch.
  3. [3]Pricing. Promptfoo.
  4. [4]promptfoo/promptfoo. GitHub.
  5. [5]Red team configuration. Promptfoo docs.
  6. [6]Command line. Promptfoo docs.
  7. [7]MCP server. Promptfoo docs.
  8. [8]Tracing. Promptfoo docs.
  9. [9]Pricing. Confident AI.
  10. [10]confident-ai/deepeval. GitHub.
  11. [11]confident-ai/deepteam. GitHub.
  12. [12]Evaluation flags and configs. DeepEval docs.
  13. [13]Prompt optimization: MIPROv2. DeepEval docs.
  14. [14]MCP reference. Confident AI docs.
  15. [15]NVIDIA/garak. GitHub.
  16. [16]garak CLI reference. garak docs.
  17. [17]garak judge detectors. garak docs.
  18. [18]microsoft/PyRIT. GitHub.
  19. [19]PyRIT documentation 1.1.0. Microsoft.
  20. [20]UKGovernmentBEIS/inspect_ai. GitHub.
  21. [21]Inspect. UK AI Security Institute.
  22. [22]inspect_ai.scorer reference. Inspect docs.
  23. [23]Pricing. Giskard.
  24. [24]Giskard. Giskard.
  25. [25]Giskard-AI/giskard-oss. GitHub.
  26. [26]Kojen for Agents: REST API & MCP Server. Kojen.
  27. [27]llms.txt. Kojen.
  28. [28]Pricing. Kojen.
  29. [29]Deprecations: Evals platform. OpenAI docs.
  30. [30]Moving from OpenAI Evals to Promptfoo. OpenAI Cookbook.

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free