Skip to content

Best of · Evaluation

The best LLM evaluation tools in 2026

The best LLM evaluation tools in 2026 are Braintrust for a polished hosted eval workflow, Langfuse for open-source tracing and evals you can self-host free, LangSmith for LangChain teams, Inspect for rigorous statistics, DeepEval for the largest metric library, and Promptfoo for evals and red-teaming in CI. Kojen fits teams that want confidence intervals and pass^k on every run and a loop that also tunes, red-teams and deploys the agent.

One change matters for anyone choosing now: OpenAI's hosted Evals went read-only on October 31, 2026 and shut down on November 30, 2026.

By Will SchulzUpdated 57 sources

The list

  1. 01Braintrust

    www.braintrust.dev

    Best for: a polished hosted eval and experiment workflow

    Well-built experiments, scorers and side-by-side comparison, online scoring of production logs, and the Loop agent for suggesting prompt and scorer edits. Unlimited users on the free tier. No confidence intervals, red-teaming or self-hosting below Enterprise.

    Pricing: Starter $0 (unlimited users, 1 GB, 10k scores, 14-day retention); Pro $249/month; Enterprise custom. Source · Open source: no

    Strengths

    • A mature, polished eval and tracing product, with unlimited users even on the free tier.
    • Loop works across logs, experiments, datasets and scorers in one place.
    • Gateway, environments and online scoring of production traffic in the same product.

    Limits

    • No confidence intervals or significance tests, and no pass@k beyond grouping trials.
    • Measures and suggests, but doesn't train weights, red-team or serve models.
    • Self-hosting is Enterprise-only and still hybrid.
  2. 02Langfuse

    langfuse.com

    Best for: open-source tracing and evals you can self-host free

    MIT-licensed and free to self-host with all core features, with cheap cloud plans. Strong tracing, datasets and LLM judges; no repetitions per item, statistics or prompt optimizer.

    Pricing: Cloud: Hobby free (50k units/month); Core $29/month; Pro $199/month; Enterprise $2,499/month. Self-hosted open source: free. Source · Open source: yes (MIT (except ee/ folders))

    Strengths

    • Genuinely open source (MIT) with free, unrestricted self-hosting.
    • Inexpensive usage-based cloud pricing, with long retention on Pro.
    • An MCP server that can write prompts, datasets and scores as well as read them.

    Limits

    • No repetitions per dataset item, so no pass@k-style reliability metrics, and no statistical tests.
    • No prompt optimizer, red-teaming or training: it observes and evaluates, you improve elsewhere.
  3. Best for: teams building on LangChain or LangGraph

    Tracing, evaluation, annotation queues and agent hosting in one product, with repetitions reported as mean and standard deviation. Per-seat pricing.

    Pricing: Developer $0 (1 seat, 5k base traces/month); Plus $39/seat/month (10k traces); Enterprise custom. Usage beyond that is pay as you go. Source · Open source: no

    Strengths

    • The natural choice for LangChain and LangGraph teams, with production hosting for agents.
    • Strong tracing, annotation queues and prompt management.
    • Cloud, hybrid and self-hosted deployment options.

    Limits

    • Per-seat pricing, and the free tier has one seat.
    • Reliability stats stop at mean and standard deviation; no intervals or pass^k.
    • No red-teaming or weight training.
  4. Best for: rigorous statistics and agent evals in code

    The UK AI Security Institute's framework has the best statistics we found (clustered standard errors, bootstrap and Wilson intervals, pass@k and pass^k reducers), sandboxed agent runs and 200+ pre-built evals. No hosted team platform.

    Pricing: Free, MIT. Source · Open source: yes (MIT)

    Strengths

    • Rigorous statistics: clustered standard errors, bootstrap and Wilson intervals, pass@k and pass^k.
    • Sandboxed agent evaluation (Docker, Kubernetes, Modal and more), including external agents.
    • A credible public-sector maintainer and a large benchmark library.

    Limits

    • Researcher-oriented: no hosted team platform or production tracing.
    • No red-team attack generation and no prompt optimization.
  5. 05Kojenour product

    kojen.ai

    Best for: evaluations you can trust, plus improving what they measure

    Every rate comes with a confidence interval, several samples per case give pass@k and pass^k, judges come from a different model family than the target in simulations, and red-team, prompt tuning and deployment share the same test cases. It doesn't trace apps running elsewhere. We make Kojen.

    Pricing: Free to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source · Open source: no

    Strengths

    • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
    • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
    • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

    Limits

    • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
    • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
    • Closed source and hosted only; no self-hosting.
  6. 06DeepEval / Confident AI

    www.confident-ai.com

    Best for: the largest library of ready-made metrics

    Open-source metrics for RAG, agents, multi-turn conversations and MCP, built-in GEPA and MIPROv2 prompt optimizers, and a hosted platform with tracing. Paid tiers start at $200 a month.

    Pricing: Free (2 seats, 1 project); Starter $200/month; Team $2,000/month; Team++ and Enterprise custom. Priced per organization, not per seat. Source · Open source: yes (Apache-2.0)

    Strengths

    • One of the largest metric libraries, including agent, multi-turn and MCP metrics.
    • Prompt optimizers (GEPA, MIPROv2) built in.
    • A hosted MCP server and tracing platform on top of the open-source library.

    Limits

    • Red-teaming is a separate library, and in the hosted product it is Enterprise-only.
    • Steep jump between paid tiers, from $200 to $2,000 a month.
  7. Best for: evals and red-teaming as code in CI

    Free, MIT-licensed YAML test suites with the broadest red-team coverage, now owned by OpenAI and recommended by OpenAI as the replacement for its hosted Evals. No statistics or production monitoring.

    Pricing: Community: free forever, including self-hosting and red-teaming up to 10k probes a month. Enterprise and On-Premise: custom pricing. Source · Open source: yes (MIT)

    Strengths

    • The widest red-team coverage available, with presets for OWASP LLM and agentic Top 10s, NIST AI RMF, MITRE ATLAS and the EU AI Act.
    • MIT-licensed config-as-code that lives in your repo and runs in CI.
    • OpenAI now names it as the migration path off OpenAI's own hosted Evals, which shut down on November 30, 2026.

    Limits

    • No statistics built in: no confidence intervals and no pass@k metric.
    • Now owned by OpenAI, which some teams testing other vendors' models will weigh.
    • Doesn't improve the model it finds problems in; there is no tuning or serving.
  8. 08Arize Phoenix

    phoenix.arize.com

    Best for: OpenTelemetry-native tracing with free self-hosting

    Free to self-host without feature gates, experiment repetitions, and automated Prompt Learning on Arize AX. Source-available under the Elastic License 2.0.

    Pricing: Phoenix self-hosted: free. Arize AX: Free (25k spans/month), Pro $50/month (50k spans), Enterprise custom. Source · Open source: yes (Elastic License 2.0 (source-available))

    Strengths

    • Free self-hosting with no feature gates.
    • OpenTelemetry-native, so instrumentation isn't tied to one vendor.
    • Automated prompt optimization (Prompt Learning) on the AX platform.

    Limits

    • Elastic License 2.0 is source-available, not OSI open source, and restricts offering it as a service.
    • Small span allowances on AX Free and Pro; several features are AX-only.
  9. Best for: RAG-specific metrics in a notebook

    The de facto standard faithfulness and context metrics, plus synthetic test-set generation. Narrow in scope, with no statistics or platform.

    Pricing: Free, Apache-2.0. No paid product listed. Source · Open source: yes (Apache-2.0)

    Strengths

    • The de facto standard RAG metrics, reused by other tools.
    • Synthetic test-set generation.
    • Small and easy to drop into a notebook or CI.

    Limits

    • Narrow scope, with no security testing.
    • No statistics and no team platform.

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

The best LLM evaluation tools in 2026: features and pricing, checked November 2026
Checked Nov 15, 2026BraintrustLangfuseLangSmithInspectDeepEval / Confident AIPromptfooKojen
What it isEvaluation & observability

Braintrust Data (independent)

Evaluation & observability

ClickHouse (acquisition announced January 16, 2026); still MIT open source

Evaluation & observability

LangChain (independent)

Evaluation framework

UK AI Security Institute

Evaluation framework

Confident AI

Red-teaming

OpenAI (acquisition announced March 9, 2026); still MIT open source

Training dojo

Kojen (that's us)

PricingStarter $0 (unlimited users, 1 GB, 10k scores, 14-day retention); Pro $249/month; Enterprise custom. SourceCloud: Hobby free (50k units/month); Core $29/month; Pro $199/month; Enterprise $2,499/month. Self-hosted open source: free. SourceDeveloper $0 (1 seat, 5k base traces/month); Plus $39/seat/month (10k traces); Enterprise custom. Usage beyond that is pay as you go. SourceFree, MIT. SourceFree (2 seats, 1 project); Starter $200/month; Team $2,000/month; Team++ and Enterprise custom. Priced per organization, not per seat. SourceCommunity: free forever, including self-hosting and red-teaming up to 10k probes a month. Enterprise and On-Premise: custom pricing. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceNoYes, MIT (except ee/ folders) RepoNoYes, MIT RepoYes, Apache-2.0 RepoYes, MIT RepoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Yes

Eval() experiments on datasets

Yes

Experiments on datasets

Yes

evaluate() on datasets

Yes

200+ pre-built evals

Yes
Yes

YAML test cases and assertions, runs in CI

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Yes

autoevals library plus custom scorers

Yes

Managed evaluator catalog and custom judges

Yes

LLM evaluators from the UI or SDK

Yes

model_graded_qa and related scorers

Yes

G-Eval, DAG and dozens of built-in metrics

Yes

Model-graded assertions

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

Partial

trialCount reruns each input; no pass@k metric

No

Experiments assume no repetitions per item

Partial

num_repetitions with mean and standard deviation; no pass@k

Yes

Epochs with pass_at(k), pass_k and at_least reducers

Partial

Repeat runs per case; no pass@k metric

Partial

eval --repeat n, but no pass@k metric

Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

No

Score deltas and improvement/regression counts

No

Averages and deltas

No

Standard deviation only

Yes

stderr (clustered), bootstrap_stderr, ci, ci_wilson

No

Not in the official docs

No

Not in the CLI docs

Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Partial

Loop agent suggests prompt edits you accept

No

Prompt versioning and A/B labels, no optimizer

Partial

Optimization jobs (promptim) via API; Polly edits prompts on request

No
Yes

PromptOptimizer with GEPA and MIPROv2

Partial

promptfoo optimize: one prompt against one provider

Yes

Prompt tuning plus search across 11 hosted models

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

No

Only a prompt-injection-detector cookbook

No
No
Partial

Safety and agentic benchmarks; no attack generator

Yes

DeepTeam library; hosted red-teaming is Enterprise-only

Yes

Dozens of plugins; Crescendo, GOAT, tree and Best-of-N strategies; OWASP/NIST/MITRE presets

Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Production tracing

Logs and traces live traffic from your app.

Yes

Tracing and online scoring

Yes

OpenTelemetry-based tracing

Yes
Partial

Inspect View log viewer, not production traffic

Yes

Tracing in Confident AI

Partial

OpenTelemetry traces per test case, not production monitoring

Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Yes

Hosted MCP server

Yes

Authenticated MCP server, read and write

Yes

Python server and hosted remote MCP with OAuth

Partial

Agents under test can use MCP tools; no Inspect MCP server

Yes

Hosted MCP server

Yes

promptfoo mcp, plus MCP red-team testing

Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

Partial

Enterprise only; you run the data plane, Braintrust the control plane

Yes

Free, no core-feature limits

Partial

Enterprise add-on

Yes
Partial

Libraries run locally; platform on-prem is Enterprise

Yes

Free in the Community tier

No

Hosted only

What separates a good evaluation tool

Most tools now run datasets and LLM judges well. The differences are in what they tell you about uncertainty. A pass rate of 80% on 20 cases has a 95% interval of roughly 58% to 92%, so a change from 80% to 85% on that set means very little; only Inspect and Kojen report intervals out of the box. Likewise, an agent that succeeds 75% of the time per try succeeds on all three of three tries only about 42% of the time, which is why pass^k matters for anything customer-facing.

The other split is between tools that observe production traffic (Braintrust, Langfuse, LangSmith, Phoenix, Weave) and tools that test before release (Inspect, DeepEval, Promptfoo, Kojen). Many teams need one of each.

How we made this list

  • We scored each tool on what an evaluation needs to be trustworthy: datasets, LLM judges, repeated samples per case (pass@k / pass^k), confidence intervals, and whether it helps you act on the result (prompt optimization, red-teaming).
  • Order reflects how broadly useful each tool is for evaluating LLM apps and agents in production, not a single score.
  • Every feature claim links to the vendor's own pricing page, docs or repository, checked on November 15, 2026. Where we couldn't confirm a feature, the table says "Not verified".
  • We make Kojen. We include it where it does the job and say where others are stronger; it is not placed first by default.
  • No vendor paid to be included, and there are no affiliate links.

Frequently asked questions

What is the best open-source LLM evaluation tool?
Langfuse (MIT) for a full platform with tracing you can self-host; Inspect (MIT) for rigorous statistics and agent evals; DeepEval (Apache-2.0) for the widest metric library; Promptfoo (MIT) for evals and red-teaming in CI.
Which LLM evaluation tools report confidence intervals?
Of the tools we checked in November 2026, Inspect (stderr, bootstrap and Wilson interval metrics) and Kojen (an interval on every rate) do. Braintrust, LangSmith, Langfuse, Phoenix, Weave, DeepEval and Promptfoo report averages, deltas or standard deviations.
What is the difference between pass@k and pass^k?
pass@k is the chance at least one of k tries succeeds and rises with k; pass^k, introduced with τ-bench, is the chance all k tries succeed and falls with k. Use pass^k for agents that must be reliable every time.
What happened to OpenAI Evals?
OpenAI's hosted Evals dashboard, API and graders went read-only on October 31, 2026 and shut down on November 30, 2026; OpenAI recommends migrating to Promptfoo. The open-source openai/evals repository still exists.

Keep reading

Sources

Checked November 2026.

  1. [1]Pricing. Braintrust.
  2. [2]Evaluate. Braintrust docs.
  3. [3]Advanced evaluations (trials). Braintrust docs.
  4. [4]Compare experiments. Braintrust docs.
  5. [5]Loop. Braintrust docs.
  6. [6]Deploy. Braintrust docs.
  7. [7]MCP. Braintrust docs.
  8. [8]Self-hosting. Braintrust docs.
  9. [9]Pricing. Langfuse.
  10. [10]Self-hosted pricing. Langfuse.
  11. [11]Langfuse is joining ClickHouse. Langfuse blog.
  12. [12]Database maker ClickHouse raises $400M, acquires AI observability startup Langfuse. SiliconANGLE.
  13. [13]LICENSE. GitHub.
  14. [14]Experiments data model. Langfuse docs.
  15. [15]Interpreting experiment results. Langfuse blog.
  16. [16]LLM-as-a-judge. Langfuse docs.
  17. [17]MCP server. Langfuse docs.
  18. [18]Pricing. LangChain.
  19. [19]Repetition. LangSmith docs.
  20. [20]Run an evaluation from the prompt playground. LangSmith docs.
  21. [21]Create optimization job. LangSmith API docs.
  22. [22]Polly. LangSmith docs.
  23. [23]Agent Server. LangSmith docs.
  24. [24]LangSmith MCP server. LangSmith docs.
  25. [25]Self-hosted LangSmith. LangSmith docs.
  26. [26]UKGovernmentBEIS/inspect_ai. GitHub.
  27. [27]Inspect. UK AI Security Institute.
  28. [28]inspect_ai.scorer reference. Inspect docs.
  29. [29]Pricing. Confident AI.
  30. [30]confident-ai/deepeval. GitHub.
  31. [31]confident-ai/deepteam. GitHub.
  32. [32]Evaluation flags and configs. DeepEval docs.
  33. [33]Prompt optimization: MIPROv2. DeepEval docs.
  34. [34]MCP reference. Confident AI docs.
  35. [35]Promptfoo is joining OpenAI. Promptfoo blog.
  36. [36]OpenAI acquires Promptfoo to secure its AI agents. TechCrunch.
  37. [37]Pricing. Promptfoo.
  38. [38]promptfoo/promptfoo. GitHub.
  39. [39]Red team configuration. Promptfoo docs.
  40. [40]Command line. Promptfoo docs.
  41. [41]MCP server. Promptfoo docs.
  42. [42]Tracing. Promptfoo docs.
  43. [43]Pricing. Arize.
  44. [44]Arize-ai/phoenix. GitHub.
  45. [45]Phoenix license. Arize docs.
  46. [46]Experiment repetitions. Arize docs.
  47. [47]Optimizing prompts. Arize docs.
  48. [48]LLM red teaming. Arize docs.
  49. [49]Phoenix MCP server. Arize docs.
  50. [50]vibrantlabsai/ragas. GitHub.
  51. [51]Ragas documentation. Ragas.
  52. [52]Optimizers API reference. Ragas docs.
  53. [53]Kojen for Agents: REST API & MCP Server. Kojen.
  54. [54]llms.txt. Kojen.
  55. [55]Pricing. Kojen.
  56. [56]Deprecations: Evals platform. OpenAI docs.
  57. [57]Moving from OpenAI Evals to Promptfoo. OpenAI Cookbook.

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free