Skip to content

Kojen vs · Evaluation

Kojen vs Braintrust

Braintrust is a mature hosted platform for running evals and tracing production LLM apps, with an agent (Loop) that suggests prompt and scorer changes; Kojen is a training dojo that evaluates an agent with confidence intervals and pass^k, red-teams it, tunes its prompt and model, and serves the result behind an OpenAI-compatible endpoint. Pick Braintrust if you mainly need to observe and score an app you run elsewhere; pick Kojen if you want measuring and improving in one loop, with statistics that tell you when a difference is real.

Braintrust is the more established product for production observability, and Kojen does not try to replace its tracing.

By Will SchulzUpdated 11 sources

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

Kojen vs Braintrust: features and pricing, checked November 2026
Checked Nov 15, 2026BraintrustKojen
What it isEvaluation & observability

Braintrust Data (independent)

Training dojo

Kojen (that's us)

PricingStarter $0 (unlimited users, 1 GB, 10k scores, 14-day retention); Pro $249/month; Enterprise custom. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceNoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Yes

Eval() experiments on datasets

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Yes

autoevals library plus custom scorers

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

Partial

trialCount reruns each input; no pass@k metric

Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

No

Score deltas and improvement/regression counts

Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Partial

Loop agent suggests prompt edits you accept

Yes

Prompt tuning plus search across 11 hosted models

Fine-tuning / LoRA

Trains model weights or adapters.

No
Partial

Text LoRA on Qwen3-8B / Llama-3.1-8B, early access; adapters can't be served yet; full fine-tuning and RLHF are previews

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

No

Only a prompt-injection-detector cookbook

Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Model endpoints

Serves a model or tuned version behind an API you call in production.

Partial

Deploys prompts and functions, plus an AI gateway; doesn't serve tuned models

Yes

OpenAI-compatible, SSE streaming, tool calls forwarded

Production tracing

Logs and traces live traffic from your app.

Yes

Tracing and online scoring

Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Yes

Hosted MCP server

Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

Partial

Enterprise only; you run the data plane, Braintrust the control plane

No

Hosted only

Where they overlap: evaluations

Both run your app or model over a dataset and grade it with LLM judges and code scorers. Braintrust's Eval() experiments, autoevals library and side-by-side experiment comparison are well built, and its free tier has unlimited users. It can rerun each input several times (trialCount) and groups the trials in the compare view, but it reports score deltas and counts of improvements and regressions, not confidence intervals or significance.

Kojen reports every rate with a 95% interval (Wilson for proportions, bootstrap for scores), and marks a run inconclusive, with no score, when more than 20% of model or judge calls fail. Prompt tuning only accepts a new prompt when a paired bootstrap test on the same cases says it beats the current one. With 1 to 10 samples per case it reports pass@k (any of k tries succeeds) and pass^k (all k succeed), the reliability measure that matters for customer-facing agents.

Where they differ: what happens after the score

Braintrust's Loop can read your logs and experiments and propose edits to prompts, scorers and datasets, which you approve. It does not train weights, generate adversarial attacks or serve models; its gateway proxies requests to providers.

Kojen's Forge runs the improvement itself: prompt tuning rewrites the system prompt against your test cases, model search tries the task on 11 hosted models, and text LoRA (early access) trains an adapter on an 8B model. Red-team simulations generate prompt-injection, jailbreak, data-exfiltration, harmful-content and PII-leak attacks with a judge from a different model family. The chosen version deploys as an OpenAI-compatible endpoint with per-prediction feedback.

Braintrust at a glance

A hosted platform for evals, tracing and prompt iteration: experiments on datasets with LLM and code scorers, online scoring of production logs, an AI gateway, and Loop, an agent that edits prompts, scorers and datasets with your approval.

Strengths

  • A mature, polished eval and tracing product, with unlimited users even on the free tier.
  • Loop works across logs, experiments, datasets and scorers in one place.
  • Gateway, environments and online scoring of production traffic in the same product.

Limits

  • No confidence intervals or significance tests, and no pass@k beyond grouping trials.
  • Measures and suggests, but doesn't train weights, red-team or serve models.
  • Self-hosting is Enterprise-only and still hybrid.

Kojen at a glance

A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.

Strengths

  • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
  • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
  • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

Limits

  • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
  • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
  • Closed source and hosted only; no self-hosting.

When to choose Braintrust, and when to choose Kojen

Choose Braintrust if…

  • You need production tracing and online scoring for an app that runs on your own stack.
  • Your team wants a polished experiment UI with unlimited seats on the free tier.
  • You want an AI gateway in front of several model providers.
  • You need Enterprise options such as a self-hosted data plane, SAML SSO or a BAA.

Choose Kojen if…

  • You want confidence intervals and pass^k on every evaluation, not just averages.
  • You need red-team and stress simulations next to your evals.
  • You want the platform to tune the prompt and pick the model, then serve it.
  • You want agents to run the whole loop over MCP or REST.

The short version: Braintrust is the better observability and experiment-tracking product; Kojen is the better choice when the goal is to make an agent measurably more reliable and ship it. Some teams will use both: Braintrust to watch production, Kojen to train and test the next version.

Frequently asked questions

How much does Braintrust cost?
As of November 2026, Braintrust's Starter plan is free with unlimited users, 1 GB of processed data, 10k scores and 14-day retention; Pro is $249 a month; Enterprise is custom. Kojen is free to start and pay as you go.
Does Braintrust show confidence intervals?
Not in its documented experiment comparison, which shows score deltas and counts of improved and regressed cases. Kojen attaches a 95% interval to every rate, so you can see when two versions overlap.
Can Braintrust red-team an agent?
We found no built-in red-teaming in Braintrust's docs beyond a prompt-injection-detector cookbook. Kojen generates adversarial attacks across five categories and scores them with a judge from a different model family.
Does Braintrust have an MCP server?
Yes, a hosted one. Kojen also has an official MCP server, listed in the MCP Registry as ai.kojen/kojen, with OAuth sign-in.
Can I self-host Braintrust or Kojen?
Braintrust offers a self-hosted data plane on Enterprise plans, with Braintrust running the control plane. Kojen is hosted only.

Keep reading

Sources

Checked November 2026.

  1. [1]Pricing. Braintrust.
  2. [2]Evaluate. Braintrust docs.
  3. [3]Advanced evaluations (trials). Braintrust docs.
  4. [4]Compare experiments. Braintrust docs.
  5. [5]Loop. Braintrust docs.
  6. [6]Deploy. Braintrust docs.
  7. [7]MCP. Braintrust docs.
  8. [8]Self-hosting. Braintrust docs.
  9. [9]Kojen for Agents: REST API & MCP Server. Kojen.
  10. [10]llms.txt. Kojen.
  11. [11]Pricing. Kojen.

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free