Where they overlap: evaluations
Both run your app or model over a dataset and grade it with LLM judges and code scorers. Braintrust's Eval() experiments, autoevals library and side-by-side experiment comparison are well built, and its free tier has unlimited users. It can rerun each input several times (trialCount) and groups the trials in the compare view, but it reports score deltas and counts of improvements and regressions, not confidence intervals or significance.
Kojen reports every rate with a 95% interval (Wilson for proportions, bootstrap for scores), and marks a run inconclusive, with no score, when more than 20% of model or judge calls fail. Prompt tuning only accepts a new prompt when a paired bootstrap test on the same cases says it beats the current one. With 1 to 10 samples per case it reports pass@k (any of k tries succeeds) and pass^k (all k succeed), the reliability measure that matters for customer-facing agents.
Where they differ: what happens after the score
Braintrust's Loop can read your logs and experiments and propose edits to prompts, scorers and datasets, which you approve. It does not train weights, generate adversarial attacks or serve models; its gateway proxies requests to providers.
Kojen's Forge runs the improvement itself: prompt tuning rewrites the system prompt against your test cases, model search tries the task on 11 hosted models, and text LoRA (early access) trains an adapter on an 8B model. Red-team simulations generate prompt-injection, jailbreak, data-exfiltration, harmful-content and PII-leak attacks with a judge from a different model family. The chosen version deploys as an OpenAI-compatible endpoint with per-prediction feedback.
Braintrust at a glance
A hosted platform for evals, tracing and prompt iteration: experiments on datasets with LLM and code scorers, online scoring of production logs, an AI gateway, and Loop, an agent that edits prompts, scorers and datasets with your approval.
- A mature, polished eval and tracing product, with unlimited users even on the free tier.
- Loop works across logs, experiments, datasets and scorers in one place.
- Gateway, environments and online scoring of production traffic in the same product.
- No confidence intervals or significance tests, and no pass@k beyond grouping trials.
- Measures and suggests, but doesn't train weights, red-team or serve models.
- Self-hosting is Enterprise-only and still hybrid.
Kojen at a glance
A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.
- Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
- Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
- Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.
- Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
- No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
- Closed source and hosted only; no self-hosting.