Skip to content

Kojen vs · Evaluation

Kojen vs LangSmith

LangSmith is LangChain's hosted platform for tracing, evaluating, managing prompts for and deploying agents, and it is the natural fit for teams building on LangChain or LangGraph; Kojen is a training dojo that evaluates a model or agent with confidence intervals and pass^k, red-teams it, tunes its prompt and model choice, and serves it behind an OpenAI-compatible endpoint. Pick LangSmith to observe and host agents built in code; pick Kojen to make an agent measurably more reliable before and after it ships.

The products deploy different things: LangSmith hosts agent code, Kojen serves models and tuned versions.

By Will SchulzUpdated 11 sources

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

Kojen vs LangSmith: features and pricing, checked November 2026
Checked Nov 15, 2026LangSmithKojen
What it isEvaluation & observability

LangChain (independent)

Training dojo

Kojen (that's us)

PricingDeveloper $0 (1 seat, 5k base traces/month); Plus $39/seat/month (10k traces); Enterprise custom. Usage beyond that is pay as you go. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceNoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Yes

evaluate() on datasets

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Yes

LLM evaluators from the UI or SDK

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

Partial

num_repetitions with mean and standard deviation; no pass@k

Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

No

Standard deviation only

Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Partial

Optimization jobs (promptim) via API; Polly edits prompts on request

Yes

Prompt tuning plus search across 11 hosted models

Fine-tuning / LoRA

Trains model weights or adapters.

No
Partial

Text LoRA on Qwen3-8B / Llama-3.1-8B, early access; adapters can't be served yet; full fine-tuning and RLHF are previews

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

No
Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Model endpoints

Serves a model or tuned version behind an API you call in production.

Partial

Hosts agents (Agent Server); doesn't serve models

Yes

OpenAI-compatible, SSE streaming, tool calls forwarded

Production tracing

Logs and traces live traffic from your app.

Yes
Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Yes

Python server and hosted remote MCP with OAuth

Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

Partial

Enterprise add-on

No

Hosted only

Evaluation and statistics

LangSmith's evaluate() runs datasets through your app with LLM or code evaluators, and its num_repetitions option reruns each example and shows the mean and standard deviation. Annotation queues let people label results. It does not report confidence intervals, significance tests or pass^k.

Kojen reports every rate with a 95% interval, accepts a tuned prompt only when a paired bootstrap test says it beats the current one, and turns several samples per case into pass@k and pass^k. For a customer-facing agent, pass^k (succeeding on every try) is usually the number that matters.

Improvement and deployment

LangSmith offers prompt optimization jobs through its API, using the experimental promptim algorithm, and Polly, an assistant that edits prompts on request. It hosts agents on its Agent Server, which needs the Plus plan or above, but it does not serve or train models.

Kojen tunes the system prompt and searches 11 hosted models in the Forge, trains LoRA adapters on 8B models in early access, red-teams the result, and deploys model versions as OpenAI-compatible endpoints with request logs, latency, error rates and per-prediction feedback that can trigger a new tuning run.

LangSmith at a glance

LangChain's hosted platform for tracing, evaluations, prompt management and agent deployment. It works with any framework, and hosts LangGraph and other agents through LangSmith Deployment.

Strengths

  • The natural choice for LangChain and LangGraph teams, with production hosting for agents.
  • Strong tracing, annotation queues and prompt management.
  • Cloud, hybrid and self-hosted deployment options.

Limits

  • Per-seat pricing, and the free tier has one seat.
  • Reliability stats stop at mean and standard deviation; no intervals or pass^k.
  • No red-teaming or weight training.

Kojen at a glance

A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.

Strengths

  • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
  • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
  • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

Limits

  • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
  • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
  • Closed source and hosted only; no self-hosting.

When to choose LangSmith, and when to choose Kojen

Choose LangSmith if…

  • Your agents are built with LangChain or LangGraph and you want tracing that understands them.
  • You want to host agent code (not just a model) in production.
  • You need annotation queues for human review at scale.
  • You need hybrid or self-hosted deployment on an Enterprise plan.

Choose Kojen if…

  • You want statistically honest results: intervals, paired tests in tuning and pass^k, not just averages.
  • You need red-teaming and stress tests in the same place as evaluation.
  • You want automatic prompt tuning and model selection that runs for you.
  • You want a model endpoint any OpenAI-compatible client can call.

The short version: LangSmith is stronger for tracing and hosting code-defined agents, especially in the LangChain ecosystem. Kojen is stronger when you need to prove an agent got better, attack it before users do, and ship the improved model.

Frequently asked questions

How much does LangSmith cost?
As of November 2026: Developer is free for one seat with 5k base traces a month, Plus is $39 per seat per month with 10k traces, and Enterprise is custom; usage beyond the allowance is pay as you go.
Does LangSmith work without LangChain?
Yes. LangSmith traces and evaluates apps built with any framework, though its agent hosting is built around LangGraph-style agents.
Does LangSmith calculate pass@k?
Not as a metric. Repetitions show a mean and standard deviation per example. Kojen computes pass@k and pass^k from 1 to 10 samples per case.
Can LangSmith optimize prompts automatically?
Partly: it exposes prompt optimization jobs through its API with the promptim algorithm, which its author describes as experimental, and Polly edits prompts when you ask. Kojen's prompt tuning runs as a managed training run scored on your test cases.
Do both have MCP servers?
Yes. LangSmith has a Python MCP server and a hosted remote one with OAuth; Kojen's MCP server is listed in the official MCP Registry as ai.kojen/kojen.

Keep reading

Sources

Checked November 2026.

  1. [1]Pricing. LangChain.
  2. [2]Repetition. LangSmith docs.
  3. [3]Run an evaluation from the prompt playground. LangSmith docs.
  4. [4]Create optimization job. LangSmith API docs.
  5. [5]Polly. LangSmith docs.
  6. [6]Agent Server. LangSmith docs.
  7. [7]LangSmith MCP server. LangSmith docs.
  8. [8]Self-hosted LangSmith. LangSmith docs.
  9. [9]Kojen for Agents: REST API & MCP Server. Kojen.
  10. [10]llms.txt. Kojen.
  11. [11]Pricing. Kojen.

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free