Evaluation and statistics
LangSmith's evaluate() runs datasets through your app with LLM or code evaluators, and its num_repetitions option reruns each example and shows the mean and standard deviation. Annotation queues let people label results. It does not report confidence intervals, significance tests or pass^k.
Kojen reports every rate with a 95% interval, accepts a tuned prompt only when a paired bootstrap test says it beats the current one, and turns several samples per case into pass@k and pass^k. For a customer-facing agent, pass^k (succeeding on every try) is usually the number that matters.
Improvement and deployment
LangSmith offers prompt optimization jobs through its API, using the experimental promptim algorithm, and Polly, an assistant that edits prompts on request. It hosts agents on its Agent Server, which needs the Plus plan or above, but it does not serve or train models.
Kojen tunes the system prompt and searches 11 hosted models in the Forge, trains LoRA adapters on 8B models in early access, red-teams the result, and deploys model versions as OpenAI-compatible endpoints with request logs, latency, error rates and per-prediction feedback that can trigger a new tuning run.
LangSmith at a glance
LangChain's hosted platform for tracing, evaluations, prompt management and agent deployment. It works with any framework, and hosts LangGraph and other agents through LangSmith Deployment.
- The natural choice for LangChain and LangGraph teams, with production hosting for agents.
- Strong tracing, annotation queues and prompt management.
- Cloud, hybrid and self-hosted deployment options.
- Per-seat pricing, and the free tier has one seat.
- Reliability stats stop at mean and standard deviation; no intervals or pass^k.
- No red-teaming or weight training.
Kojen at a glance
A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.
- Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
- Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
- Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.
- Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
- No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
- Closed source and hosted only; no self-hosting.