Skip to content

Kojen vs · Prompt optimization

Kojen vs DSPy

DSPy is a free, open-source Python framework for writing LLM pipelines as code and optimizing their prompts, few-shot examples and even weights against a metric you define; Kojen is a hosted service that tunes a model's instructions and picks the best of 11 hosted models, then evaluates, red-teams and deploys the result without you writing an optimizer. Choose DSPy if you want maximum control inside your own Python codebase; choose Kojen if you want the optimize-evaluate-ship loop run for you, with confidence intervals and an endpoint at the end.

They can work together for single-prompt tasks: an instruction DSPy optimized can be pasted into Kojen, evaluated with confidence intervals and red-teamed on a hosted model.

By Will SchulzUpdated 11 sources

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

Kojen vs DSPy: features and pricing, checked November 2026
Checked Nov 15, 2026DSPyKojen
What it isPrompt optimization

Open-source project started at Stanford NLP

Training dojo

Kojen (that's us)

PricingFree, MIT-licensed library. You pay your own model and GPU costs. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceYes, MIT RepoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Yes

dspy.Evaluate over a devset with your metric

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Partial

Built-in LLM metrics (SemanticF1, CompleteAndGrounded); otherwise you write the judge

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

No

Not built in; you can script repeated runs

Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

No

Reports a single score

Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Yes

GEPA, MIPROv2, COPRO, SIMBA, Bootstrap few-shot and more

Yes

Prompt tuning plus search across 11 hosted models

Fine-tuning / LoRA

Trains model weights or adapters.

Partial

BootstrapFinetune; RL via Arbor is labelled experimental; bring your own GPUs

Partial

Text LoRA on Qwen3-8B / Llama-3.1-8B, early access; adapters can't be served yet; full fine-tuning and RLHF are previews

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

No
Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Model endpoints

Serves a model or tuned version behind an API you call in production.

No

Library; deploy in your own code

Yes

OpenAI-compatible, SSE streaming, tool calls forwarded

Production tracing

Logs and traces live traffic from your app.

Partial

Through the MLflow integration

Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Partial

Uses MCP tools inside programs; no DSPy MCP server

Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

Yes

Runs wherever your Python runs

No

Hosted only

The core difference: a library versus a loop

DSPy asks you to restructure your pipeline into signatures and modules, write a metric function, and run an optimizer such as GEPA or MIPROv2 over a training set. In return it can optimize multi-step programs end to end, which no hosted tool matches today. GEPA, its reflective optimizer, reported beating GRPO reinforcement learning by 6% on average while using up to 35 times fewer rollouts.

Kojen works one level up. You bring a task description and test cases; the Forge's prompt tuning rewrites the system prompt, scored by an LLM judge on those cases, and model search tries the same task across 11 hosted models to find the cheapest one that holds up. The winning version is saved, can be evaluated with confidence intervals and several samples per case (pass@k and pass^k), red-teamed for prompt injection and data exfiltration, and served behind an OpenAI-compatible endpoint.

Weight training

DSPy's BootstrapFinetune distils a program's traces into fine-tuned weights, and its GRPO path via Arbor is labelled experimental; both run on GPUs you provide. Kojen's text LoRA trains a real adapter on Qwen3-8B or Llama-3.1-8B-Instruct on rented GPUs, paid in credits for the time used, but it is early access for enabled accounts and the adapters can't serve evaluations or deployments yet. Neither is a full managed fine-tuning service.

DSPy at a glance

A Python framework for programming LLM pipelines as modules and optimizing them against a metric you write. Its optimizers rewrite instructions (MIPROv2, COPRO, GEPA, SIMBA), pick few-shot demos (the Bootstrap family) and can fine-tune weights (BootstrapFinetune, experimental GRPO via Arbor). Version 3.5.0 shipped on November 10, 2026.

Strengths

  • The widest set of optimizers anywhere, covering instructions, few-shot demos and weights.
  • Optimizes whole multi-step programs, not one prompt at a time.
  • Strong research lineage: GEPA, now a built-in optimizer, was an ICLR 2026 oral.

Limits

  • A real learning curve: you restructure code into signatures and modules and write your own metrics.
  • No hosted UI, managed training or serving; RL needs your own GPUs and is experimental.
  • Defaults change between releases (3.5.0 changed the default adapter), so optimized programs need re-evaluating.

Kojen at a glance

A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.

Strengths

  • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
  • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
  • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

Limits

  • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
  • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
  • Closed source and hosted only; no self-hosting.

When to choose DSPy, and when to choose Kojen

Choose DSPy if…

  • Your pipeline is multi-step (retrieval, tools, several LLM calls) and you want to optimize it as a whole.
  • You are comfortable in Python and want full control over the metric, the optimizer and the data.
  • You need a free, MIT-licensed tool that runs entirely on your own infrastructure.
  • You want to experiment with the newest research optimizers as they land.

Choose Kojen if…

  • You want prompt tuning and model selection without writing an optimizer or a metric function.
  • You need to know whether the improvement is real: rates with confidence intervals, a paired test before a tuned prompt is accepted, pass^k for reliability.
  • You want red-team and stress tests and a production endpoint in the same place as the tuning.
  • Your agents should drive the loop themselves over MCP or REST.

The short version: DSPy is the stronger optimizer for engineers who will write code around it; Kojen is the faster path from a task and test cases to a tested, deployed model. Kojen does not run multi-module DSPy programs, so for those DSPy plus your own evaluation code remains the better fit.

Frequently asked questions

Is DSPy free?
Yes. DSPy is MIT-licensed open source. You pay only for the model calls and any GPUs its optimizers use.
Does Kojen use DSPy?
No. Kojen's prompt tuning is its own hosted loop: it rewrites the system prompt, scores each candidate with an LLM judge on your test cases and keeps the best, saving it as a model version.
Can DSPy fine-tune models?
Yes, partly. BootstrapFinetune fine-tunes the models inside a program from its own traces, and an RL path through Arbor exists but its own docs call it experimental. You supply the GPUs.
What do DSPy's optimizers change?
Instructions (MIPROv2, COPRO, GEPA, SIMBA), few-shot demonstrations (BootstrapFewShot and its variants, LabeledFewShot, KNNFewShot) or weights (BootstrapFinetune). GEPA stopped being experimental in DSPy 3.5.0, released November 10, 2026.
Can I evaluate a DSPy-optimized prompt in Kojen?
Partly. Kojen evaluates its hosted models with a system prompt, so you can paste a DSPy-optimized instruction into a Kojen project and evaluate and red-team it on one of the 11 hosted models. It can't run a multi-module DSPy program or call an endpoint hosted elsewhere.

Keep reading

Sources

Checked November 2026.

  1. [1]stanfordnlp/dspy. GitHub.
  2. [2]DSPy 3.5.0 release notes. GitHub.
  3. [3]dspy.Evaluate. DSPy docs.
  4. [4]BootstrapFinetune. DSPy docs.
  5. [5]Tutorial: Online RL over a multi-module DSPy program. DSPy docs.
  6. [6]Use MCP in DSPy. DSPy docs.
  7. [7]GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv (Agrawal et al.).
  8. [8]Kojen for Agents: REST API & MCP Server. Kojen.
  9. [9]llms.txt. Kojen.
  10. [10]Pricing. Kojen.
  11. [11]Large Language Models as Optimizers (OPRO). arXiv (Yang et al., ICLR 2024).

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free