Skip to content

Kojen vs · Fine-tuning

Kojen vs OpenAI fine-tuning

OpenAI's fine-tuning API has been closed to new organizations since May 7, 2026, and existing customers can't create new fine-tuning jobs after January 6, 2027, so for most teams the real question is what to use instead. Kojen is one option: it tunes the system prompt and picks the best of 11 hosted models (including GPT-4.1 mini) against your test cases, evaluates with confidence intervals, red-teams, and serves the result behind an OpenAI-compatible endpoint, with LoRA training on open 8B models in early access.

Kojen can't fine-tune OpenAI's models, and nothing outside OpenAI can.

By Will SchulzUpdated 7 sources

Side-by-side comparison

"Partial" means the feature exists with real limits, given in the note. "Not verified" means we couldn't confirm it from the vendor's own material.

Kojen vs OpenAI fine-tuning: features and pricing, checked November 2026
Checked Nov 15, 2026OpenAI fine-tuningKojen
What it isFine-tuning

OpenAI

Training dojo

Kojen (that's us)

PricingClosed to new users. Existing customers: training from $3 (gpt-4o-mini) to $25 (gpt-4.1, gpt-4o) per 1M tokens, plus inference on the tuned model. SourceFree to start, no card needed; then pay as you go. GPU training (LoRA) is paid in credits billed by the second at the GPU's live price. Team and volume pricing by email. Source
Open sourceNoNo
Evaluations on datasets

Run a model or app over a set of test cases and score the results.

Partial

Hosted Evals went read-only and shuts down Nov 30, 2026

Yes

Test cases scored for accuracy, hallucination, bias and latency

LLM-as-judge scoring

A model grades outputs against a rubric or reference.

Partial

Model graders, tied to the retiring Evals platform

Yes

Judge kept out of the target's model family in simulations

pass@k / pass^k

Several samples per case, reported as any-success (pass@k) or all-success (pass^k).

Not verified
Yes

1–10 samples per case

Confidence intervals

Rates reported with an interval or a significance test, not just a point estimate.

Not verified
Yes

Wilson and bootstrap intervals on every rate

Prompt optimization

Automatically rewrites instructions or examples to raise a metric.

Partial

Dataset-backed prompt optimizer is being retired with Evals

Yes

Prompt tuning plus search across 11 hosted models

Fine-tuning / LoRA

Trains model weights or adapters.

Partial

SFT and DPO on gpt-4.1 family; existing customers only, until Jan 6, 2027

Partial

Text LoRA on Qwen3-8B / Llama-3.1-8B, early access; adapters can't be served yet; full fine-tuning and RLHF are previews

Red-teaming

Generates adversarial attacks (prompt injection, jailbreaks, data leaks) and scores them.

No
Yes

Prompt injection, jailbreak, data exfiltration, harmful content, PII leak

Model endpoints

Serves a model or tuned version behind an API you call in production.

Yes

Tuned models served by OpenAI until the base model retires

Yes

OpenAI-compatible, SSE streaming, tool calls forwarded

Production tracing

Logs and traces live traffic from your app.

Partial

Platform logs and trace grading

Partial

Request logs, latency, errors and feedback for Kojen endpoints only; no tracing SDK

Official MCP server

An MCP server from the vendor that lets agents drive the product.

Not verified
Yes

ai.kojen/kojen in the MCP Registry, OAuth sign-in

Self-host

You can run it on your own infrastructure.

No

No weight export

No

Hosted only

What OpenAI changed

OpenAI's deprecations page sets the timeline: fine-tuning job creation closed to organizations that had never fine-tuned on May 7, 2026; to those with no fine-tuned-model inference in the previous 60 days on July 2, 2026; and to active existing customers on January 6, 2027. Inference on existing tuned models continues until each underlying base model is deprecated, and tuned weights can't be exported.

The hosted Evals platform that paired with fine-tuning went read-only on October 31, 2026 and shuts down on November 30, 2026, with OpenAI recommending Promptfoo instead. Its dataset-backed prompt optimizer goes with it.

A migration path in Kojen

Many fine-tunes exist to make a model follow a format or a policy reliably, which prompt tuning can often do. Kojen's Forge rewrites the system prompt against your test cases and searches 11 hosted models for the cheapest one that passes, then reports each candidate's accuracy with a confidence interval and pass^k over several samples, so you can check the replacement is genuinely as good as the fine-tune.

Deployments are OpenAI-compatible chat-completions endpoints with real streaming, tool calls and response_format forwarded, so switching usually means changing the base URL and key. If you need trained weights, text LoRA on Qwen3-8B or Llama-3.1-8B-Instruct is available in early access, though those adapters can't be served from Kojen yet. Together AI, Vertex AI and Bedrock also offer managed fine-tuning of other models.

OpenAI fine-tuning at a glance

Managed supervised (SFT), preference (DPO) and vision fine-tuning of OpenAI's own models, served by OpenAI. It is being wound down: closed to new organizations since May 7, 2026, and existing customers can't create new jobs after January 6, 2027.

Strengths

  • Tunes OpenAI's own closed models, with nothing to run yourself.
  • Simple for teams already on the OpenAI API.
  • Tuned models keep serving until their base model is deprecated.

Limits

  • Closed to new organizations, and job creation ends for everyone on January 6, 2027.
  • No weight export: a tuned model disappears when its base model is retired.
  • The surrounding Evals platform shuts down on November 30, 2026.

Kojen at a glance

A hosted dojo for AI agents: LLM-judged evaluations with confidence intervals and pass@k / pass^k, red-team and stress simulations, prompt tuning and model search, LoRA training in early access, and OpenAI-compatible endpoints, all driveable over MCP and REST.

Strengths

  • Measures and improves in one loop: evaluations, red-teaming, prompt tuning, model search and deployment share the same project and test cases.
  • Honest statistics by default: every rate has a confidence interval, tuned prompts must beat the current one in a paired test, and pass@k / pass^k come from several samples per case.
  • Agent-native: the full loop is exposed over an MCP server and a REST API with an OpenAPI spec, with the same 24 tools as the in-app agent.

Limits

  • Young product: weight training is early access on 8B models, LoRA adapters can't be served yet, and full fine-tuning and RLHF are previews.
  • No tracing SDK for apps running elsewhere, so it is not a replacement for an observability platform.
  • Closed source and hosted only; no self-hosting.

When to choose OpenAI fine-tuning, and when to choose Kojen

Choose OpenAI fine-tuning if…

  • You are an active existing customer and need to retrain an OpenAI model before January 6, 2027.
  • Your product depends on a tuned OpenAI model that keeps serving until its base model retires.
  • You need a tuned closed frontier model specifically.

Choose Kojen if…

  • You can't create OpenAI fine-tuning jobs any more and need a working replacement.
  • Your fine-tune mainly enforced a format or policy that prompt tuning may match, and you want to measure that.
  • You want an OpenAI-compatible endpoint, so client code barely changes.
  • You want evaluations and red-teaming to replace OpenAI's retiring Evals.

The short version: For existing customers, OpenAI fine-tuning still works until January 6, 2027, and tuned models keep serving after that. For everyone else it is no longer an option. Kojen is a practical replacement when prompt tuning and model choice can close the gap, and it gives you the evidence to know whether they did.

Frequently asked questions

Is OpenAI fine-tuning being discontinued?
Job creation is. OpenAI's deprecations page says organizations that had never fine-tuned lost access on May 7, 2026, and active existing customers can't create new jobs after January 6, 2027. Inference on existing tuned models continues until the base model is deprecated.
What did OpenAI fine-tuning cost?
For existing customers as of November 2026, training costs $3 per million tokens for gpt-4o-mini, $5 for gpt-4.1-mini and $25 for gpt-4.1 or gpt-4o, plus inference on the tuned model.
Can I export my OpenAI fine-tuned model?
No. OpenAI doesn't offer weight downloads, which is why a tuned model disappears when its base model is retired. Keep your training data so you can rebuild elsewhere.
What replaces OpenAI Evals?
OpenAI recommends Promptfoo. Kojen also runs LLM-judged evaluations with confidence intervals, plus red-team and stress simulations, on hosted models and endpoints.
Is Kojen's endpoint a drop-in replacement for the OpenAI API?
For chat completions, largely yes: Kojen deployments expose /api/v1/endpoints/{deployment}/chat/completions with SSE streaming, and forward tools, tool_calls and response_format to the model.

Keep reading

Sources

Checked November 2026.

  1. [1]Deprecations. OpenAI docs.
  2. [2]Pricing. OpenAI docs.
  3. [3]Model optimization. OpenAI docs.
  4. [4]Kojen for Agents: REST API & MCP Server. Kojen.
  5. [5]llms.txt. Kojen.
  6. [6]Pricing. Kojen.
  7. [7]Moving from OpenAI Evals to Promptfoo. OpenAI Cookbook.

Try it on your own test cases.

Run an LLM-judged evaluation with confidence intervals, a red-team pass and a prompt-tuning run on your agent. Free to start, no card needed, then pay as you go.

Start training free