Skip to content

Prompt optimization

Automatic prompt optimization in the Forge

Most tasks get better faster from a better prompt or a better-suited model than from new weights. The Forge does both for real on hosted models: prompt tuning rewrites the system prompt from the cases it fails, and model search scores your task across the catalog. Each run saves a model version you can evaluate and deploy.

Tune a prompt free

Free to start · Pay as you go · No card needed

What it does

Prompt tuning that learns from failures

Each round, a proposer reads the current best prompt's failures (the inputs, the model's answers and the grader's reasons, never the expected answers) and writes rewrites. Optional few-shot demos come from training cases.

Accepted only when it really helps

A candidate is scored on a minibatch, then the rest of the training cases, and is accepted only if the 90% bootstrap interval of its gain is above zero. Rewrites that leak a known answer are rejected unseen.

Model search across 11 models

Score the task on 3–5 models from Meta, Mistral, Alibaba, Google, DeepSeek, OpenAI and Anthropic; the best by accuracy, then partial credit, then latency, wins.

Promotion you can defend

The winner becomes the project's active version only if it beats the baseline and the current version on held-out cases, by a significant McNemar test or a 10-point gain. Otherwise it's saved as a draft with the reason.

How it works

  1. 1Create a project, describe the task and add input/expected rows (or let Kojen generate cases).
  2. 2In the Forge, choose prompt tuning or model search and a base model.
  3. 3Watch rounds score in real time; runs keep going if you close the tab.
  4. 4Evaluate the saved version, then deploy it as an endpoint.

Limits, plainly

  • Up to 100 cases, 5 rounds and 4 candidates per round; at least 20 cases (or 30%) are held out.

Questions

Is prompt tuning as good as fine-tuning?
Often it's enough, and it's far cheaper to try first. Prompt optimization changes what the model is told; fine-tuning changes its weights. When a tuned prompt plateaus on your held-out cases, Text LoRA (early access) trains real weights on Qwen3-8B or Llama-3.1-8B-Instruct.
Which models can the Forge tune?
Llama 3.1 8B, Llama 3.1 70B, Llama 3.3 70B, Mistral Small, Ministral 8B, Qwen 3 32B, Gemma 4 31B, DeepSeek V3.2, GPT-4.1 mini, Gemini 2.5 Flash and Claude Haiku 5.5.
Do I need GPUs?
No. Prompt tuning and model search call hosted models through an AI gateway; GPUs are only for LoRA training.

Read more

More of Kojen

Tune a prompt free

Free to start · Pay as you go · No card needed