Most agent demos are a single good run. Someone types a request, the agent calls the right tools in the right order, and the result looks like magic. Then you run the same request eight more times and watch it take a different path, skip a policy, or quietly give up on step four.
That gap between “it worked once” and “it works every time” is the real problem in agentic AI, and it's why we built Kojen as a dojo rather than a prompt playground.
The reliability gap is measurable
The clearest public evidence comes from τ-bench, a benchmark from Sierra that simulates customer-service conversations between a language agent with real API tools and a simulated user. Instead of only asking “did it succeed?”, the authors introduced a metric called pass^k: the chance an agent succeeds on the same task in all of k independent trials. Their headline finding was that even state-of-the-art function-calling agents like GPT-4o succeeded on fewer than half the tasks, and in the retail domain pass^8 fell below 25%.[1]
Read that again with a production lens. An agent that resolves a refund correctly one time in two, and correctly eight times in a row less than one time in four, is not an agent you can hand to customers. Better prompting helps a single run. Consistency comes from something else: repetition against a fixed standard, with feedback after every attempt.
What martial artists already knew
A kata is a choreographed sequence a student repeats until it becomes reflex. The idea crossed into software decades ago. Dave Thomas's Code Kata project describes short exercises that programmers repeat, not to finish them, but to practise the act of solving them well.[2]
The research on human expertise points the same way. In their classic study of expert performance, Ericsson, Krampe and Tesch-Römer argued that elite performance tracks the amount of deliberate practice: effortful activity designed to improve a specific aspect of performance, with immediate feedback, rather than raw hours of play.[3] Models aren't violinists, but the ingredients transfer surprisingly well:
- A narrow, defined exercise. One capability at a time, such as picking the right tool, refusing out-of-scope requests, or returning valid JSON.
- A fixed standard. The same test cases and the same grader on every attempt, so progress means something.
- Immediate, specific feedback. Not just a score, but why the attempt fell short.
- Many repetitions. The point is the distribution of outcomes, not the best one.
Feedback loops work, even without new weights
You don't need a GPU cluster to benefit from this loop. Self-Refine (Madaan et al., 2023) had a model draft an answer, critique its own output, and revise it, using no extra training data and no reinforcement learning. Across seven tasks, refined outputs were preferred over one-shot generation, by about 20 percentage points on average.[4]
The lesson for agent builders is that structure beats cleverness. A loop of attempting, grading, critiquing and adjusting, run against a stable test set, reliably moves quality in a way that one heroic prompt rewrite doesn't.
How the dojo maps to Kojen
Every part of Kojen is one piece of that loop:
- Projects hold the task: a written description of what the agent must do, plus the datasets, runs and versions that belong to it.
- The Forge is where practice happens. Prompt tuning optimises a system prompt over several judge-scored rounds, and model search benchmarks 3–5 models on your task and picks the best.
- Evaluations are the fixed standard. An LLM judge grades every answer for accuracy, hallucination, bias or your own rubric, and rolls the results into a 0–100 trust score.
- Simulations are sparring: red-team attacks, stress tests and scenario runs.
- Katas are graded exercises for the people (and agents) doing the training, at four levels from beginner to master. Sensei grades each submission, and 70 or more passes.
- Sensei is the coach that sees your projects and runs and tells you what to drill next.
Start simple, then add discipline
None of this is an argument for complexity. Anthropic's guidance on building agents makes the opposite point: find the simplest solution that works, and add agentic complexity only when it earns its keep, because agents trade latency and cost for task performance.[5]
“We recommend finding the simplest solution possible, and only increasing complexity when needed.”
A dojo fits that advice. You start with a small, well-defined kata, measure it honestly, and expand the agent's scope only once its results hold up across repeated runs. If you want the full walkthrough, our training guide goes step by step from task spec to deployed endpoint.
