LLM evaluations you can trust before you ship
A Kojen evaluation runs the model under test on your test cases, has an LLM judge grade every answer, and measures latency on every real call. You get per-case verdicts with the judge's reasons, bootstrap confidence intervals and a 0–100 trust score, so you know whether a change actually helped.
What it does
Five kinds of check
Accuracy (correct vs expected, exact match first, then the judge with partial credit), hallucination (unsupported or fabricated claims, grounded on the expected answer or context), bias (paired prompts that differ only in a demographic attribute, judged in both orders), custom (your rubric, scored 0–10, pass at 7) and comprehensive (accuracy, hallucination and bias together).
A judge from another model family
The judge is never from the same family as the model it grades, so a model doesn't mark its own homework. Generated test cases come from a third family.
Honest numbers
Errors count as failures, every score carries a confidence interval, and if more than 20% of calls fail the result is marked inconclusive instead of getting a trust score. Sample several answers per case to get pass@k and pass^k.
Clean test sets
Only a dataset's held-out test split is graded, minus any case the model version was trained or selected on. No dataset? Cases are generated from the project's task and saved, so reruns use the same ones.
Anything you can call
Evaluate a base model with a system prompt, a model version from the Forge, the result of a training run, or a live deployment. Compare puts runs side by side.
Latency included
p50, p95 and p99 latency are measured on the real calls, alongside the trust score (comprehensive weights accuracy 50%, hallucination 30% and bias 20%).
How it works
- 1Create a project and describe the task, or start from a template.
- 2Upload a CSV, JSON or JSONL test set with input and expected columns, or let Kojen generate cases.
- 3Pick the check and the model, version or deployment to test, then run it.
- 4Read the per-case verdicts, fix what failed in the Forge, and run it again.
Limits, plainly
- Default 30 cases per evaluation, up to 100; up to 10 samples per case and 300 model calls per run.
- Model calls count toward a daily per-user token allowance.
Questions
- What is an LLM-as-a-judge evaluation?
- A second model reads each answer and decides whether it meets the bar: correct against the expected answer, free of unsupported claims, or passing your rubric. Kojen's judge gives a verdict and a reason for every case and is always from a different model family than the model under test.
- Do I need a labelled dataset?
- No. With a dataset Kojen grades its held-out test split; without one it generates test cases from your project's task and saves them, so later runs are comparable. Labelled cases give the most trustworthy accuracy numbers.
- Can I evaluate a model that's already deployed?
- Yes. Point the evaluation at a deployment and it calls that deployment like any client would, so the result reflects what's actually serving.