AI Training Foundations
Lesson 2 of 4Beginner 8 min

Define the task and its success criteria

A model can only get better at something you can measure. Turn an idea into a task spec with inputs, outputs and a score.

In this lesson you'll

  • Write a task spec with clear inputs and outputs
  • Pick metrics before you pick a model
  • Recognise when you don't need machine learning at all

Google's Rules of Machine Learning opens with two rules that apply just as well to LLMs: don't be afraid to launch without machine learning, and design and implement metrics first [1]. If a regex, a lookup table or a short checklist does the job, ship that. If it doesn't, you need a way to tell good output from bad before any training starts.

The task spec

  1. Job: one sentence. "Classify inbound support tickets into billing, bug, feature request or other."
  2. Input: what the model receives, and its format.
  3. Output: the exact shape you want back, such as one label, JSON with fixed keys, or a reply under 120 words.
  4. Success criteria: how you'll score an answer, and the score that counts as good enough to ship.
  5. Constraints: latency budget, cost ceiling, what the model must never do.

Accuracy alone is rarely enough. The HELM benchmark makes this point by scoring every model on several metrics at once, including accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, because a model that wins on one often loses on another [2]. Write down which of those matter for your task.

In Kojen

Put the spec in the project description. Training and evaluation runs use it to generate test cases when you haven't uploaded a dataset, so a precise description directly improves your runs.

Key takeaways

  • Decide how you'll score an answer before deciding how to produce one.
  • Specify the output format exactly; it makes grading possible.
  • Track more than one metric, and agree on the ship threshold up front.

Practise in the dojo

Create a project and write its description as a task spec: job, input, output, success criteria and constraints.

Create a project

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. According to Google's Rules of ML, what should come before building the model?
2. Why does HELM score models on several metrics?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]Zinkevich, M. (2017). Rules of Machine Learning: Best Practices for ML Engineering. Google for Developers.
  2. [2]Liang, P. et al. (2022). Holistic Evaluation of Language Models. TMLR 2023 · arXiv:2211.09110.