Curate data that teaches
A thousand carefully chosen examples can do more than a hundred thousand scraped ones. Clean, cover the edge cases, hold some back, and write it down.
In this lesson you'll
- Prefer quality and coverage over volume
- Split data so evaluation stays honest
- Document a dataset so others can trust it
The LIMA study fine-tuned a 65B-parameter LLaMA model on only 1,000 carefully curated prompts and responses, with no preference training, and found its answers were equivalent to or preferred over GPT-4's in 43% of cases [1]. The authors' conclusion: most of a model's knowledge comes from pretraining, and a small, high-quality set is enough to teach it the format and style you want.
What good data looks like
- Correct: every expected answer is one you'd sign off on.
- Consistent: the same kind of input always gets the same kind of output.
- Covering: easy cases, hard cases, edge cases and inputs the model should refuse.
- Representative: drawn from the inputs you'll actually see in production.
Hold some back
Never score a model on examples it was tuned on. Keep a held-out test split that no training or prompt-tuning step sees, and only look at it to decide whether to ship [2].
Write it down
Datasheets for Datasets proposes that every dataset ship with answers to a standard set of questions: why it was created, what it contains, how it was collected and labelled, and what it should and shouldn't be used for [3]. Even a short version of this prevents a lot of misuse later.
In Kojen
Upload CSV, JSON or JSONL on Datasets. To use one as a test set, give it an input column (such as input or prompt) and an expected-answer column (such as expected or output). Forge runs hold out 30% for a final check automatically.
Key takeaways
- Quality and coverage beat volume.
- Keep a held-out test set that training never touches.
- Document where the data came from and what it's for.
Upload a dataset with input and expected columns, then try the Data cleaner agent to produce a cleaned copy.
Check your understanding
Score 70 or more to complete the lesson. Progress is saved in this browser.
Further reading
Sources
Checked against the original sources in November 2026.
- Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023 · arXiv:2305.11206.
- Zinkevich, M. (2017). Rules of Machine Learning: Best Practices for ML Engineering. Google for Developers.
- Gebru, T. et al. (2018). Datasheets for Datasets. Communications of the ACM · arXiv:1803.09010.