AI Training Foundations
Lesson 3 of 4Beginner 7 min

Choose a base model

Bigger is usually more capable and always slower and dearer. Pick on measured accuracy against your latency and cost budget.

In this lesson you'll

  • Weigh capability against latency and cost
  • Run a fair head-to-head on your own task
  • Avoid choosing a model from leaderboards alone

Scaling laws tell us that larger models trained on more data reach lower loss [1]. That's a statement about averages over broad text, not a guarantee about your task. A small model with a good prompt often matches a large one on narrow, well-specified work, at a fraction of the latency.

A fair comparison

  1. Shortlist three to five models across sizes and providers.
  2. Use the same prompt and the same test set for each.
  3. Record accuracy, p95 latency and cost per thousand calls.
  4. Pick the cheapest model that clears your ship threshold, then use latency as the tie-break.

Public benchmarks are a useful filter, but HELM found that model rankings change from scenario to scenario and metric to metric [2]. Your own test set is the only benchmark that measures your task.

Routing calls through a gateway makes this easy, because switching providers becomes a change of model string rather than a new integration [3].

In Kojen

Model search in the Forge does this comparison for you: it scores your task on 3–5 models and picks the best by accuracy, with latency as the tie-break. Llama 3.1 8B is the default base model.

Key takeaways

  • Benchmark on your task, not someone else's leaderboard.
  • Hold the prompt and test set fixed when comparing models.
  • Choose the smallest model that clears the bar.

Practise in the dojo

In the Forge, start a model search on your project and compare the accuracy and latency of each candidate.

Run model search

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. What's the fairest way to compare candidate models?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
  2. [2]Liang, P. et al. (2022). Holistic Evaluation of Language Models. TMLR 2023 · arXiv:2211.09110.
  3. [3]Vercel (2025). AI Gateway documentation. vercel.com/docs.