Prompt Tuning, Fine-Tuning & Reasoning
Lesson 5 of 5Intermediate 10 min

Reasoning models and verifiable rewards

The biggest shift since 2024: models trained with reinforcement learning to think before they answer, and the option to spend more compute at inference time instead of on a bigger model.

In this lesson you'll

  • Explain what makes a reasoning model different
  • Describe reinforcement learning with verifiable rewards (RLVR) and GRPO
  • Decide when test-time compute is worth its latency

Chain-of-thought started as a prompting trick [1]. Reasoning models make it a trained behaviour. OpenAI's o1 was trained with large-scale reinforcement learning to reason with a chain of thought before answering [2], and DeepSeek-R1 showed that reasoning behaviours such as self-verification and reflection can emerge from reinforcement learning alone, without supervised reasoning demonstrations first [3]. By 2026 most frontier and many open model families ship a reasoning or "thinking" mode.

Rewards you can check

Preference rewards come from a learned model that can be gamed [4]. For tasks with a checkable answer, such as maths, code that must pass tests, or output that must match a schema, you can reward the model with a program instead. Tülu 3 named this reinforcement learning with verifiable rewards (RLVR) [5], and DeepSeek-R1 used rule-based accuracy and format rewards [3]. Both build on GRPO, an RL algorithm from DeepSeekMath that scores each answer against a group of sampled answers to the same prompt, removing the need for a separate value model [6].

Design takeaway

The more of your task you can verify with code (exact labels, valid JSON, passing tests), the cheaper and more trustworthy both training and evaluation become. Write verifiers first.

Test-time compute

Reasoning models let you trade latency for accuracy at inference time. Snell et al. found that allocating test-time compute adaptively per prompt was over 4× more efficient than best-of-N sampling, and that on problems where a smaller model already has some success, extra test-time compute can beat a model 14× larger [7]. Self-consistency is the simplest version of the same idea [8].

  • Use reasoning for multi-step problems: planning, maths, code, tricky policy decisions.
  • Skip it for classification, extraction and short rewrites, where it mostly adds latency and cost.
  • Measure both ways. Run the same evaluation with reasoning on and off, and compare accuracy against p95 latency.

In Kojen

Kojen doesn't train models with RL today. You can still apply the lesson: give tasks exact expected answers so evaluations grade them reliably, and use Compare to weigh a larger or reasoning-capable model against a smaller one on accuracy and latency.

Key takeaways

  • Reasoning models are trained with RL to think before answering.
  • Verifiable rewards (RLVR) avoid much of the reward hacking of learned reward models.
  • Test-time compute is a dial; turn it up only where the task needs it.

Practise in the dojo

Put a small and a larger model side by side on the same evaluation and decide whether the accuracy gain is worth the latency.

Compare models

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. What makes a reward "verifiable" in RLVR?
2. What does GRPO avoid needing?
3. When is extra test-time reasoning least worth it?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022 · arXiv:2201.11903.
  2. [2]OpenAI (2024). OpenAI o1 System Card. arXiv:2412.16720.
  3. [3]DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature 645 (2025) · arXiv:2501.12948.
  4. [4]Gao, L., Schulman, J. & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.
  5. [5]Lambert, N. et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.
  6. [6]Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
  7. [7]Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. ICLR 2025 · arXiv:2408.03314.
  8. [8]Wang, X. et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023 · arXiv:2203.11171.