Prompt Tuning, Fine-Tuning & Reasoning

Get more out of a model, step by step. Write prompts that hold up, let an optimiser search for better ones, compare models fairly, and learn what LoRA, QLoRA, RLHF, DPO and 2026's reasoning models really change, and what they cost.

Lessons

1. 1. Prompting fundamentals

   System prompts, examples and step-by-step reasoning are the first tools to reach for. Each has evidence behind it and limits to know.

   9 min

2. 2. Automatic prompt optimisation

   Instead of guessing at prompt edits, let a model propose candidates and keep the ones that score best.

   8 min

3. 3. Fine-tuning efficiently with LoRA and QLoRA

   Fine-tuning every weight is expensive. LoRA trains a tiny add-on instead; QLoRA does it on a quantised model to fit on one GPU.

   9 min

4. 4. Learning from preferences: RLHF and DPO

   Some qualities are easier to compare than to write down. Preference training teaches a model which answer people prefer.

   10 min

5. 5. Reasoning models and verifiable rewards

   The biggest shift since 2024: models trained with reinforcement learning to think before they answer, and the option to spend more compute at inference time instead of on a bigger model.

   10 min

Sources

Every source cited across this course's lessons.

1. [1]Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020 · arXiv:2005.14165.
2. [2]Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024 · arXiv:2307.03172.
3. [3]Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022 · arXiv:2201.11903.
4. [4]Wang, X. et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023 · arXiv:2203.11171.
5. [5]Zhou, Y. et al. (2022). Large Language Models Are Human-Level Prompt Engineers. ICLR 2023 · arXiv:2211.01910.
6. [6]Yang, C. et al. (2023). Large Language Models as Optimizers. ICLR 2024 · arXiv:2309.03409.
7. [7]Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 · arXiv:2106.09685.
8. [8]Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023 · arXiv:2305.14314.
9. [9]Hugging Face (2025). PEFT: Parameter-Efficient Fine-Tuning documentation. huggingface.co.
10. [10]Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023 · arXiv:2305.11206.
11. [11]Christiano, P. et al. (2017). Deep reinforcement learning from human preferences. NeurIPS 2017 · arXiv:1706.03741.
12. [12]Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022 · arXiv:2203.02155.
13. [13]Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
14. [14]Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 · arXiv:2305.18290.
15. [15]Gao, L., Schulman, J. & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.
16. [16]OpenAI (2024). OpenAI o1 System Card. arXiv:2412.16720.
17. [17]DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature 645 (2025) · arXiv:2501.12948.
18. [18]Lambert, N. et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.
19. [19]Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
20. [20]Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. ICLR 2025 · arXiv:2408.03314.