Say you have $10,000 to spend on a computer for training language models. From scratch, you can realistically build a model of about 1 billion parameters, roughly GPT-2 class. It will write fluent text, but it won't reason well or know many facts.
Fine-tuning is where that machine shines. The same box can adapt open models with tens of billions of parameters, and the result will usually beat anything you could train yourself.
What $10,000 buys in late 2026
Less than it did a year ago. On 3 November 2026, Tom's Hardware's tracker showed a best US price of $6,919 for an RTX 5090, against an MSRP of $1,999.[1] Add the rest of a capable workstation and we estimate you're close to $10,000 with one card, so a dual-GPU build no longer fits this budget. These prices are moving quickly, so check them before you buy.
| Option | Memory | Best for |
|---|---|---|
| RTX 5090 workstation[2] | 32 GB | Training speed and the widest software support |
| Mac Studio M5 Ultra, from $5,499[3] | 96 GB | Running large models locally |
| DGX Spark[4] | 128 GB | Fitting large models, with 273 GB/s of memory bandwidth limiting speed |
The 5090 has the least memory of the three, but it's the best fit for training. Most training tools are built around NVIDIA's CUDA, which counts against the Mac, and its 1,792 GB/s of memory bandwidth[2] is more than six times the DGX Spark's. The rest of this article assumes that build.
Training from scratch
Compute for a training run is about 6 × parameters × tokens.[5] The 5090's dense BF16 peak is 209.5 TFLOPS.[2] If you reach 40% of peak, which is in line with what Meta reported on H100s,[6] that's about 84 TFLOPS. Consumer hardware may do better or worse depending on your code, so these are estimates:
| Model | Tokens | Time on one 5090 | What you get |
|---|---|---|---|
| 125M | 10B | about 1 day | Coherent short text, good for learning |
| 1B | 20B | about 17 days | Roughly GPT-2 class |
| 1B | 100B | about 12 weeks | Better, but still weak at reasoning |
| 3B | 60B | about 5 months | Doesn't fit without extra memory tricks |
Memory sets the ceiling. Standard mixed-precision training needs 16 bytes per parameter before activations,[7] so a 1B model takes 16 GB and fits on the card, while a 3B model takes 48 GB and doesn't. You'd need to move optimiser state into system memory or use a lower-precision optimiser, and both add complexity.
You also can't compete with open models at the same size. Llama 3.1 8B trained on about 15 trillion tokens,[8] which by the same arithmetic would take this machine around 270 years. Even a small open model has seen far more data than you can afford to show yours.
A from-scratch run can still be worth doing. It's the best way to learn how training really works, and Karpathy's nanochat is a good starting point.[9] It can also suit narrow, non-language data, such as logs or biological sequences, where a general model has little to offer.
Fine-tuning, where this machine is strong
LoRA freezes a model's weights and trains small added matrices.[10] QLoRA stores those frozen weights in 4 bits, which let its authors fine-tune a 65B model on a single 48 GB GPU.[11] On a 32 GB card, weights take about half a byte per parameter in 4 bits, so here's what fits by our estimate:
- 7B–13B models: comfortable, with room for longer sequences.
- About 32B: possible but tight, since weights alone take about 16 GB.
- 70B: no. Its 4-bit weights take about 35 GB, more than the card has.
Should you just rent?
For most people, yes. Runpod's Secure Cloud listed an RTX 5090 at $1.19 an hour and an H100 at $3.99 on 4 November 2026.[12] Assuming an 800 W system and the US residential average of about 18.2 cents per kWh,[13] here's the break-even against renting the same GPU:
rental = $1.19 per hour power = 0.8 kW × $0.182 per kWh = $0.15 per hour saving = $1.19 − $0.15 = $1.04 per hour breakeven = $10,000 / $1.04 = about 9,600 hours (13 months nonstop)
At 50% utilisation, that takes about 2.2 years. Faster rented hardware can also beat the 5090 on total cost. Our 1B-parameter, 20B-token run would take about 84 H100-hours, or around $340, and it would finish in days rather than weeks on one GPU. The same run on a rented 5090 costs about $470.
Buying makes sense in three situations:
- You'll keep the machine busy for a year or more.
- Your data can't leave your own hardware.
- You want to run many small experiments without watching a meter.
For the method behind these numbers, take The Economics of Model Training, a free Academy course.
