Skip to content

VRAM reference

gpt-oss-120b on an RTX 5090

No fine-tuning method fits gpt-oss-120b on one RTX 5090 (32 GB) at 2,048 tokens. 4-bit inference: no, the weights alone are 65 GB.

By Will Schulz · Checked 15 November 2026 · Prices as of November 2026

Verdict by method

Fine-tuningMemory at 2K tokensOne RTX 5090?Longest sequence (batch 1)Largest batch at 2KLargest LoRA rank
QLoRA (4-bit base)66 GBNo———
LoRA (16-bit base)238 GBNo———
Full fine-tuning1,873 GBNo——n/a
InferenceMemory at 8KOne RTX 5090?Longest contextConcurrent 4K sequences
16-bit (BF16/FP16)234 GBNo——
8-bit117 GBNo——
4-bit66 GBNo——
  • QLoRA: no, it needs about 66 GB at 2K tokens, more than the 32 GB available.
  • LoRA: no, it needs about 238 GB at 2K tokens, more than the 32 GB available.
  • Full fine-tuning: no, it needs about 1,873 GB at 2K tokens, more than the 32 GB available.

Where the memory goes

At 2,048 tokens per sequence, batch 1, LoRA rank 16, gradient checkpointing on
MethodWeightsTraining stateActivationsTotalOn one RTX 509080 GB GPUs needed
QLoRA (4-bit base)62 GB0.2 GB4.0 GB66 GBDoesn't fit1
LoRA (16-bit base)234 GB0.2 GB4.0 GB238 GBDoesn't fit4
Full fine-tuning234 GB1,636 GB4.0 GB1,873 GBDoesn't fit27
One sequence, fp16 KV cache
PrecisionWeightsKV cache at 8KTotalAt 128K contextOn one RTX 5090
16-bit (BF16/FP16)234 GB0.3 GB234 GB238 GBDoesn't fit
8-bit117 GB0.3 GB117 GB122 GBDoesn't fit
4-bit65 GB0.3 GB66 GB70 GBDoesn't fit

The cheapest way to run it

Renting an RTX 5090 costs $1.39 an hour on Runpod Secure Cloud ($0.69 on Runpod Community Cloud). Since no fine-tuning method fits, the table shows the cheapest hardware that does.

Job (default settings)MemoryCheapest rental that fitsPer hour
QLoRA fine-tuning66 GBA100 80GB$2.19
LoRA fine-tuning238 GB6 × L40S$6.54
4-bit inference66 GBA100 80GB$2.19
16-bit inference234 GB6 × L40S$6.54

On-demand rates seen on 15 November 2026: Runpod Secure Cloud, Lambda for the A100 40GB. Consumer cards count as single GPUs only; data-centre GPUs up to eight per node.

Try other settings

JobMemoryRTX 5090 (32 GB)Smallest single GPU
QLoRA (4-bit base)2K tokens × batch 1, rank 1666 GB3 GPUsA100 80GB (80 GB)
LoRA (16-bit base)2K tokens × batch 1, rank 16238 GB9 GPUsNone: needs several
Full fine-tuning2K tokens × batch 11,873 GB66 GPUsNone: needs several
Inference, 16-bit (BF16/FP16)8K-token context × 1 sequence234 GB9 GPUsNone: needs several
Inference, 8-bit8K-token context × 1 sequence117 GB5 GPUsH200 (141 GB)
Inference, 4-bit8K-token context × 1 sequence66 GB3 GPUsA100 80GB (80 GB)

A job fits when the estimate is within 90% of the card's memory. Training assumes gradient checkpointing and FlashAttention; multi-GPU counts assume the memory splits evenly (FSDP, ZeRO-3 or tensor parallelism).

How we estimate it

Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).

  • Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
  • LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
  • QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
  • Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
  • Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
  • Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.

Speed estimates use RTX 5090's dense BF16 peak of 209.5 TFLOPS, about 4 FLOPs per active parameter per token for LoRA and 40% utilisation; see fine-tuning and training costs for the method.

Frequently asked questions

Can I fine-tune gpt-oss-120b on an RTX 5090?
QLoRA: no, it needs about 66 GB at 2K tokens, more than the 32 GB available. LoRA: no, it needs about 238 GB at 2K tokens, more than the 32 GB available. Full fine-tuning: no, it needs about 1,873 GB at 2K tokens, more than the 32 GB available.
Can I run gpt-oss-120b on an RTX 5090?
16-bit (BF16/FP16) inference: no, the weights alone are 234 GB. 8-bit inference: no, the weights alone are 117 GB. 4-bit inference: no, the weights alone are 65 GB.

Sources

  1. [1]Hugging Face. openai/gpt-oss-120b config.json
  2. [2]OpenAI. openai/gpt-oss-120b model card
  3. [3]OpenAI. gpt-oss-120b & gpt-oss-20b Model Card (arXiv:2508.10925)
  4. [4]NVIDIA. RTX 5090 specifications
  5. [5]Runpod. GPU pricing (updated 12 November 2026)
  6. [6]Lambda. GPU cloud pricing
  7. [7]Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
  8. [8]Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
  9. [9]Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
  10. [10]Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)

Measure before you rent the GPU.

Prompt tuning and model search in Kojen run on hosted models against your own test cases, so you know whether fine-tuning gpt-oss-120b is worth the GPU time.

Start training free