gpt-oss-120b on an H200
Yes, you can fine-tune gpt-oss-120b on one H200 (141 GB) with QLoRA. 4-bit inference: yes, 65 GB of weights, room for 128K tokens of context or 395 concurrent 4K-token sequences.
Verdict by method
| Fine-tuning | Memory at 2K tokens | One H200? | Longest sequence (batch 1) | Largest batch at 2K | Largest LoRA rank |
|---|---|---|---|---|---|
| QLoRA (4-bit base) | 66 GB | Yes | 32K tokens | 16 | 1024 |
| LoRA (16-bit base) | 238 GB | No | — | — | — |
| Full fine-tuning | 1,873 GB | No | — | — | n/a |
| Inference | Memory at 8K | One H200? | Longest context | Concurrent 4K sequences |
|---|---|---|---|---|
| 16-bit (BF16/FP16) | 234 GB | No | — | — |
| 8-bit | 117 GB | Yes | 128K tokens | 64 |
| 4-bit | 66 GB | Yes | 128K tokens | 395 |
- QLoRA: yes, about 66 GB at 2K tokens, batch 1, rank 16; up to 32K-token sequences at batch 1 or batch 16 at 2K, LoRA rank up to 1024.
- LoRA: no, it needs about 238 GB at 2K tokens, more than the 141 GB available.
- Full fine-tuning: no, it needs about 1,873 GB at 2K tokens, more than the 141 GB available.
Where the memory goes
| Method | Weights | Training state | Activations | Total | On one H200 | 80 GB GPUs needed |
|---|---|---|---|---|---|---|
| QLoRA (4-bit base) | 62 GB | 0.2 GB | 4.0 GB | 66 GB | Fits | 1 |
| LoRA (16-bit base) | 234 GB | 0.2 GB | 4.0 GB | 238 GB | Doesn't fit | 4 |
| Full fine-tuning | 234 GB | 1,636 GB | 4.0 GB | 1,873 GB | Doesn't fit | 27 |
| Precision | Weights | KV cache at 8K | Total | At 128K context | On one H200 |
|---|---|---|---|---|---|
| 16-bit (BF16/FP16) | 234 GB | 0.3 GB | 234 GB | 238 GB | Doesn't fit |
| 8-bit | 117 GB | 0.3 GB | 117 GB | 122 GB | Fits |
| 4-bit | 65 GB | 0.3 GB | 66 GB | 70 GB | Fits |
The cheapest way to run it
Renting an H200 costs $5.29 an hour on Runpod Secure Cloud ($3.59 on Runpod Community Cloud). QLoRA on it processes about 15,126 tokens a second by our estimate, so 10 million training tokens take 11 minutes and cost about $0.97.
| Job (default settings) | Memory | Cheapest rental that fits | Per hour |
|---|---|---|---|
| QLoRA fine-tuning | 66 GB | A100 80GB | $2.19 |
| LoRA fine-tuning | 238 GB | 6 × L40S | $6.54 |
| 4-bit inference | 66 GB | A100 80GB | $2.19 |
| 16-bit inference | 234 GB | 6 × L40S | $6.54 |
On-demand rates seen on 15 November 2026: Runpod Secure Cloud, Lambda for the A100 40GB. Consumer cards count as single GPUs only; data-centre GPUs up to eight per node. Kojen's GPU training can launch the H200 on Runpod, Vast.ai or Lambda, or pick the cheapest GPU in stock for you.
Try other settings
| Job | Memory | H200 (141 GB) | Smallest single GPU |
|---|---|---|---|
| QLoRA (4-bit base)2K tokens × batch 1, rank 16 | 66 GB | Fits | A100 80GB (80 GB) |
| LoRA (16-bit base)2K tokens × batch 1, rank 16 | 238 GB | 2 GPUs | None: needs several |
| Full fine-tuning2K tokens × batch 1 | 1,873 GB | 15 GPUs | None: needs several |
| Inference, 16-bit (BF16/FP16)8K-token context × 1 sequence | 234 GB | 2 GPUs | None: needs several |
| Inference, 8-bit8K-token context × 1 sequence | 117 GB | Fits | H200 (141 GB) |
| Inference, 4-bit8K-token context × 1 sequence | 66 GB | Fits | A100 80GB (80 GB) |
A job fits when the estimate is within 90% of the card's memory. Training assumes gradient checkpointing and FlashAttention; multi-GPU counts assume the memory splits evenly (FSDP, ZeRO-3 or tensor parallelism).
How we estimate it
Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).
- Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
- LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
- QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
- Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
- Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
- Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.
Speed estimates use H200's dense BF16 peak of 989 TFLOPS, about 4 FLOPs per active parameter per token for LoRA and 40% utilisation; see fine-tuning and training costs for the method.
Frequently asked questions
- Can I fine-tune gpt-oss-120b on an H200?
- QLoRA: yes, about 66 GB at 2K tokens, batch 1, rank 16; up to 32K-token sequences at batch 1 or batch 16 at 2K, LoRA rank up to 1024. LoRA: no, it needs about 238 GB at 2K tokens, more than the 141 GB available. Full fine-tuning: no, it needs about 1,873 GB at 2K tokens, more than the 141 GB available.
- Can I run gpt-oss-120b on an H200?
- 16-bit (BF16/FP16) inference: no, the weights alone are 234 GB. 8-bit inference: yes, 117 GB of weights, room for 128K tokens of context or 64 concurrent 4K-token sequences. 4-bit inference: yes, 65 GB of weights, room for 128K tokens of context or 395 concurrent 4K-token sequences.
- What does fine-tuning gpt-oss-120b on an H200 cost?
- With QLoRA at our planning rate of 40% utilisation, about 11 minutes for 10 million training tokens, around $0.97 at $5.29 an hour (Runpod Secure Cloud, November 2026).
Related
- gpt-oss-120b VRAM requirementsEvery GPU and method
- gpt-oss-120b on RTX 509032 GB
- gpt-oss-120b on H100 SXM80 GB
- gpt-oss-120b on B200180 GB
- Llama 3.1 70B on H200Meta
- Llama 3.3 70B on H200Meta
- Qwen2.5 72B on H200Alibaba Qwen
- DeepSeek-R1-Distill-Llama-70B on H200DeepSeek
Sources
- Hugging Face. openai/gpt-oss-120b config.json
- OpenAI. openai/gpt-oss-120b model card
- OpenAI. gpt-oss-120b & gpt-oss-20b Model Card (arXiv:2508.10925)
- NVIDIA. H200 specifications
- Runpod. GPU pricing (updated 12 November 2026)
- Lambda. GPU cloud pricing
- Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
- Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
- Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
- Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)
Measure before you rent the GPU.
Prompt tuning and model search in Kojen run on hosted models against your own test cases, so you know whether fine-tuning gpt-oss-120b is worth the GPU time.
Start training free