Llama 3.1 8B on an L40S
Yes, you can fine-tune Llama 3.1 8B on one L40S (48 GB) with QLoRA or LoRA. 4-bit inference: yes, 4.5 GB of weights, room for 128K tokens of context or 72 concurrent 4K-token sequences.
Verdict by method
| Fine-tuning | Memory at 2K tokens | One L40S? | Longest sequence (batch 1) | Largest batch at 2K | Largest LoRA rank |
|---|---|---|---|---|---|
| QLoRA (4-bit base) | 9.4 GB | Yes | 16K tokens | 12 | 512 |
| LoRA (16-bit base) | 20 GB | Yes | 16K tokens | 8 | 512 |
| Full fine-tuning | 132 GB | No | — | — | n/a |
| Inference | Memory at 8K | One L40S? | Longest context | Concurrent 4K sequences |
|---|---|---|---|---|
| 16-bit (BF16/FP16) | 17 GB | Yes | 128K tokens | 50 |
| 8-bit | 9.1 GB | Yes | 128K tokens | 65 |
| 4-bit | 5.6 GB | Yes | 128K tokens | 72 |
- QLoRA: yes, about 9.4 GB at 2K tokens, batch 1, rank 16; up to 16K-token sequences at batch 1 or batch 12 at 2K, LoRA rank up to 512.
- LoRA: yes, about 20 GB at 2K tokens, batch 1, rank 16; up to 16K-token sequences at batch 1 or batch 8 at 2K, LoRA rank up to 512.
- Full fine-tuning: no, it needs about 132 GB at 2K tokens, more than the 48 GB available.
Where the memory goes
| Method | Weights | Training state | Activations | Total | On one L40S | 80 GB GPUs needed |
|---|---|---|---|---|---|---|
| QLoRA (4-bit base) | 5.7 GB | 0.7 GB | 3.0 GB | 9.4 GB | Fits | 1 |
| LoRA (16-bit base) | 16 GB | 0.7 GB | 3.0 GB | 20 GB | Fits | 1 |
| Full fine-tuning | 16 GB | 112 GB | 3.0 GB | 132 GB | Doesn't fit | 2 |
| Precision | Weights | KV cache at 8K | Total | At 128K context | On one L40S |
|---|---|---|---|---|---|
| 16-bit (BF16/FP16) | 16 GB | 1.1 GB | 17 GB | 33 GB | Fits |
| 8-bit | 8.0 GB | 1.1 GB | 9.1 GB | 25 GB | Fits |
| 4-bit | 4.5 GB | 1.1 GB | 5.6 GB | 22 GB | Fits |
The cheapest way to run it
Renting an L40S costs $1.09 an hour on Runpod Secure Cloud ($0.79 on Runpod Community Cloud). LoRA on it processes about 4,508 tokens a second by our estimate, so 10 million training tokens take 37 minutes and cost about $0.67.
| Job (default settings) | Memory | Cheapest rental that fits | Per hour |
|---|---|---|---|
| QLoRA fine-tuning | 9.4 GB | RTX 3090 | $0.50 |
| LoRA fine-tuning | 20 GB | RTX 3090 | $0.50 |
| 4-bit inference | 5.6 GB | RTX 3090 | $0.50 |
| 16-bit inference | 17 GB | RTX 3090 | $0.50 |
On-demand rates seen on 15 November 2026: Runpod Secure Cloud, Lambda for the A100 40GB. Consumer cards count as single GPUs only; data-centre GPUs up to eight per node. Kojen's GPU training can launch the L40S on Runpod, Vast.ai or Lambda, or pick the cheapest GPU in stock for you.
Try other settings
| Job | Memory | L40S (48 GB) | Smallest single GPU |
|---|---|---|---|
| QLoRA (4-bit base)2K tokens × batch 1, rank 16 | 9.4 GB | Fits | RTX 3090 (24 GB) |
| LoRA (16-bit base)2K tokens × batch 1, rank 16 | 20 GB | Fits | RTX 3090 (24 GB) |
| Full fine-tuning2K tokens × batch 1 | 132 GB | 4 GPUs | B200 (180 GB) |
| Inference, 16-bit (BF16/FP16)8K-token context × 1 sequence | 17 GB | Fits | RTX 3090 (24 GB) |
| Inference, 8-bit8K-token context × 1 sequence | 9.1 GB | Fits | RTX 3090 (24 GB) |
| Inference, 4-bit8K-token context × 1 sequence | 5.6 GB | Fits | RTX 3090 (24 GB) |
A job fits when the estimate is within 90% of the card's memory. Training assumes gradient checkpointing and FlashAttention; multi-GPU counts assume the memory splits evenly (FSDP, ZeRO-3 or tensor parallelism).
How we estimate it
Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).
- Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
- LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
- QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
- Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
- Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
- Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.
Speed estimates use L40S's dense BF16 peak of 362 TFLOPS, about 4 FLOPs per active parameter per token for LoRA and 40% utilisation; see fine-tuning and training costs for the method.
Frequently asked questions
- Can I fine-tune Llama 3.1 8B on an L40S?
- QLoRA: yes, about 9.4 GB at 2K tokens, batch 1, rank 16; up to 16K-token sequences at batch 1 or batch 12 at 2K, LoRA rank up to 512. LoRA: yes, about 20 GB at 2K tokens, batch 1, rank 16; up to 16K-token sequences at batch 1 or batch 8 at 2K, LoRA rank up to 512. Full fine-tuning: no, it needs about 132 GB at 2K tokens, more than the 48 GB available.
- Can I run Llama 3.1 8B on an L40S?
- 16-bit (BF16/FP16) inference: yes, 16 GB of weights, room for 128K tokens of context or 50 concurrent 4K-token sequences. 8-bit inference: yes, 8.0 GB of weights, room for 128K tokens of context or 65 concurrent 4K-token sequences. 4-bit inference: yes, 4.5 GB of weights, room for 128K tokens of context or 72 concurrent 4K-token sequences.
- What does fine-tuning Llama 3.1 8B on an L40S cost?
- With LoRA at our planning rate of 40% utilisation, about 37 minutes for 10 million training tokens, around $0.67 at $1.09 an hour (Runpod Secure Cloud, November 2026).
Related
- Llama 3.1 8B VRAM requirementsEvery GPU and method
- Llama 3.1 8B on RTX 309024 GB
- Llama 3.1 8B on RTX 409024 GB
- Llama 3.1 8B on RTX 509032 GB
- Llama 3.1 8B on A100 40GB40 GB
- Llama 3.1 8B on A100 80GB80 GB
- Llama 3.1 8B on H100 SXM80 GB
- Fine-tuning Llama 3.1 8B: costPer million tokens
- Qwen3 8B on L40SAlibaba Qwen
- Qwen3 14B on L40SAlibaba Qwen
- Qwen3 32B on L40SAlibaba Qwen
- Qwen3 30B-A3B on L40SAlibaba Qwen
- Gemma 3 27B on L40SGoogle DeepMind
Sources
- Hugging Face. meta-llama/Llama-3.1-8B-Instruct config.json
- Meta. meta-llama/Llama-3.1-8B-Instruct model card
- Meta. The Llama 3 Herd of Models (arXiv:2407.21783)
- NVIDIA. L40S specifications
- Runpod. GPU pricing (updated 12 November 2026)
- Lambda. GPU cloud pricing
- Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
- Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
- Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
- Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)
Skip the setup: Kojen trains Llama 3.1 8B LoRA.
Kojen trains LoRA adapters on Llama 3.1 8B from your input/expected rows on the cheapest GPU in stock, billed by the second (for accounts with GPU training enabled).
Start training free