Skip to content

VRAM reference

Mistral 7B VRAM requirements

Fine-tuning Mistral 7B takes about 6.3 GB of GPU memory with QLoRA, 17 GB with LoRA on 16-bit weights and 117 GB for full fine-tuning, at 2,048 tokens per sequence. Running it takes 16 GB in 16-bit or 5.2 GB in 4-bit with a 8K-token context.

By Will Schulz · Architecture and prices checked 15 November 2026 · Prices as of November 2026

Fine-tuning memory

At 2,048 tokens per sequence, batch 1, LoRA rank 16, gradient checkpointing on
MethodWeightsTraining stateActivationsTotalSmallest single GPU80 GB GPUs needed
QLoRA (4-bit base)4.1 GB0.7 GB1.5 GB6.3 GBRTX 3090 (24 GB)1
LoRA (16-bit base)14 GB0.7 GB1.5 GB17 GBRTX 3090 (24 GB)1
Full fine-tuning14 GB101 GB1.5 GB117 GBH200 (141 GB)2

On a 24 GB card such as the RTX 4090: QLoRA: yes, about 6.3 GB at 2K tokens, batch 1, rank 16; up to 16K-token sequences at batch 1 or batch 11 at 2K, LoRA rank up to 256.

Inference memory

One sequence, fp16 KV cache
PrecisionWeightsKV cache at 8KTotalAt 32K contextSmallest single GPU
16-bit (BF16/FP16)14 GB1.1 GB16 GB19 GBRTX 3090 (24 GB)
8-bit7.2 GB1.1 GB8.3 GB12 GBRTX 3090 (24 GB)
4-bit4.1 GB1.1 GB5.2 GB8.4 GBRTX 3090 (24 GB)

Which GPUs fit Mistral 7B

One GPU unless a count is shown. Follow a GPU's name for its maximum sequence length, batch size, LoRA rank and rental cost.

GPUQLoRALoRAFull fine-tuningRun, fp16Run, int8Run, int4
RTX 3090 24 GBYesYes6×YesYesYes
RTX 4090 24 GBYesYes6×YesYesYes
RTX 5090 32 GBYesYes5×YesYesYes
L40S 48 GBYesYes3×YesYesYes
A100 40GB 40 GBYesYes4×YesYesYes
A100 80GB 80 GBYesYes2×YesYesYes
H100 SXM 80 GBYesYes2×YesYesYes
H200 141 GBYesYesYesYesYesYes
B200 180 GBYesYesYesYesYesYes

What's specific to Mistral 7B

  • v0.3 drops the sliding window of the original release and uses full attention over 32K tokens.
  • A 32K-token vocabulary keeps training logits small, so it trains at longer sequence lengths than Llama 3.1 8B on the same card.
ArchitectureMistral 7B
Parameters7.25B
Layers32
Hidden size4,096
MLP width14,336
Attention heads / KV heads32 / 8 (head dim 128)
Vocabulary32,768 tokens
Context32K tokens
LicenseApache 2.0
ReleasedMay 2024 (v0.3)

Try other settings

Pre-filled for Mistral 7B. Change the sequence length, batch, LoRA rank, context or GPU.

JobMemoryRTX 4090 (24 GB)Smallest single GPU
QLoRA (4-bit base)2K tokens × batch 1, rank 166.3 GBFitsRTX 3090 (24 GB)
LoRA (16-bit base)2K tokens × batch 1, rank 1617 GBFitsRTX 3090 (24 GB)
Full fine-tuning2K tokens × batch 1117 GB6 GPUsH200 (141 GB)
Inference, 16-bit (BF16/FP16)8K-token context × 1 sequence16 GBFitsRTX 3090 (24 GB)
Inference, 8-bit8K-token context × 1 sequence8.3 GBFitsRTX 3090 (24 GB)
Inference, 4-bit8K-token context × 1 sequence5.2 GBFitsRTX 3090 (24 GB)

A job fits when the estimate is within 90% of the card's memory. Training assumes gradient checkpointing and FlashAttention; multi-GPU counts assume the memory splits evenly (FSDP, ZeRO-3 or tensor parallelism).

How we estimate it

Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).

  • Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
  • LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
  • QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
  • Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
  • Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
  • Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.

Frequently asked questions

Can I fine-tune Mistral 7B on a 24 GB GPU?
On an RTX 3090 or 4090 (24 GB): QLoRA: yes, about 6.3 GB at 2K tokens, batch 1, rank 16; up to 16K-token sequences at batch 1 or batch 11 at 2K, LoRA rank up to 256. LoRA: yes, about 17 GB at 2K tokens, batch 1, rank 16; up to 8K-token sequences at batch 1 or batch 4 at 2K, LoRA rank up to 128. Estimates use gradient checkpointing and leave 10% of memory free.
How much VRAM do you need to run Mistral 7B?
About 16 GB in 16-bit and 5.2 GB in 4-bit, at 8K tokens of context. The smallest single GPU here for 4-bit is the RTX 3090 (24 GB); for 16-bit, the RTX 3090 (24 GB).
How many GPUs does full fine-tuning Mistral 7B take?
About 117 GB with mixed-precision Adam at 2K tokens, so at least 2 × 80 GB H100s with the training state sharded (FSDP or ZeRO-3). LoRA needs 17 GB and QLoRA 6.3 GB.
How long a context can Mistral 7B handle on an 80 GB GPU?
Its KV cache costs 4.3 GB at the full 32K tokens (fp16 cache). On one 80 GB GPU: 32K tokens with 16-bit weights, 32K tokens with 4-bit weights.

Terms used on this page: LoRA, QLoRA, KV cache, gradient checkpointing, quantization. For any model size, use the fine-tuning VRAM calculator; for the method in depth, the Academy lesson Memory first: what hardware can train what.

Sources

  1. [1]Hugging Face. mistralai/Mistral-7B-Instruct-v0.3 config.json
  2. [2]Mistral AI. mistralai/Mistral-7B-Instruct-v0.3 model card
  3. [3]Mistral AI. Mistral 7B (arXiv:2310.06825)
  4. [4]Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
  5. [5]Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
  6. [6]Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
  7. [7]Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)
  8. [8]Chen et al., 2016. Training Deep Nets with Sublinear Memory Cost (arXiv:1604.06174)

Architecture read from config.json and parameter count from the Hugging Face safetensors metadata on 15 November 2026.

Find out if you need to fine-tune at all.

Kojen's prompt tuning and model search run for real on your own test cases, on hosted models. Measure the cheaper levers before you rent a GPU.

Start training free