Mixtral 8x7B VRAM requirements
Fine-tuning Mixtral 8x7B takes about 26 GB of GPU memory with QLoRA, 95 GB with LoRA on 16-bit weights and 749 GB for full fine-tuning, at 2,048 tokens per sequence. Running it takes 94 GB in 16-bit or 27 GB in 4-bit with a 8K-token context. It is a mixture of experts: 12.9B of its 46.7B parameters work on each token, but all of them must be in memory.
Fine-tuning memory
| Method | Weights | Training state | Activations | Total | Smallest single GPU | 80 GB GPUs needed |
|---|---|---|---|---|---|---|
| QLoRA (4-bit base) | 24 GB | 0.2 GB | 1.7 GB | 26 GB | RTX 5090 (32 GB) | 1 |
| LoRA (16-bit base) | 93 GB | 0.2 GB | 1.7 GB | 95 GB | H200 (141 GB) | 2 |
| Full fine-tuning | 93 GB | 654 GB | 1.7 GB | 749 GB | None | 11 |
On a 24 GB card such as the RTX 4090: QLoRA: no, it needs about 26 GB at 2K tokens, more than the 24 GB available.
Inference memory
| Precision | Weights | KV cache at 8K | Total | At 32K context | Smallest single GPU |
|---|---|---|---|---|---|
| 16-bit (BF16/FP16) | 93 GB | 1.1 GB | 94 GB | 98 GB | H200 (141 GB) |
| 8-bit | 47 GB | 1.1 GB | 48 GB | 51 GB | A100 80GB (80 GB) |
| 4-bit | 26 GB | 1.1 GB | 27 GB | 31 GB | RTX 5090 (32 GB) |
Which GPUs fit Mixtral 8x7B
One GPU unless a count is shown. Follow a GPU's name for its maximum sequence length, batch size, LoRA rank and rental cost.
| GPU | QLoRA | LoRA | Full fine-tuning | Run, fp16 | Run, int8 | Run, int4 |
|---|---|---|---|---|---|---|
| RTX 3090 24 GB | 2× | 5× | 35× | 5× | 3× | 2× |
| RTX 4090 24 GB | 2× | 5× | 35× | 5× | 3× | 2× |
| RTX 5090 32 GB | Yes | 4× | 27× | 4× | 2× | Yes |
| L40S 48 GB | Yes | 3× | 18× | 3× | 2× | Yes |
| A100 40GB 40 GB | Yes | 3× | 21× | 3× | 2× | Yes |
| A100 80GB 80 GB | Yes | 2× | 11× | 2× | Yes | Yes |
| H100 SXM 80 GB | Yes | 2× | 11× | 2× | Yes | Yes |
| H200 141 GB | Yes | Yes | 6× | Yes | Yes | Yes |
| B200 180 GB | Yes | Yes | 5× | Yes | Yes | Yes |
What's specific to Mixtral 8x7B
- Not 56B: the experts share attention layers, so it has 46.7B parameters, of which 12.9B are used per token (Mistral AI).
- Memory follows the 46.7B total, so 16-bit inference needs about 93 GB even though it runs at the speed of a 13B model.
| Architecture | Mixtral 8x7B |
|---|---|
| Parameters | 46.7B total, 12.9B active per token |
| Experts | 8 per layer, 2 used per token |
| Layers | 32 |
| Hidden size | 4,096 |
| Expert MLP width | 14,336 |
| Attention heads / KV heads | 32 / 8 (head dim 128) |
| Vocabulary | 32,000 tokens |
| Context | 32K tokens |
| License | Apache 2.0 |
| Released | December 2023 |
Try other settings
Pre-filled for Mixtral 8x7B. Change the sequence length, batch, LoRA rank, context or GPU.
| Job | Memory | RTX 4090 (24 GB) | Smallest single GPU |
|---|---|---|---|
| QLoRA (4-bit base)2K tokens × batch 1, rank 16 | 26 GB | 2 GPUs | RTX 5090 (32 GB) |
| LoRA (16-bit base)2K tokens × batch 1, rank 16 | 95 GB | 5 GPUs | H200 (141 GB) |
| Full fine-tuning2K tokens × batch 1 | 749 GB | 35 GPUs | None: needs several |
| Inference, 16-bit (BF16/FP16)8K-token context × 1 sequence | 94 GB | 5 GPUs | H200 (141 GB) |
| Inference, 8-bit8K-token context × 1 sequence | 48 GB | 3 GPUs | A100 80GB (80 GB) |
| Inference, 4-bit8K-token context × 1 sequence | 27 GB | 2 GPUs | RTX 5090 (32 GB) |
A job fits when the estimate is within 90% of the card's memory. Training assumes gradient checkpointing and FlashAttention; multi-GPU counts assume the memory splits evenly (FSDP, ZeRO-3 or tensor parallelism).
How we estimate it
Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).
- Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
- LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
- QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
- Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
- Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
- Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.
Frequently asked questions
- Can I fine-tune Mixtral 8x7B on a 24 GB GPU?
- On an RTX 3090 or 4090 (24 GB): QLoRA: no, it needs about 26 GB at 2K tokens, more than the 24 GB available. LoRA: no, it needs about 95 GB at 2K tokens, more than the 24 GB available. Estimates use gradient checkpointing and leave 10% of memory free.
- How much VRAM do you need to run Mixtral 8x7B?
- About 94 GB in 16-bit and 27 GB in 4-bit, at 8K tokens of context. The smallest single GPU here for 4-bit is the RTX 5090 (32 GB); for 16-bit, the H200 (141 GB).
- How many GPUs does full fine-tuning Mixtral 8x7B take?
- About 749 GB with mixed-precision Adam at 2K tokens, so at least 11 × 80 GB H100s with the training state sharded (FSDP or ZeRO-3). LoRA needs 95 GB and QLoRA 26 GB.
- Why does Mixtral 8x7B need so much memory if only 12.9B parameters are active?
- It is a mixture of experts: each token uses 2 of 8 experts per layer, but which ones changes token to token, so all 46.7B parameters stay in GPU memory. Speed follows the active parameters; memory follows the total. Our LoRA figures adapt attention only; adapting every expert at rank 16 adds about 3.8 GB of adapter training state.
- How long a context can Mixtral 8x7B handle on an 80 GB GPU?
- Its KV cache costs 4.3 GB at the full 32K tokens (fp16 cache). On one 80 GB GPU: 16-bit weights don't fit, 32K tokens with 4-bit weights.
Related
- Mixtral 8x7B on RTX 409024 GB · max settings and cost
- Mixtral 8x7B on RTX 509032 GB · max settings and cost
- Mixtral 8x7B on A100 80GB80 GB · max settings and cost
- Mixtral 8x7B on H100 SXM80 GB · max settings and cost
- Mistral 7B VRAMMistral AI
Terms used on this page: LoRA, QLoRA, KV cache, gradient checkpointing, quantization, mixture of experts. For any model size, use the fine-tuning VRAM calculator; for the method in depth, the Academy lesson Memory first: what hardware can train what.
Sources
- Hugging Face. mistralai/Mixtral-8x7B-Instruct-v0.1 config.json
- Mistral AI. mistralai/Mixtral-8x7B-Instruct-v0.1 model card
- Mistral AI. Mixtral of Experts (arXiv:2401.04088)
- Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
- Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
- Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
- Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)
- Chen et al., 2016. Training Deep Nets with Sublinear Memory Cost (arXiv:1604.06174)
Architecture read from config.json and parameter count from the Hugging Face safetensors metadata on 15 November 2026.
Find out if you need to fine-tune at all.
Kojen's prompt tuning and model search run for real on your own test cases, on hosted models. Measure the cheaper levers before you rent a GPU.
Start training free