Skip to content

VRAM reference

Mixtral 8x7B on an A100 80GB

Yes, you can fine-tune Mixtral 8x7B on one A100 80GB (80 GB) with QLoRA. 4-bit inference: yes, 26 GB of weights, room for 32K tokens of context or 85 concurrent 4K-token sequences.

By Will Schulz · Checked 15 November 2026 · Prices as of November 2026

Verdict by method

Fine-tuningMemory at 2K tokensOne A100 80GB?Longest sequence (batch 1)Largest batch at 2KLargest LoRA rank
QLoRA (4-bit base)26 GBYes32K tokens281024
LoRA (16-bit base)95 GBNo———
Full fine-tuning749 GBNo——n/a
InferenceMemory at 8KOne A100 80GB?Longest contextConcurrent 4K sequences
16-bit (BF16/FP16)94 GBNo——
8-bit48 GBYes32K tokens47
4-bit27 GBYes32K tokens85
  • QLoRA: yes, about 26 GB at 2K tokens, batch 1, rank 16; up to 32K-token sequences at batch 1 or batch 28 at 2K, LoRA rank up to 1024.
  • LoRA: no, it needs about 95 GB at 2K tokens, more than the 80 GB available.
  • Full fine-tuning: no, it needs about 749 GB at 2K tokens, more than the 80 GB available.

Where the memory goes

At 2,048 tokens per sequence, batch 1, LoRA rank 16, gradient checkpointing on
MethodWeightsTraining stateActivationsTotalOn one A100 80GB80 GB GPUs needed
QLoRA (4-bit base)24 GB0.2 GB1.7 GB26 GBFits1
LoRA (16-bit base)93 GB0.2 GB1.7 GB95 GBDoesn't fit2
Full fine-tuning93 GB654 GB1.7 GB749 GBDoesn't fit11
One sequence, fp16 KV cache
PrecisionWeightsKV cache at 8KTotalAt 32K contextOn one A100 80GB
16-bit (BF16/FP16)93 GB1.1 GB94 GB98 GBDoesn't fit
8-bit47 GB1.1 GB48 GB51 GBFits
4-bit26 GB1.1 GB27 GB31 GBFits

The cheapest way to run it

Renting an A100 80GB costs $2.19 an hour on Runpod Secure Cloud ($1.39 on Runpod Community Cloud). QLoRA on it processes about 1,887 tokens a second by our estimate, so 10 million training tokens take 1.5 hours and cost about $3.22.

Job (default settings)MemoryCheapest rental that fitsPer hour
QLoRA fine-tuning26 GBL40S$1.09
LoRA fine-tuning95 GB3 × L40S$3.27
4-bit inference27 GBL40S$1.09
16-bit inference94 GB3 × L40S$3.27

On-demand rates seen on 15 November 2026: Runpod Secure Cloud, Lambda for the A100 40GB. Consumer cards count as single GPUs only; data-centre GPUs up to eight per node. Kojen's GPU training can launch the A100 80GB on Runpod, Vast.ai or Lambda, or pick the cheapest GPU in stock for you.

Try other settings

JobMemoryA100 80GB (80 GB)Smallest single GPU
QLoRA (4-bit base)2K tokens × batch 1, rank 1626 GBFitsRTX 5090 (32 GB)
LoRA (16-bit base)2K tokens × batch 1, rank 1695 GB2 GPUsH200 (141 GB)
Full fine-tuning2K tokens × batch 1749 GB11 GPUsNone: needs several
Inference, 16-bit (BF16/FP16)8K-token context × 1 sequence94 GB2 GPUsH200 (141 GB)
Inference, 8-bit8K-token context × 1 sequence48 GBFitsA100 80GB (80 GB)
Inference, 4-bit8K-token context × 1 sequence27 GBFitsRTX 5090 (32 GB)

A job fits when the estimate is within 90% of the card's memory. Training assumes gradient checkpointing and FlashAttention; multi-GPU counts assume the memory splits evenly (FSDP, ZeRO-3 or tensor parallelism).

How we estimate it

Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).

  • Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
  • LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
  • QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
  • Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
  • Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
  • Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.

Speed estimates use A100 80GB's dense BF16 peak of 312 TFLOPS, about 4 FLOPs per active parameter per token for LoRA and 40% utilisation; see fine-tuning and training costs for the method.

Frequently asked questions

Can I fine-tune Mixtral 8x7B on an A100 80GB?
QLoRA: yes, about 26 GB at 2K tokens, batch 1, rank 16; up to 32K-token sequences at batch 1 or batch 28 at 2K, LoRA rank up to 1024. LoRA: no, it needs about 95 GB at 2K tokens, more than the 80 GB available. Full fine-tuning: no, it needs about 749 GB at 2K tokens, more than the 80 GB available.
Can I run Mixtral 8x7B on an A100 80GB?
16-bit (BF16/FP16) inference: no, the weights alone are 93 GB. 8-bit inference: yes, 47 GB of weights, room for 32K tokens of context or 47 concurrent 4K-token sequences. 4-bit inference: yes, 26 GB of weights, room for 32K tokens of context or 85 concurrent 4K-token sequences.
What does fine-tuning Mixtral 8x7B on an A100 80GB cost?
With QLoRA at our planning rate of 40% utilisation, about 1.5 hours for 10 million training tokens, around $3.22 at $2.19 an hour (Runpod Secure Cloud, November 2026).

Sources

  1. [1]Hugging Face. mistralai/Mixtral-8x7B-Instruct-v0.1 config.json
  2. [2]Mistral AI. mistralai/Mixtral-8x7B-Instruct-v0.1 model card
  3. [3]Mistral AI. Mixtral of Experts (arXiv:2401.04088)
  4. [4]NVIDIA. A100 80GB specifications
  5. [5]Runpod. GPU pricing (updated 12 November 2026)
  6. [6]Lambda. GPU cloud pricing
  7. [7]Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
  8. [8]Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
  9. [9]Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
  10. [10]Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)

Measure before you rent the GPU.

Prompt tuning and model search in Kojen run on hosted models against your own test cases, so you know whether fine-tuning Mixtral 8x7B is worth the GPU time.

Start training free