Pre-launch: quote requests are open; self-serve checkout is not open yet. Request a quote
Request a quote

Free planning tool

GPU memory calculator for LLM inference

Weights, KV cache and runtime overhead for a transformer model, and the minimum number of GPUs of each class to hold it.

What this tool answers

How much GPU memory a model needs to serve requests: the weights (parameters × bytes per parameter), the KV cache (which grows with context length and the number of concurrent sequences) and runtime overhead. It then shows how many GPUs of each common class are needed, using vendor-published memory sizes.

Calculator

Model
Fills in the architecture from the model’s published configuration.
Grouped-query attention: often fewer than attention heads.
Load
Activations, CUDA context, fragmentation. An assumption, not a constant.
vLLM’s default gpu-memory-utilization is 0.9.

Results appear here as you type.

How the estimate is calculated

Three parts are added together. Units are decimal gigabytes (1 GB = 109 bytes), the same unit GPU vendors use for memory sizes.

Formula
PartFormula
Weightsparameters × bytes per parameter
KV cache2 (keys and values) × layers × KV heads × head dimension × KV bytes × context length × concurrent sequences
Overhead(weights + KV cache) × overhead %
GPUs neededtotal ÷ (GPU memory × usable %), rounded up; tensor-parallel groups are usually 1, 2, 4 or 8 GPUs

Worked example: Llama 3.1 8B at BF16

Worked example

Weights: 8.03 billion × 2 bytes = 16.06 GB.

KV cache per token: 2 × 32 layers × 8 KV heads × 128 × 2 bytes = 131,072 bytes (128 KiB). At an 8,192-token context and one sequence: 131,072 × 8,192 = 1.07 GB. With 16 concurrent sequences it is 17.2 GB, more than the weights.

With 10% overhead, one sequence needs about 18.8 GB: it fits on a single 24 GB L4 at 90% usable memory (21.6 GB). Sixteen sequences need about 36.6 GB, which needs a 48 GB L40S or larger.

The second half of that example is the point most sizing mistakes miss: concurrency and context length, not model size, often decide the GPU. The GPU memory guide works through 70B models and multi-GPU layouts.

GPU memory sizes used

Vendor-published memory per GPU
GPUMemory
NVIDIA L424 GB
NVIDIA L40S48 GB
NVIDIA A100 40GB40 GB
NVIDIA A100 80GB80 GB
NVIDIA H100 SXM80 GB
NVIDIA H100 NVL94 GB
NVIDIA H200141 GB
NVIDIA B200180 GB
AMD Instinct MI300X192 GB

B200 is taken as 180 GB per GPU from NVIDIA’s HGX reference architecture (1,440 GB across eight GPUs); some NVIDIA pages state 192 GB. H100 in PCIe form is now sold as H100 NVL with 94 GB. Usable memory is always lower than the nameplate figure. Listing a GPU here does not mean BroadNet has it in stock: GPU availability is confirmed per quote.

Assumptions and limits

  • Dense transformer models only. Mixture-of-experts models must hold all experts in memory even though only some are active per token.
  • Inference only. Fine-tuning needs gradients and optimizer state: roughly 3 to 4 times the weight memory or more for full fine-tuning, much less with LoRA/QLoRA.
  • Paged KV caches (vLLM) allocate in blocks, so real use is close to, but not exactly, the formula.
  • Quantized weights add small per-group scale factors that this estimate ignores.
  • Throughput is a separate question: a model that fits may still be too slow. Use the model-serving cost tool with a measured tokens-per-second figure.

How we test this tool

The formula is pinned by unit tests that run on every build, including: Llama 3.1 8B at FP16 and 8k context (16.06 GB weights, 131.1 KB of KV cache per token, fits one L4); Llama 3.1 70B at FP16 with 16 sequences (needs 3 H100s, so a tensor-parallel group of 4); INT4 weights exactly half of FP8; and rejection of negative or impossible inputs.

Frequently asked questions

Why does the KV cache matter so much?

It grows linearly with context length and with the number of sequences served at once. At long contexts or high concurrency it can be larger than the model weights.

Does FP8 or INT4 reduce quality?

It can. Many models keep close to their original quality at FP8; INT4 usually loses more. Test with your own evaluation set before committing to a precision.

Can I split a model across GPUs?

Yes. Tensor parallelism splits each layer across GPUs in one server (best with NVLink); pipeline parallelism splits layers across GPUs or servers. Both add communication overhead.

Sources

  1. NVIDIA L4 Tensor Core GPU, NVIDIA
  2. NVIDIA L40S GPU, NVIDIA
  3. NVIDIA A100 Tensor Core GPU datasheet, NVIDIA
  4. NVIDIA H100 Tensor Core GPU, NVIDIA
  5. NVIDIA H200 Tensor Core GPU, NVIDIA
  6. NVIDIA HGX AI Factory reference architecture: components (H100, H200, B200), NVIDIA
  7. AMD Instinct MI300X accelerator, AMD
  8. Mastering LLM techniques: inference optimization (KV cache sizing), NVIDIA Technical Blog
  9. Llama 3.1 model card and configuration, Meta
  10. vLLM engine arguments (gpu-memory-utilization), vLLM project
GPU memory for LLM inferenceThe arithmetic for weights, KV cache and overhead, worked for Llama 3.1 8B and 70B at three precisions, with cited GPU m…AI and GPU hostingGPU servers and GPU cloud for inference, fine-tuning and training: GPU classes we can quote, VRAM sizing for 8B and 70B …Model-serving costCost per million tokens from GPU price, throughput and utilization.

Inputs stay in your browser. Tool maths last reviewed ; the unit tests that pin these formulas are listed in the evidence hub.

Turn the estimate into a quote

Send us the inputs you used. We reply with a configuration that fits and a written price.

Request a quoteAll planning tools