GPU memory for LLM inference, worked out from the model config

Assumptions, and when this does not apply

The arithmetic covers inference of dense decoder-only transformer models (Llama, Mistral-style and similar) served by an engine that keeps a KV cache, such as vLLM, TensorRT-LLM or Hugging Face TGI. It uses decimal units: 1 GB = 1,000,000,000 bytes, which is how GPU vendors state memory. Tools such as nvidia-smi report MiB, so check the real total on the machine with nvidia-smi --query-gpu=memory.total --format=csv.

It does not cover, or needs adjusting for:

• Training and fine-tuning. Gradients and optimizer states add several times the weight memory. Full fine-tuning with Adam needs far more than inference.
• Mixture-of-experts models. All experts must be resident, so weight memory follows total parameters even though compute follows active parameters.
• Multimodal models. Vision encoders and image tokens add memory and tokens not counted here.
• Models with different attention layouts (sliding-window, multi-head latent attention). Read the KV formula from the model's own architecture.
• Speculative decoding and LoRA adapters. Draft models and adapters add their own weights and cache.

The three parts of GPU memory

1. Weights. Parameters x bytes per parameter. 16-bit formats (FP16, BF16) use 2 bytes, FP8 uses 1, and 4-bit formats (INT4, AWQ, GPTQ) use about 0.5 plus small per-group scale factors. The simplest check is the total size of the model's .safetensors files.
2. KV cache. For every token in every active sequence, each layer stores one key and one value vector per KV head: 2 x layers x KV heads x head dimension x bytes per element bytes per token. Multiply by tokens = context length x concurrent sequences. NVIDIA's inference optimization guide gives the same per-token formula.
3. Runtime overhead. CUDA or ROCm context, activations for the batch being processed, communication buffers and allocator fragmentation. It varies with engine, batch size and settings. This guide uses a stated planning assumption: the larger of 2 GB or 10% of weights plus KV cache. Replace it with a measured number once the model is running.

Then compare the total with usable GPU memory, not nameplate memory. Engines leave a margin: vLLM, for example, takes a configurable fraction (gpu_memory_utilization; the default has been 0.9 in many releases, check your version) and uses what is left after weights and activations for KV cache blocks. This guide treats 90% of nameplate as usable.

The Llama 3.1 8B and 70B configuration numbers

The inputs come from the models' published configuration (config.json on the Hugging Face model pages, which require accepting Meta's license to view) and Table 3 of Meta's Llama 3 paper. Head dimension is hidden size divided by attention heads.

Grouped-query attention is why these models are cheaper to serve than their size suggests: 8 KV heads instead of 32 or 64 cuts KV cache by 4x (8B) and 8x (70B) compared with one KV head per attention head.

Weights at FP16, FP8 and INT4

The INT4 figures are lower bounds. Many 4-bit checkpoints keep the token embedding and output layers at 16-bit. For these models that is 2 x 128,256 x 4,096 = 1.05 billion parameters (8B) and 2 x 128,256 x 8,192 = 2.10 billion (70B). At 2 bytes instead of 0.5 that adds about 1.6 GB and 3.2 GB. Group-wise scales add roughly another 1-3%. Read the checkpoint's file sizes before you commit.

Precision also changes quality and speed. FP8 is widely used for serving on GPUs with native FP8 support; 4-bit formats save the most memory but should be checked on your own evaluation set.

KV cache at 8k and 32k context, 1 and 16 sequences

These are worst cases: every sequence at full context at the same moment. Paged KV cache engines such as vLLM allocate blocks as tokens arrive, so average use is lower when prompts are short. Size for the worst case you will allow (via max_model_len and max_num_seqs in vLLM), not the average you hope for. An FP8 KV cache halves this part, with an accuracy trade-off you should test.

Totals: weights plus KV cache plus overhead

Each cell is weights + 16-bit KV cache + overhead (the larger of 2 GB or 10%), in GB.

Two lessons stand out. First, at long context and real concurrency the KV cache, not the weights, decides the GPU: the 8B model at INT4 still needs 80 GB for 16 sequences at 32k. Second, quantizing weights helps the 70B model most at short context; at 32k x 16 the cache dominates whatever the weight precision.

Public GPU memory sizes

Nameplate memory and bandwidth from vendor datasheets and product pages. These are facts about the hardware, not an offer; GPU classes can be quoted subject to supplier availability, with the exact model confirmed in writing.

NVIDIA's DGX B200 page states system totals; the per-GPU figures are that total divided by eight. Other B200 products may state different per-GPU capacity, so check the datasheet for the exact system you are quoted.

Worked examples: which GPU fits

Tensor parallelism across GPUs

Tensor parallelism splits each layer's weight matrices across N GPUs in one server, so each GPU holds about 1/N of the weights and 1/N of the KV heads. Points that change the arithmetic:

• Divisibility. Serving engines generally require the number of attention heads to be divisible by N, so N is usually 2, 4 or 8.
• KV heads. With 8 KV heads, N up to 8 divides the cache evenly. Beyond 8, KV heads are replicated, and the cache stops shrinking per GPU.
• Overhead does not divide. Each GPU has its own runtime context and buffers, so budget overhead per GPU.
• Interconnect. Each layer needs collective communication between the GPUs for every token. NVLink or Infinity Fabric links keep this fast; over PCIe alone, scaling beyond 2 GPUs often disappoints. Confirm the topology in the quote.
• Across servers, pipeline parallelism (splitting layers between nodes) is more common than tensor parallelism, because network links are slower than in-server GPU links.

In vLLM the setting is tensor_parallel_size. Test the exact layout before buying: a configuration that fits on paper can still miss latency targets.

Memory bandwidth sets the speed ceiling

During decoding, every new token requires reading the weights and that sequence's KV cache from GPU memory. That gives a theoretical upper bound for one sequence: tokens per second at most bandwidth divided by bytes read per token. This is a ceiling, not a benchmark; real figures are lower.

Batching raises total throughput because the weights are read once per step for all sequences in the batch. That is why the KV cache budget, which caps the batch, matters as much as raw speed. Turn throughput into cost per million tokens with model-serving economics.

Checklist before you request GPU capacity

• Exact model and revision, with config.json values for layers, KV heads and head dimension.
• Weight precision and the checkpoint's real file size.
• Maximum context length you will allow, not the model maximum.
• Peak concurrent sequences, from expected users and request duration.
• KV cache precision (16-bit or FP8) and an accuracy check if FP8.
• Total memory from the formula, compared with 90% of nameplate.
• If more than one GPU: tensor parallel size, head divisibility and interconnect.
• Throughput and latency targets, and a test plan to check them.
• Numbers entered in the GPU memory calculator and attached to the quote request.

Common mistakes

• Counting only the weights. At 32k context and 16 users, the 8B cache is 4.3x the FP16 weights.
• Using attention heads instead of KV heads. For Llama 3.1 that overstates the cache by 4x (8B) or 8x (70B).
• Leaving the context at the model maximum. Llama 3.1 allows 131,072 tokens. If you only need 8k, cap it; otherwise the engine may reserve for the maximum or reject the configuration.
• Treating INT4 size as exact. Embedding and output layers at 16-bit and quantization scales add gigabytes.
• Planning at 100% of nameplate. Runtime context, fragmentation and engine margins take real memory.
• Splitting across GPUs without fast links. Tensor parallelism over slow links can fit the model and still miss latency.
• Mixing GB and GiB. 80 GB is about 74.5 GiB. Pick one unit and keep it.

Related