GPU memory decides most AI hosting choices. Add model weights (parameters times bytes per parameter), the KV cache for your context and concurrency, and about 10 percent overhead, then pick the smallest GPU or group that holds it. An 8B model at FP16 needs roughly 35 GB under moderate load; a 70B model needs one large GPU at FP8 or several at FP16. We quote monthly or on a term.
Who GPU hosting is for, and who should skip it
A good fit
- Teams serving open-weight models (Llama, Mistral, Qwen and similar) to their own users and wanting predictable monthly cost.
- Products where prompts or outputs must not leave infrastructure you control.
- Fine-tuning runs (LoRA or full) that take days, not minutes.
- Steady inference load high enough that per-token API pricing has become the larger bill.
Probably not a fit
- Occasional experiments of a few hours: hourly on-demand GPU clouds suit that better, and we do not offer per-second or hourly billing.
- Low or uncertain traffic: a hosted model API is usually cheaper until volume is steady. Check with the model-serving cost tool.
- Large multi-node pre-training on thousands of GPUs: that is a capacity reservation conversation with a hyperscale supplier, not a quote-led server.
BroadNet Cloud does not own GPUs or data centers. We size, quote, provision and support GPU capacity from established suppliers. Which GPU models are available, in which region and when, depends on the supplier at the time of the quote. The quote names the supplier, the region and the exact server.
GPU classes we can quote
These are the vendor-published figures for each GPU class. They are facts about the hardware, not a statement of what is available today; availability is confirmed per quote.
| GPU | Memory per GPU | Memory bandwidth | GPU-to-GPU link | Typical use |
|---|---|---|---|---|
| NVIDIA L4 | 24 GB GDDR6 | 300 GB/s | PCIe Gen4 only | Small-model inference, embeddings, video; 72 W card |
| NVIDIA L40S | 48 GB GDDR6 with ECC | 864 GB/s | PCIe Gen4 only (no NVLink) | 7B to 13B inference, image generation, light fine-tuning |
| NVIDIA A100 | 40 GB HBM2 or 80 GB HBM2e | 1,555 GB/s (40 GB); up to 2,039 GB/s (80 GB SXM) | NVLink 600 GB/s (SXM; bridge for 2 PCIe cards) | Fine-tuning and inference where FP8 is not needed |
| NVIDIA H100 | 80 GB HBM3 (SXM) or 94 GB (NVL) | 3.35 TB/s (SXM); 3.9 TB/s (NVL) | NVLink 900 GB/s (SXM); 600 GB/s (NVL) | 70B-class inference with FP8, fine-tuning, training |
| NVIDIA H200 | 141 GB HBM3e | 4.8 TB/s | NVLink 900 GB/s | Large models on fewer GPUs, long context |
| NVIDIA B200 | 180 GB HBM3e (1,440 GB across 8 GPUs in DGX B200) | 64 TB/s across 8 GPUs | NVLink 1.8 TB/s per GPU | Largest models, training, high-throughput serving |
| AMD Instinct MI300X | 192 GB HBM3 | 5.3 TB/s peak | AMD Infinity Fabric (1.5 TB across 8 GPUs) | Large models on one GPU, ROCm software stack |
Two practical notes. First, FP8 math needs hardware support: Ada (L4, L40S), Hopper (H100, H200), Blackwell (B200) and CDNA 3 (MI300X) have it; Ampere (A100) does not, so FP8 checkpoints run there only through weight-only kernels with different speed. Second, memory bandwidth, not raw compute, usually limits how fast a single request generates tokens, because each new token reads the whole model from memory.
How to size GPU memory
Inference memory has three parts:
- Weights = parameters x bytes per parameter. FP16 or BF16 is 2 bytes, FP8 is 1 byte, INT4 is about 0.5 bytes plus a few percent for quantization scales.
- KV cache = 2 (keys and values) x layers x KV heads x head dimension x bytes per value, per token, multiplied by context length and by the number of sequences served at once. Servers such as vLLM allocate this in pages, so it is used efficiently, but it still has to fit.
- Overhead for the CUDA or ROCm context, activations and framework buffers. We assume 10 percent of weights plus KV cache; measure your own.
Leave headroom as well: vLLM by default uses 90 percent of GPU memory, so plan against 90 percent of the card.
Worked example: an 8B model
Assumptions: a model shaped like Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128, about 8 billion parameters), 8,192-token context, 16 sequences in flight.
KV cache per token at FP16: 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes, about 0.131 MB. Per sequence: 8,192 x 0.131 MB = 1.07 GB. For 16 sequences: 17.2 GB.
- FP16 weights, FP16 cache: 16 GB + 17.2 GB = 33.2 GB, plus 10 percent = about 36.5 GB. Fits one L40S (48 GB). Does not fit an L4.
- FP8 weights, FP8 cache: 8 GB + 8.6 GB = 16.6 GB, plus 10 percent = about 18.3 GB. Fits one L4 (90 percent of 24 GB is 21.6 GB).
- INT4 weights, FP16 cache: about 4.5 GB + 17.2 GB = 21.7 GB, plus 10 percent = about 23.9 GB. Too tight for an L4: quantizing weights alone barely helps, because the cache dominates.
Worked example: a 70B model
Assumptions: a model shaped like Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128, about 70 billion parameters), the same 8,192-token context and 16 sequences.
KV cache per token at FP16: 2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes, about 0.328 MB. Per sequence: 2.68 GB. For 16 sequences: 42.9 GB.
- FP16 weights, FP16 cache: 140 GB + 42.9 GB = 182.9 GB, plus 10 percent = about 201 GB. Needs several GPUs: for example 2 x H200 (282 GB), 2 x MI300X (384 GB) or 4 x H100 80 GB (320 GB), split with tensor parallelism.
- FP8 weights, FP8 cache: 70 GB + 21.5 GB = 91.5 GB, plus 10 percent = about 101 GB. Fits one H200 (127 GB usable at 90 percent), one MI300X or one B200; needs two H100 80 GB.
- INT4 weights, FP8 cache: about 37 GB + 21.5 GB = 58.5 GB, plus 10 percent = about 64 GB. Fits one H100 80 GB (72 GB usable).
Lower precision saves memory but can change output quality. Test FP8 and INT4 versions on your own evaluation set before committing to hardware. Our GPU memory calculator runs this arithmetic for your model, and the GPU memory guide explains each term in depth.
Fine-tuning and training need much more
Full fine-tuning with mixed-precision Adam holds weights, gradients, an FP32 master copy and two optimizer states: about 16 bytes per parameter before activations, as the ZeRO paper sets out. For an 8B model that is about 128 GB, so it spans two or more 80 GB GPUs or uses optimizer sharding. LoRA and QLoRA train small adapter matrices on a frozen (often quantized) base model and can fit an 8B fine-tune on one 24 GB to 48 GB card.
Interconnect: PCIe versus NVLink and SXM
When a model is split across GPUs, every generated token needs the GPUs to exchange data. How fast they can do that decides whether adding GPUs helps.
| Factor | PCIe cards (L4, L40S, H100 NVL, A100 PCIe) | SXM / OAM modules (A100, H100, H200, B200 SXM; MI300X) |
|---|---|---|
| GPU-to-GPU bandwidth | PCIe Gen4 64 GB/s or Gen5 128 GB/s; NVLink bridges pair 2 to 4 cards on some models | 600 GB/s to 1.8 TB/s per GPU across all GPUs in the server |
| Best for | One model per GPU, many independent replicas | One model split across 4 or 8 GPUs; training |
| Server shape | Standard servers with 1 to 8 cards | Purpose-built 4 or 8 GPU baseboards |
| Cost shape | Lower entry point | Higher, sold as whole servers |
Rule of thumb: if the model fits on one GPU, run replicas on PCIe cards and load-balance between them. If it needs tensor parallelism across four or eight GPUs, ask for an NVLink or Infinity Fabric baseboard. Multi-node training also needs a high-bandwidth fabric between servers, such as InfiniBand or RoCE; whether a supplier can provide it is confirmed per quote.
Storage and network for models and datasets
- Local NVMe for model weights and dataset shards in use. Loading a 140 GB checkpoint from slow storage turns every restart into a long wait.
- Object storage for datasets, checkpoints and finished adapters, kept apart from the GPU server so a rebuild does not lose them.
- Checkpoints are large. A full training checkpoint with optimizer states is about 16 bytes per parameter: about 1.1 TB for a 70B model. Size storage for several of them.
- Network transfer. Moving datasets in takes time. Estimate it before booking GPU time you cannot use yet.
Worked example
A 2 TB dataset over a 1 Gb/s link: 2 x 1012 bytes x 8 bits / 109 bits per second = 16,000 seconds, about 4.4 hours at full line rate. Over 10 Gb/s, about 27 minutes. Real transfers run slower; plan with the migration capacity tool.
Billing models compared honestly
GPU capacity is sold in three broad ways across the market. Each suits a different usage pattern.
| Model | How it works | Suits | Watch out for |
|---|---|---|---|
| On-demand (per second or hourly) | Start and stop instances; pay for time used | Experiments, bursts, short jobs | Highest unit rate; capacity may not be there when you want it |
| Reserved monthly or term | Commit to capacity for a month or longer at a lower effective rate | Steady inference, planned fine-tuning | You pay whether you use it or not |
| Dedicated server | A whole GPU server assigned to you for a term | Constant load, data control, custom stacks | Hardware failures need a replacement path; scaling means another server |
BroadNet Cloud quotes GPU capacity monthly or on a fixed term today. We do not offer per-second or hourly billing. If your GPUs would sit idle most of the month, on-demand elsewhere may cost less, and we will say so. The arithmetic: divide the monthly quote by the hours you will really use, and compare that with an hourly rate. The model-serving cost tool turns it into cost per million tokens.
Reference configurations
| Shape | GPUs | Host | Storage | Price |
|---|---|---|---|---|
| Small-model inference | 1 to 2 x L4 or L40S | Single socket, 128 to 256 GB RAM | NVMe for weights | Price shown at quote |
| 70B serving at FP8 | 1 x H200 or MI300X, or 2 x H100 | Dual socket, 512 GB+ RAM | NVMe for weights and cache | Price shown at quote |
| Fine-tuning node | 4 to 8 x A100 80 GB or H100 SXM | Dual socket, 1 TB+ RAM | Large NVMe scratch plus object storage | Price shown at quote |
| Large-model training | 8 x H200, B200 or MI300X per server, several servers | Supplier reference design | Shared parallel or object storage | Price shown at quote |
Use cases
Private chat assistant or RAG over company documents
An 8B to 70B model plus an embedding model, serving internal staff. Data stays on infrastructure you control, and load is steady during working hours. One H200 or MI300X at FP8, or a pair of L40S for a smaller model, is a common starting point.
Product feature with steady traffic
When API bills grow month on month, a reserved GPU often wins. Model the break-even with real token counts before switching, including the engineering time to run the stack.
Domain fine-tuning
LoRA on a single large-memory GPU for days, or full fine-tuning on an NVLink server. Datasets and checkpoints sit in object storage so the GPU server can be returned at the end of the term.
Batch generation and evaluation
Overnight runs that score or generate millions of records keep GPUs busy and benefit from high concurrency, which favors large memory for big KV caches.
GPU hosting compared with neighboring options
| Factor | Dedicated GPU server | GPU cloud instance | Hosted model API | CPU bare metal |
|---|---|---|---|---|
| Control of model and data | Full | Full within the instance | Provider's terms | Full |
| Cost shape | Fixed monthly or term | Monthly or term (from us) | Per token | Fixed monthly or term |
| Scaling | Add servers | Add instances, supplier capacity permitting | Instant | Add servers |
| Ops effort | Highest | High | Lowest | High |
| Best for | Steady heavy load, custom stacks | Steady load without whole servers | Low or uncertain volume | Embeddings and small models at low volume |
Software you run, and who is responsible for what
The serving stack is yours. Common choices: the NVIDIA CUDA toolkit or AMD ROCm as the platform; vLLM, Hugging Face Text Generation Inference (TGI) or NVIDIA Triton Inference Server for serving; PyTorch with your training framework of choice. We can install drivers and a base image if agreed in the quote, but model selection, licenses, tuning and the application are your responsibility.
| Area | BroadNet Cloud | Supplier | You |
|---|---|---|---|
| Facility, power, cooling, GPU hardware | - | Responsible | - |
| Sizing advice and quote accuracy | Responsible | - | Supplies model details |
| Hardware fault cases | Raises and tracks | Replaces | Reports |
| GPU drivers and base image | Confirmed at quote | - | Confirmed at quote |
| CUDA / ROCm stack, serving software | - | - | Responsible |
| Models, licenses, data, output | - | - | Responsible |
| Backups of checkpoints and data | If backup is bought | - | Otherwise you |
Support terms and how buying works
Every GPU quote states in writing: the GPU model and count, interconnect, host CPU and RAM, storage, network and transfer, supplier and region, the term and renewal rule, the hardware replacement path the supplier offers, support scope and hours, and how data is removed at the end. The general policy is on the support page.
- Size it. Run the GPU memory calculator with your model, context and concurrency.
- Request a quote. Send the result through the quote form or the configurator on pricing, with your preferred term and region.
- Check availability. We ask suppliers which matching GPUs they can provide, and come back with options if your first choice is not available.
- Accept the written quote. Supplier, hardware, price and terms are fixed in it.
- Verify on handover. Run
nvidia-smiorrocm-smiand a short load test before you put traffic on it.
Frequently asked questions
Can you get H100 or H200 GPUs?
They are among the classes we can quote. Whether a supplier can provide them in your region, and when, is confirmed in the quote; if not, we propose the nearest alternative.
Do you bill by the hour?
No. GPU capacity is quoted monthly or on a fixed term today. For short experiments, an hourly on-demand GPU cloud is usually the better choice.
Which GPU do I need for a 70B model?
At FP8 with moderate concurrency, about 100 GB: one H200, one MI300X or one B200, or two H100 80 GB. At FP16 you need around 200 GB across several GPUs. Run the calculator with your own context length and concurrency.
Should I choose AMD MI300X or NVIDIA?
MI300X offers 192 GB per GPU, so large models fit on fewer cards. It runs ROCm rather than CUDA; vLLM and PyTorch support it, but check every library you depend on before committing.
Do you provide the model or the serving software?
No. You choose and run the model and the stack, such as vLLM, TGI or Triton. We can install GPU drivers and a base image if the quote includes it.
Is NVLink necessary?
Only when one model is split across several GPUs or for multi-GPU training. If each model fits on one GPU, PCIe cards running independent replicas are simpler and usually cheaper.
Where do my prompts and data go?
Only to the server you rent and any storage you add. Nothing is sent to a model provider unless your own software sends it.
Sources
- NVIDIA L4 Tensor Core GPU, NVIDIA
- NVIDIA L40S GPU, NVIDIA
- NVIDIA A100 Tensor Core GPU datasheet, NVIDIA
- NVIDIA H100 Tensor Core GPU, NVIDIA
- NVIDIA H200 Tensor Core GPU, NVIDIA
- NVIDIA DGX B200, NVIDIA
- NVIDIA HGX platform, NVIDIA
- AMD Instinct MI300X accelerator data sheet, AMD
- The Llama 3 Herd of Models, Meta AI (arXiv)
- Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, arXiv
- vLLM documentation, vLLM project
- Text Generation Inference documentation, Hugging Face
- NVIDIA Triton Inference Server documentation, NVIDIA
- AMD ROCm documentation, AMD
Related
Last reviewed . Facts on this page are checked against the sources listed; anything not yet contracted is marked “confirmed at quote”.