How the cost is calculated
| Monthly cost | GPU price per hour × GPUs × 730 hours |
|---|---|
| Tokens per month | throughput × 3,600 × 730 × utilization |
| Cost per 1M tokens | monthly cost ÷ tokens per month × 1,000,000 |
| Break-even volume | monthly cost ÷ API price per 1M tokens |
730 hours is the average month (8,760 hours ÷ 12). Dedicated or reserved GPUs are paid for whether busy or idle, which is why utilization dominates the result.
Worked example
Worked example (hypothetical inputs)
Two GPUs at 3.00 per hour each cost 6.00 per hour, or 4,380 per 730-hour month. At 1,000 tokens per second and 50% utilization the replica serves 1,000 × 3,600 × 730 × 0.5 = 1,314 million tokens a month. Cost per million tokens: 4,380 ÷ 1,314 = 3.33. Against an API price of 5.00 per million, break-even is 4,380 ÷ 5 = 876 million tokens a month.
How to measure throughput honestly
- Fix the workload. Use your real prompt and output lengths; throughput at 500-token prompts says little about 30,000-token prompts.
- Sweep concurrency. Measure at 1, 8, 32 and 64 concurrent requests and note latency at each; pick the highest concurrency that still meets your latency target.
- Count output tokens. Report output tokens per second, and track input tokens separately if prefill is significant.
- Repeat. Three runs, report the median, and record the software versions.
Our benchmark method describes the GPU test we will run on GPU plans before publishing any throughput figure.
Assumptions and limits
- Excludes storage, network egress, engineering time and idle replicas kept for redundancy. Add them with the TCO calculator.
- API prices often differ for input and output tokens; enter a blended figure for your traffic mix.
- Use one currency for every input.
How we test this tool
Unit tests pin the worked example above (3.333 per million tokens, 4,380 per month, 1,314 million tokens, break-even 876 million) and reject zero GPUs or utilization above 100%.
Frequently asked questions
Why is my cost per token so high at low traffic?
Because you pay for the GPU by the hour whether it is busy or not. At 5% utilization the same GPU costs eight times more per token than at 40%.
Should I use an API or self-host?
APIs win at low or unpredictable volume. Self-hosting tends to win at steady high volume, when data must stay on infrastructure you control, or when you need a model the APIs do not offer.
Related
Inputs stay in your browser. Tool maths last reviewed ; the unit tests that pin these formulas are listed in the evidence hub.