Capacity planning and cost modelling for LLM systems
Start from users: daily active users times requests per user gives requests per day, and the peak-hour share gives peak requests per second. Multiply by tokens per request for token volume and API cost, or, if self-hosting, size GPU memory for the weights plus the KV cache, and divide peak token throughput by measured tokens per second per GPU for the number of GPUs. Then change one assumption at a time, because token count per request and peak concurrency usually matter more than the model's price.
Why model it before building
Designs get approved with no idea whether they need 2 GPUs or 40, or cost $500 or $50,000 a month. A one-page model with your assumptions written down changes the conversation: everyone can see which assumption drives the cost, and challenge it.
Model
requests_per_day = daily_active_users × requests_per_user
peak_rps = requests_per_day × peak_hour_share / 3600
tokens_per_day = requests_per_day × (input_tokens + output_tokens)
api_cost_month = 30 × requests_per_day × (input_tokens × input_price + output_tokens × output_price)
gpu_memory = weights + KV cache at peak concurrency + overhead
gpus_needed = peak output tokens per second / measured tokens per second per GPU, plus headroom
A worked example
def plan(dau: int, req_per_user: float, peak_share: float, tok_in: int, tok_out: int,
price_in: float, price_out: float, tps_per_gpu: float, out_tps_per_request: float) -> dict:
per_day = dau * req_per_user
peak_rps = per_day * peak_share / 3600
api_month = 30 * per_day * (tok_in * price_in + tok_out * price_out) / 1_000_000
seconds_per_answer = tok_out / out_tps_per_request
concurrent = peak_rps * seconds_per_answer
gpus = max(1, round(concurrent * out_tps_per_request / tps_per_gpu * 1.3 + 0.5))
return {"requests/day": round(per_day), "peak rps": round(peak_rps, 1),
"concurrent at peak": round(concurrent), "API $/month": round(api_month), "GPUs": gpus}
base = dict(dau=20_000, req_per_user=6, peak_share=0.15, tok_in=2_000, tok_out=300,
price_in=0.40, price_out=1.60, tps_per_gpu=2_500, out_tps_per_request=40)
print("base: ", plan(**base))
print("prompt x2: ", plan(**{**base, "tok_in": 4_000}))
print("users x5: ", plan(**{**base, "dau": 100_000}))
print("peak share 30%:", plan(**{**base, "peak_share": 0.30}))
The prices and throughput numbers are placeholders. Measure tokens per second per GPU on your own model, settings and prompt lengths, because published numbers rarely match your workload.
Reading the results:
- Doubling the prompt length raises the API cost by more than 60%, because input tokens are most of the bill, but it doesn't change the GPU count here, because GPUs here are sized by output throughput at peak. Long prompts still increase time to first token and KV cache memory.
- More users scale both.
- A sharper peak doesn't change the API bill, and it does change how many GPUs you need, because self-hosted capacity must cover the peak.
GPU memory
Before throughput, check that the model fits:
- Weights: parameters × bytes per parameter. A 70 billion parameter model needs about 140 GB at 16-bit and about 35 to 42 GB at 4-bit. See quantization.
- KV cache: grows with context length and concurrent requests, and can be as large as the weights. See inference optimization.
- Overhead: activations, the inference engine, fragmentation.
If weights plus the KV cache at your peak concurrency don't fit, you need more GPUs, a smaller or more quantized model, or shorter contexts.
Hidden costs
The model isn't the only cost:
- Engineers to run and maintain self-hosted infrastructure.
- Monitoring, tracing and eval runs.
- Vector database, cache and storage.
- Re-indexing and re-embedding when documents or models change.
- Idle GPUs outside peak hours.
Sensitivity
Build the model in a spreadsheet with assumptions in one column and results in another, and add one row per assumption showing the result when that assumption doubles. The rows that move the result most are the assumptions to measure first.
After the first month of real usage, compare the model with reality and update it.
From the floor
In my webinar for leaders on cloud APIs vs open-source models, total cost of ownership was the second chapter, and I showed a small calculator I built with a coding assistant: you enter how many million tokens you expect, and it compares the API cost with the cost of running your own model, including the hidden costs of changes, monitoring and maintenance.