LLM pricing math: cost per request, per user and per month
LLM APIs charge per token, with separate prices for input tokens (everything you send) and output tokens (everything the model writes), and output usually costs several times more than input. The cost of one request is input tokens times the input price plus output tokens times the output price, divided by a million. The numbers that surprise people are the hidden inputs: the system prompt, the conversation history that is resent on every turn, retrieved documents, tool definitions and the many calls an agent makes for one task, so estimate those before you build and log real usage from the first day.
How LLM APIs charge
Every provider publishes prices per million tokens, and the structure is similar everywhere.
- Input tokens: everything in the request, including the system prompt, history, documents, tool definitions and the user's message.
- Output tokens: everything the model generates. This is usually several times more expensive than input, because generating tokens one by one uses more compute than reading them.
- Cached input tokens: if the start of your prompt is identical to a recent request, many providers charge much less for that cached part.
- Reasoning tokens: for reasoning models, the hidden thinking tokens are billed as output tokens, even though you may never see them.
- Images, audio and documents are converted into tokens too, with their own counting rules.
- Batch processing: many providers offer a discount, often around half price, for requests you don't need answered immediately, processed within hours.
Prices change often, and usually go down, so always check the provider's current pricing page. The examples on this page use made-up round prices to show the math.
A subscription is not the API
People sometimes estimate costs from the monthly subscription for a chat app, like a ChatGPT or Claude plan. That subscription is a product for one person using the app. When you build an application, you pay for the API separately, per token, and the bill depends entirely on how much your application sends and receives.
Cost of one request
The basic formula is:
cost = (input tokens × input price + output tokens × output price) ÷ 1000000
def request_cost(input_tokens: int, output_tokens: int, input_price: float, output_price: float, cached_tokens: int = 0, cached_price: float = 0.0) -> float:
uncached = input_tokens - cached_tokens
return (uncached * input_price + cached_tokens * cached_price + output_tokens * output_price) / 1_000_000
print(round(request_cost(2000, 400, input_price=1.00, output_price=4.00), 5))
With an example price of $1 per million input tokens and $4 per million output tokens, a request with 2000 input tokens and 400 output tokens costs $0.0036. That looks like nothing, and that is exactly why LLM costs surprise people: the cost is in the multiplication by requests, turns, users and days.
Conversation history is resent every turn
A model has no memory, so a chat application sends the whole conversation again with every new message. You pay for every earlier message again on every turn.
Take a very small example. You ask "How are you?" and the model replies "I am fine, what about you?". Then you ask "What is an apple?". The second request contains the first question, the first answer and the new question, so you pay for all of them as input, not only for your new four words.
Over a longer conversation, this grows fast.
def conversation_cost(turns: int, system_tokens: int, user_tokens: int, reply_tokens: int, input_price: float, output_price: float) -> float:
total, history = 0.0, 0
for _ in range(turns):
input_tokens = system_tokens + history + user_tokens
total += request_cost(input_tokens, reply_tokens, input_price, output_price)
history += user_tokens + reply_tokens
return total
for turns in [1, 5, 10, 20, 40]:
print(turns, round(conversation_cost(turns, 800, 60, 250, 1.00, 4.00), 4))
With an 800 token system prompt, 60 token messages and 250 token replies, a 10 turn conversation costs about 1.75 times as much as 10 separate one-turn questions would, and a 40 turn conversation costs more than 4 times as much, because the input grows on every turn. The history part of the cost grows roughly with the square of the number of turns, so the longer the conversation, the faster it adds up.
This is why context management, trimming and summarising old turns, matters for cost as well as quality.
RAG requests
In a retrieval-augmented generation system, every question also carries the retrieved chunks. If you add 8 chunks of 400 tokens each, that is 3200 extra input tokens per question, often far more than the question itself.
Retrieving fewer but better chunks, with a reranking step, usually improves answers and cuts cost at the same time.
Agents
An agent makes several model calls for one task: it plans, calls a tool, reads the result, decides the next step, and repeats. Each of those calls includes the system prompt, the tool definitions, and everything that has happened so far in the task.
So an agent task can easily use 10 to 50 times the tokens of a single question. Ten tool definitions with detailed descriptions can add a few thousand tokens to every single call. Put a limit on the number of steps and on the spend per task, and log the cost per task, not only per call.
Cost per user per month
To estimate a monthly bill, work from how people will use the feature.
monthly cost = active users × sessions per user per month × requests per session × average cost per request
For example, with example prices:
| Assumption | Value |
|---|---|
| Active users | 2000 |
| Sessions per user per month | 12 |
| Requests per session | 6 |
| Average input tokens per request (system prompt, history, chunks) | 4000 |
| Average output tokens per request | 350 |
| Example price | $1 input and $4 output per million tokens |
Each request costs (4000 × 1 + 350 × 4) ÷ 1000000 = $0.0054. The monthly total is 2000 × 12 × 6 × 0.0054, about $778, or about $0.39 per active user.
Then compare that per-user number with what the feature is worth: the subscription price, the support time it saves, or the revenue it brings. If the AI feature costs more per user than the user pays, it needs a different design, a cheaper model, or a different price.
Costs people forget
The model calls in the main flow are not the only costs.
- Retries and failures: failed requests, invalid outputs that need a repair call, and timeouts that were still billed.
- Evaluation: running your test set on every prompt change, on several models, several times each.
- Embeddings: embedding all your documents, re-embedding them when you change the model, and embedding every query.
- Vector database and storage.
- Logging and tracing tools, which are often priced by volume.
- Development and testing: your team's own usage while building can be a real share of the bill.
- Guardrail and routing models that run before or after the main model.
How to reduce cost
Roughly in order of impact:
- Use a smaller model where it is good enough, and route only the hard requests to a bigger one.
- Send less: trim the system prompt and tool descriptions, summarise history, retrieve fewer chunks.
- Use prompt caching by keeping the long, stable part of the prompt identical and at the start. See prompt caching.
- Limit output: set a maximum output length, and ask for concise answers or structured output instead of long prose when code is the reader.
- Cache full responses for repeated questions.
- Use batch processing for work that doesn't need an immediate answer, like nightly document processing.
- Avoid reasoning models for tasks that don't need reasoning.
API or your own hardware?
At some volume, running an open-weight model on your own GPUs can become cheaper than paying per token. Where that point is depends on many numbers: the model size, the GPUs needed, how busy you can keep them, the engineers needed to run them, and the API price you compare against.
The shape of the comparison is predictable. API cost grows in a straight line with usage. Self-hosting has a large fixed cost, the hardware, the team and the operations, and then grows much more slowly. At low volume, the API is almost always cheaper, and at very high, steady volume, self-hosting can win, if you can keep the hardware busy.
Always include people in the self-hosting side of the calculation. The GPUs are often not the biggest cost, the engineers who deploy, monitor and update the model are.
Track real cost from day one
Estimates are a starting point. Real usage always differs, so log the actual token counts the API returns for every call, with the model, the feature and the user or tenant, and look at cost per feature and per user every week. A token tracker that writes one line per call to a JSONL file, as on the API page, is enough to start. Set budget alerts with your provider so a bug that loops can't run up an unlimited bill.
From the floor
In the webinar on cloud APIs versus open-source LLMs, I showed a small total cost of ownership calculator I built with a coding assistant. You enter the number of tokens per month, the API price, the GPUs, the engineers and their salaries, and it plots both cost curves. At half a million tokens a month, the API was clearly cheaper, and as we moved the volume up to billions of tokens, the self-hosted line, which starts high and stays flat, crossed the API line, which is the shape every company should look at before deciding.
In the same webinar, I used the OpenAI tokenizer to show that a short sentence of four words was nine tokens, and then walked through a two-turn conversation to show that the second question carried the first question and answer with it, so you pay for the history again on every turn.