Cost monitoring and FinOps for AI applications
Attribute every token to a feature, a user or tenant, a model and a prompt version, so you can see where the money goes. Set budgets and alerts per feature and per day, review the top cost drivers every week, and fix them with caching, routing to smaller models, prompt trimming and output limits. Report cost per successful task, not only total spend, because a cheaper model that fails more often can cost more per task.
Why LLM costs surprise teams
The monthly bill doubles and nobody knows which feature caused it. That happens because LLM cost depends on things that change every day: how many users, how long their conversations get, how big the prompts grew after the last edit, how many steps an agent took, which model each request reached.
And as AI gets cheaper per token, usage tends to grow faster than prices fall, so total spend often rises anyway. That's the Jevons paradox applied to tokens. Visibility is what keeps that growth tied to value.
Track tokens from the first call
Log, for every model call:
- Input tokens, cached input tokens and output tokens (and reasoning tokens, if the model reports them).
- The model and its price at the time.
- The feature or endpoint.
- The user or tenant.
- The prompt version.
- The task or session it belongs to, and whether the task succeeded.
Dimensions that matter
| Dimension | Why |
|---|---|
| Feature or endpoint | Find the expensive features |
| Tenant or user | Pricing, abuse, fair use limits |
| Model | Routing opportunities |
| Prompt version | Catch prompts that grew longer |
| Cached vs uncached tokens | Check that caching works |
| Success vs failure | Cost per successful task |
A simple cost tracker
from collections import defaultdict
PRICES = {"small": (0.10, 0.40), "large": (2.00, 8.00)}
class CostTracker:
def __init__(self) -> None:
self.by_feature = defaultdict(float)
self.tasks = defaultdict(lambda: {"cost": 0.0, "success": None})
def record(self, feature: str, task_id: str, model: str, tokens_in: int, tokens_out: int) -> None:
price_in, price_out = PRICES[model]
cost = (tokens_in * price_in + tokens_out * price_out) / 1_000_000
self.by_feature[feature] += cost
self.tasks[task_id]["cost"] += cost
def finish(self, task_id: str, success: bool) -> None:
self.tasks[task_id]["success"] = success
def cost_per_successful_task(self) -> float:
done = [t for t in self.tasks.values() if t["success"] is not None]
wins = sum(t["success"] for t in done)
return sum(t["cost"] for t in done) / wins if wins else float("inf")
tracker = CostTracker()
for i in range(100):
model = "small" if i % 4 else "large"
tracker.record("support_chat", f"t{i}", model, tokens_in=2500, tokens_out=400)
tracker.finish(f"t{i}", success=i % 10 != 0)
for feature, cost in tracker.by_feature.items():
print(f"{feature}: ${cost:.3f} total")
print(f"cost per successful task: ${tracker.cost_per_successful_task():.5f}")
The prices are placeholders per million tokens. Use your provider's current prices, and update them when they change.
Budgets and alerts
- A budget per feature per day, with an alert at 80% and a hard limit at 100%.
- A limit per user or tenant, so one account (or one bug) can't spend everything.
- An alert on unusual spikes compared with the same hour last week.
- A spending limit at the provider level, as the last line of defence.
Weekly review
Look at:
- Daily cost by feature.
- Cost per successful task, trending over releases.
- The 10 most expensive requests of the week, and why they were expensive.
- Cache hit rates.
- Share of traffic per model.
The 10 most expensive requests usually point to a fix: a loop in an agent, a document stuffed into every prompt, a missing output limit.
Levers, roughly in order
- Caching: skip the model for repeated questions (semantic caching) and use prompt caching for shared prefixes.
- Routing: send simple requests to small models (routing).
- Trimming: shorter system prompts, fewer and smaller retrieved chunks, compressed history.
- Output limits: concise instructions and maximum tokens.
- Batch APIs for work that doesn't need an immediate answer, often at a lower price.
- Fewer calls per task: combine steps, and limit agent loops (agent cost control).
- Self-hosting or fine-tuning a smaller model, when volume justifies it.
From the floor
On day 1 of Build30, we made the first LLM call and built a token tracker in the same session, because tokens have limits on free tiers and paid APIs charge per token, so tracking them from the first call is how you understand what your app will cost.
In the customer support agent, I explained the router as a cost decision first: LLM inference is the most expensive part of the process, so similar queries go to the cache, and other queries go to external sources or a smaller model where possible, to limit the queries the LLM is generating.