AI system design interview: cut your LLM bill in half
Measure first: attribute tokens and cost by feature, model, prompt version and user, because a few features or users usually drive most of the spend. Then apply levers from cheapest to most expensive to build: prompt caching, response caching, trimming context, capping output, routing to smaller models, batch APIs and budgets for agent loops, and self-host or fine-tune only for one narrow, high-volume task. Run your evals after every change, and report cost per successful task, not cost per call.
Question
Your LLM spend doubled this quarter. Cut it in half without lowering quality.
First, measure
Attribute every call: feature, model, prompt version, input and output tokens, cached tokens, user or tenant. Then answer three questions: which features cost the most, which prompts grew, and whether a few users or agent loops account for a large share. Doubled spend often has one cause, a new feature, a longer prompt, or an agent that loops.
Levers, cheapest first
| Lever | Typical effect | Risk to quality |
|---|---|---|
| Prompt caching for stable prefixes | Lower input cost on cached tokens | None |
| Exact and semantic response caching | Removes repeat calls | Wrong hits if thresholds are loose |
| Trim system prompt, history and retrieved context | Fewer input tokens | Missing context |
| Cap output length | Fewer output tokens, which cost more per token | Truncated answers |
| Route simple requests to smaller models | Large savings on simple traffic | Misroutes |
| Batch APIs for work that can wait | Lower price per token | Delay |
| Budgets and step limits for agents | Stops runaway loops | Tasks cut short |
| Self-host or fine-tune a small model for one task | Lower cost at high volume | Engineering and maintenance cost |
Guarding quality
Run the eval suite after each change and watch online metrics for a week. A saving that drops task success just moves the cost to support or rework.
Report it as cost per successful task
If a cheaper model costs half as much per call but fails twice as often and needs retries, you've saved nothing. Cost per successful task captures that.
What interviewers listen for
Measurement before action, cost per successful task as the metric, and a quality gate on every saving.
From the floor
In the leaders session on cloud APIs vs open-source LLMs, I described the hybrid fleet: routine questions go to a cheap API, while a model fine-tuned on a firm's own data handles the work where that data gives an edge, which is the same idea as routing for cost, applied as a decision about where each kind of request should run.