How to choose an LLM for your product
Don't choose a model from a leaderboard. Write down what your task needs (quality, latency, cost, privacy, context length, languages, tools), use public benchmarks and price lists only to build a shortlist of 3 to 5 models, then run all of them on 30 to 100 real examples from your own use case and compare quality, failure rate, latency and cost per request. Pick the cheapest model that clearly meets your quality bar, keep a fallback, and repeat the test whenever a new model comes out.
Start from the task, not the model
Most teams start by asking which model is best. That question has no answer on its own, because "best" depends entirely on the job.
The model that writes the best essay may be too slow for a support chat. The model that tops a coding benchmark may cost twenty times more than one that handles your simple extraction task perfectly well. And for some tasks, the best model is no model at all, because a script or a normal workflow solves the problem with less cost and less maintenance.
So the first step is to describe the job.
Write down your requirements
Answer these before you look at any model.
- Task: what exactly does the model do? Classify a ticket, extract fields from an invoice, answer questions from documents, write code, run an agent with tools?
- Quality bar: what counts as a correct answer, and how many mistakes are acceptable? A wrong product suggestion and a wrong medication dose are very different failures.
- Latency: how fast does it need to respond? A chat needs the first token in about a second, while a nightly batch job can take minutes.
- Volume: how many requests per day now, and in a year?
- Budget: how much can each request cost, given what the feature is worth?
- Data: can this data leave your infrastructure? Are there residency rules about which country it is processed in?
- Context: how long are the inputs, including documents and history?
- Languages: which languages do your users write in?
- Modalities: text only, or images, audio, PDFs?
- Capabilities: does it need structured output, tool calling, long context, or reasoning?
Some of these remove options immediately. If the data can't leave your servers, closed API models are out unless you have an agreement for a private deployment. If you need a response in under a second for millions of requests a day, the largest reasoning models are out.
Build a shortlist
Now use public information to find 3 to 5 candidates.
Leaderboards and benchmarks
Benchmarks test models on standard tasks, like general knowledge, maths, coding or following instructions. Sites like LMArena, where people vote on which of two anonymous answers they prefer, and Artificial Analysis, which tracks quality, speed and price across providers, are good places to see the landscape.
They are useful for a shortlist and not for a decision, for several reasons:
- Your task is not their task. A model that is great at competition maths can be average at extracting fields from your invoices.
- Benchmark questions can leak into training data, so scores can be higher than real ability.
- Voting leaderboards reward answers people like, which isn't always the same as answers that are correct.
- Providers report their own results under conditions that favour their model.
And one impressive example in a demo or a social media post is an observation, not a ranking. It tells you a model can do something once, not how often it does it on your kind of input.
Provider documentation
Check each candidate's documentation for the facts that rule models in or out: context window, maximum output, supported languages, structured output and tool calling support, data retention terms, available regions, and rate limits for your account tier.
Price
Look at the input and output price per million tokens, and estimate the cost for your typical request, as the pricing page shows. A model that is 10 times cheaper per token but needs 3 times more tokens to do the job (for example a reasoning model that thinks a lot) is not 10 times cheaper.
Build your own evaluation set
This is the step that decides. You need a set of examples that represent what your application will really see.
- Collect 30 to 100 real inputs. Use real user questions, real documents and real tickets if you have them, with sensitive details removed.
- Include the hard cases on purpose: long inputs, messy formatting, other languages, ambiguous requests, questions the model shouldn't answer, and inputs that tried to break your system.
- Write down what a good output looks like for each, either the exact expected answer, or the facts and fields it must contain.
- Decide how you will score: exact match for fields, code-based checks for format, another model as a judge for open-ended answers (checked against your own judgement on a sample), and human review for the cases that matter most.
This set is useful far beyond choosing a model. It becomes the test you run every time you change a prompt, a retrieval setting or a model version, which is the core of evals and reliability.
Run the comparison
Run every shortlisted model on the whole set.
- Use the same prompt for all models first, then allow small model-specific adjustments, because a prompt written for one model can underrate another.
- Run each example more than once, because outputs vary, as the nondeterminism page explains.
- Use realistic settings: the same temperature, maximum tokens and tools you will use in production.
Measure these for each model:
- Quality: the share of examples that meet your bar.
- Failure rate: invalid JSON, refusals, cut-off answers, wrong tool calls.
- Latency: time to first token and total time, at the median and at the slow end (the 95th percentile), because users notice the slow requests.
- Cost: actual tokens used per request multiplied by the price.
A scorecard
Put the results in one table, so the trade-offs are visible.
| Model | Quality (pass rate) | Failures | Median latency | 95th percentile latency | Cost per 1000 requests | Data stays in-house |
|---|---|---|---|---|---|---|
| Model A | ||||||
| Model B | ||||||
| Model C |
Then decide which columns matter most for this feature. For a support chat, latency and cost may weigh as much as a few points of quality. For a legal or medical feature, quality and data control come first, and cost is secondary.
Making the decision
A good default is to choose the cheapest, fastest model that clearly passes your quality bar, not the model with the highest score. The gap between the top model and a cheaper one is often small on a specific task, and the cost difference is often large.
If no single model fits every type of request, consider routing: a small, cheap model for simple requests and a stronger one for the hard cases, with a classifier or simple rules deciding which goes where. That is the hybrid fleet idea from the model families page.
Plan for the model to change
Whatever you choose, it won't be your model forever. New models come out every few months, prices drop, and providers retire older versions.
- Keep the model name in configuration and call models through one module in your code, so switching is a configuration change, not a rewrite.
- Pin a specific model version in production, and upgrade deliberately.
- Keep a tested fallback model from a different provider.
- When a promising new model is released, run your evaluation set on it before believing the announcement.
Common mistakes
- Choosing the model before defining the task.
- Testing with five handmade examples that all look alike.
- Comparing models on quality only, and discovering the cost and latency in production.
- Treating one benchmark, one demo or one good answer as proof that a model is better at everything.
- Never re-testing after the decision, so the application is stuck on an old, expensive model.
From the floor
In the Build30 tokenization session, I opened Artificial Analysis to show builders how models compare on quality, speed and price at the same time. It is a good place to start a shortlist, and the decision still has to come from running the shortlisted models on the problem in front of you.
When I write the weekly AI roundup, I follow the same rule I give builders here: if I have only read a release announcement, I say the company reports the result, and if I tested the model, I say what I tested and what happened. One good result is an observation, never a ranking.