Model routing and LLM gateways
A router decides which model handles each request: simple ones go to a small, cheap model, complex ones to a large or reasoning model, sensitive ones to a self-hosted model, and some don't need a model at all. A gateway sits between your app and all providers, giving one API with key management, rate limits, fallbacks, caching and logging. Routing is often the biggest single cost lever in an LLM system, and the router is a component that can be wrong, so evaluate it like any other.
Why route
If every request, including "hi" and "thanks", goes to your most expensive model, you pay frontier prices for greetings. Most traffic in most products is simple. Sending each request to the cheapest model that can handle it well cuts cost and often latency, without hurting quality on the hard requests.
Kinds of router
| Router | How it decides | Good for |
|---|---|---|
| Rules | Keywords, length, user tier, endpoint, file type | Clear, stable categories |
| Classifier | A small model labels the query (simple, standard, complex) | Mixed traffic with patterns |
| Semantic router | Compares the query's embedding with example queries for each route | Intent-based routing |
| Cascade | Try the cheap model, escalate if a check fails | When you can check answers cheaply |
| LLM router | A small LLM chooses the route, sometimes with reasons | Complex routing decisions |
Routing can go further than small versus large. It can pick a specific model for each kind of request: one for code, one for long documents, one for a language, one for sensitive data.
Routes that skip the model
Not every request needs an LLM:
- An exact or semantic cache hit.
- A direct answer from a database or an FAQ.
- A deterministic function, like a calculation or a status lookup.
A simple router
import re
ROUTES = {
"no_model": re.compile(r"^(hi|hello|thanks|thank you|ok)\W*$", re.I),
"self_hosted": re.compile(r"\b(salary|diagnosis|aadhaar|passport|medical)\b", re.I),
}
def route(query: str) -> str:
if ROUTES["no_model"].match(query.strip()):
return "no_model"
if ROUTES["self_hosted"].search(query):
return "self_hosted"
if len(query.split()) > 60 or any(w in query.lower() for w in ("compare", "analyse", "plan", "why")):
return "large_model"
return "small_model"
for q in ["thanks!", "What are your opening hours?", "Compare our Q2 and Q3 churn and explain why it changed", "Summarise my medical report"]:
print(f"{route(q):<12} <- {q}")
Rules like these are a fine start. When they stop being enough, label real queries and train or prompt a classifier.
Evaluating the router
Routing adds a decision step, which adds a little latency and a new way to fail: sending a hard question to a weak model.
- Label around 300 real queries as simple, standard or complex, by what model they need.
- Measure how often the router sends a complex query to the small model. That's the error that hurts quality.
- Measure how often it sends a simple query to the large model. That's the error that costs money.
- Track quality per route in production, not only overall.
Gateways
A gateway sits between your application and every model provider, and gives you:
- One API for many providers, often OpenAI-compatible.
- Key management, so applications don't hold provider keys.
- Rate limits and budgets per team, user or tenant.
- Fallbacks and retries across providers. See reliability patterns.
- Caching.
- Logging and cost tracking in one place.
Options include open-source proxies like LiteLLM, API gateways like Kong with AI plugins, and managed gateways from cloud and AI infrastructure providers.
The router and the gateway are different jobs, even when one tool does both: the gateway is the single door to all models, and the router decides which model each request should use.
Measuring the result
- Share of traffic per route.
- Quality (eval pass rate) per route.
- Cost saved compared with sending everything to the large model.
- Latency added by the routing step.
From the floor
In the production pipeline I teach, the dynamic routing engine sits right after the cache: if the query is simple, there's no need to load a large model on GPUs, so it goes to a small language model that can even run on a CPU, and only complex queries reach a large reasoning model. A builder asked whether an intent router adds latency, and my answer was yes, it does, so you test different solutions, and if you route correctly, choosing between a small model, a large one or a specific model for that kind of query, you reduce the overall cost and latency of the system.
In my webinar for leaders, I called the same idea a hybrid fleet: a model router decides between a powerful API, a fast and cheap open-source model, and a secure fine-tuned model, and when a builder asked how to create the router, my answer was that it depends on the queries, with if-else rules, a smaller fine-tuned model or an agent as options.