AI system design interview: design a model router
Put a gateway in front of every model call, classify each request with rules first and a small classifier where rules aren't enough, send simple requests to a small model, hard ones to a large or reasoning model, and sensitive ones to a self-hosted model as a hard rule. Add a cascade that escalates when a cheap model's output fails validation, and evaluate the router on labelled requests, measuring savings against an all-large-model baseline.
Question
Design a router that sends each request to the right model to cut cost without hurting quality.
Clarify
- Available models, their prices and latency.
- Privacy rules: which data can't go to external APIs.
- Traffic mix: how many requests are simple lookups vs multi-step reasoning.
- Latency budget for the routing decision itself.
Routing strategies
| Strategy | How it works | When it fits |
|---|---|---|
| Rules | Route by feature, tenant, data class, input length | Always first: free and predictable |
| Classifier | A small model or embedding similarity to labelled examples | Mixed traffic within one feature |
| Cascade | Try the cheap model, validate, escalate on failure | When outputs can be checked automatically |
| LLM as router | A model decides | Rarely worth its own latency and cost |
Architecture
A gateway receives all calls. Rules route by data class first, so sensitive requests go to the self-hosted model whatever the classifier says. A classifier labels the rest as simple, standard or complex, and each label maps to a model. For structured outputs, a cascade validates the cheap model's answer against the schema and business rules, and escalates on failure. Every decision is logged with the route and reason.
Deep dive: evaluating the router
Label 500 real requests with the cheapest model that answers each one correctly. Measure misroutes in both directions, and weight hard requests sent to weak models more, because those hurt users, while easy requests sent to strong models only cost money.
Failure modes
| Failure | Mitigation |
|---|---|
| Router latency eats the savings | Rules and small classifiers, not an LLM call |
| Traffic drifts and routes go stale | Re-label a sample monthly, watch quality per route |
| Sensitive data sent outside | Data-class rules before any classifier, tested in CI |
| Quality drops silently on the cheap route | Online sampling and evals per route |
Metrics
Traffic share per route, quality per route, cost saved against the all-large baseline, and router latency.
What interviewers listen for
That the router is evaluated, privacy routing is a hard rule, and savings are measured against a baseline.
From the floor
In the leaders session on cloud APIs vs open-source LLMs, I drew the flow as a user query going into a model router that decides whether a request goes to an API, to an open-source model we host, or to an on-premise model fine-tuned on a law firm's own data, and when a participant asked how to create the router, I said it depends on the query: you can start with if-else rules, fine-tune a smaller model to decide, or use an agent.