Hybrid fleet pattern: APIs, open models and fine-tuned models behind one router
A hybrid fleet runs several kinds of models behind one router: a powerful hosted API for general and complex requests, a fast, cheap open model for simple ones, and a secure fine-tuned model on your own infrastructure for sensitive or specialised ones. The router picks per request, so you get cost savings, privacy and quality where each one matters. It replaces the "API or open source" argument with a different question: which model is right for this request?
Why one model isn't enough
Teams argue "API vs open source" as if it were one choice for the whole company.
- Choose only a hosted API, and you fail your privacy requirements for sensitive data, and pay frontier prices for greetings.
- Choose only self-hosting, and you fail on quality for complex general tasks, or on budget for GPUs sized for peak traffic.
Most products have a mix of requests, and each kind has a different best model.
Fleet
| Request type | Route | Why |
|---|---|---|
| Greeting, simple FAQ | Cache or a small open model | Cheapest, fastest |
| General writing, complex reasoning | Hosted API | Best quality for open-ended tasks |
| Sensitive data, regulated domain | Self-hosted fine-tuned model | Data stays inside, domain accuracy |
| Narrow high-volume task (classification, extraction) | Small fine-tuned or distilled model | Low cost at volume |
Router
The router decides per request. It can be rules, a classifier, a semantic router or a small model, and it must be evaluated like any other component: measure how often it sends a sensitive request to the external API, and how often it sends a hard question to the small model. See model routing and gateways.
For privacy, make the safe direction the default: if the router isn't sure whether a request contains sensitive data, it should go to the self-hosted model, not the external API.
What you need to run it
- A gateway with one API in front of all models, so the application doesn't care which model answered.
- An inference server for the self-hosted models. See inference serving.
- Evals per route, because each model fails differently.
- Cost and quality tracked per route.
- Fallbacks within each tier, and rules for which fallbacks are allowed (a sensitive request should never fall back to an external API).
Getting started
Classify a week of your traffic into the three or four buckets above, and estimate the cost and privacy impact of each route compared with sending everything to one model. If one bucket is 90% of traffic, start by routing just that one.
Measuring it
- Cost per request by route, against a single-model baseline.
- Quality per route.
- Routing errors, especially sensitive requests that reached an external API.
From the floor
This is the pattern I recommended in my leaders webinar on cloud APIs vs open-source models, with a law firm as the example: a question like "how are you?" goes to a free or cheap external API, because there's no reason to use your own GPUs for it, and a complex legal question goes to the model the firm fine-tuned on its own 20 years of case data, hosted on-premises where only its own people can reach it. I summarised it as a model router covering three things: the powerful API, the fast and cheap open-source model, and the secure fine-tuned model.