LLMOps and production for AI engineers
LLMOps is the work of shipping and running LLM systems in production: serving the model, keeping latency and cost under control, tracing every request, testing every change, releasing safely and responding when something breaks. A model call that works in a notebook is a small part of a production system. The rest is the operations around it, and that is where most AI projects succeed or stall.
What changes in production
On your laptop, you have one user (you), a handful of test questions, a model that never times out, and nobody watching the bill. In production, you have:
- Real users who ask things you never tested, at the same time, at peak hours.
- Real data, with personal information that has to be protected.
- Real failures: provider outages, slow tools, malformed outputs.
- A real bill that grows with every token.
- Changes every week: prompts, models, documents, code.
LLMOps is the set of practices that make all of that manageable.
MLOps and LLMOps
MLOps grew up around training and serving your own models: data pipelines, training runs, model registries, drift detection. LLMOps keeps most of that, and adds what's specific to LLM applications:
| MLOps | LLMOps adds |
|---|---|
| Model versions | Prompt versions, and provider model versions you don't control |
| Accuracy on a test set | Evals with validated judges, because outputs are open-ended |
| Feature pipelines | Retrieval pipelines, indexes and context assembly |
| Prediction latency | Time to first token, streaming, token budgets |
| Compute cost | Cost per token, per request and per task, which can change with every prompt edit |
| Model monitoring | Tracing of multi-step chains and agents, and guardrails on inputs and outputs |
Production loop
- Build and test locally, with evals from the start.
- Containerise, so the app runs the same everywhere.
- Run tests and evals on every change in CI.
- Release gradually, with a fast rollback.
- Trace every request, monitor quality, latency and cost.
- Turn what you learn in production into new tests and better data.
What this section covers
| Page | The question it answers |
|---|---|
| Notebook to production checklist | What do I need before real users touch this? |
| Inference serving | Hosted API, managed endpoint or my own server? |
| Inference optimization | Why is serving slow, and how do I speed it up? |
| Latency and streaming | Where does the time go, and how do I make it feel fast? |
| Semantic caching | How do I avoid paying twice for the same answer? |
| Model routing and gateways | Which model should handle each request? |
| Observability and tracing | What happened in this request? |
| Cost monitoring | Where is the money going? |
| CI/CD for AI | How do I test and ship changes safely? |
| Rollouts and model upgrades | How do I change models without breaking things? |
| Incident response | What do I do when the AI misbehaves? |
| Data flywheels | How does the product improve after launch? |
| Run LLMs locally | How do I run models on my own hardware? |
From the floor
In the customer support agent deployment session, I explained that deployment follows the same phases for every project, even when the tools change: build locally, containerise with Docker, add CI/CD, orchestrate the artifacts, monitor, and define the infrastructure, and that every project we deploy gets its own repository.