From notebook to production: the LLM app checklist
Every production AI project goes through six phases: build and test locally, containerise, set up CI/CD, manage artifacts (models, indexes, prompts) with versions, add monitoring, and define the infrastructure as code. On top of that, an LLM app needs an eval suite, input and output guardrails, request tracing, cost limits and a rollback plan before real users touch it. Production readiness is a list, not a feeling, so write the list and check it before launch.
Why demos stall
The demo works. Then it meets real users, real data, real load and real failures, and nobody planned for any of them. The model is rarely the reason, and the missing pieces are usually the ones on this page.
Six phases
Phase 1: build and test locally
- A clean environment with pinned requirements.
- Configuration and secrets in environment variables, never in code. See secrets management.
- Unit tests for the code, and a first eval set for the outputs.
- An API around your logic (FastAPI or Flask), so the rest of the system can call it.
Phase 2: containerise
Docker solves the "it works on my machine" problem: the container holds everything the app needs, so it runs the same on a laptop, a server or a cloud service. Use Docker Compose for the dependencies (vector database, cache, tracing backend) in development. See Docker for AI apps.
Phase 3: CI/CD
On every change, run unit tests and the regression eval suite, build the container, and deploy if everything passes. GitHub Actions is the common starting point. See CI/CD for AI.
Phase 4: artifact orchestration
An LLM app has more artifacts than code: prompts, model versions or fine-tuned adapters, embedding models, vector indexes, evaluation sets. Version each of them, and record which versions a release uses, so you can reproduce any answer and roll back any change.
Phase 5: monitoring
Track metrics (requests, errors, latency, tokens, cost) and traces of every request. Prometheus and Grafana are common for metrics, and an LLM tracing tool like Langfuse for traces. See observability and tracing.
Phase 6: infrastructure as code
Define servers, networks, databases and permissions in code (Terraform or similar), so environments are reproducible. For a first project, a platform like Render or a managed container service can be enough, and infrastructure as code becomes important as the system grows.
LLM-specific checklist
- Eval suite with regression and capability sets, run in CI
- Input guardrails (PII masking, prompt injection checks) and output validation
- Every request traced, with the prompt version and the model version
- Timeouts, retries and a fallback path for model and tool calls
- Rate limits and spend caps per user and per day
- Semantic or prompt caching where it fits
- Secrets in a secret manager, with separate keys per environment
- Feedback capture (thumbs, edits, escalations) linked to traces
- Rollback for code, prompts and model versions
- A named owner, and an incident runbook with kill switches
Count the unchecked boxes before launch. Ship with zero, or with a written reason for each one.
Before the first real user
- Run the full eval suite on the exact build you're deploying.
- Load test at your expected peak, and check latency and error rates.
- Turn off the primary model in staging and check the fallback.
- Check the cost estimate against real token counts from your load test.
- Make sure someone is watching the dashboards for the first days.
From the floor
For the financial report RAG in AI Engineer HQ, we followed these phases in order: build locally first with a Flask API, Chroma, and Llama 3.1 through Ollama, with a Prometheus exporter on the Flask app so every API call could be monitored, then containerisation, CI/CD with GitHub Actions, orchestration for artifacts and model weights, monitoring with Prometheus and Grafana, and the infrastructure for deployment.
In the customer support agent deployment session, I explained containerisation with the problem it solves: I share code with you, you run it on your machine, and it doesn't work, while a Docker container carries everything the project needs and runs on any machine, laptop or server. For a first deployment, I suggested Render, which has built-in monitoring, before moving to infrastructure as code for bigger applications.