CI/CD for AI applications
A CI/CD pipeline for an AI app runs the normal tests, then the LLM regression eval suite, then builds the container and deploys it, and it blocks the merge when evals regress. Treat prompts, model versions, tool schemas and retrieval settings as code that triggers the pipeline when they change. Green unit tests prove the code runs, not that the outputs are still right, so the eval step is what makes this CI/CD for AI rather than CI/CD for a web app.
What CI and CD mean
Continuous integration (CI): every change is automatically tested when it's pushed, so problems are caught before they're merged. Continuous deployment (CD): every change that passes is automatically built and released, so shipping is routine instead of an event.
GitHub Actions is the common starting point: a YAML file in your repository describes the jobs, and GitHub runs them on every push or pull request.
What triggers the pipeline in an AI app
More things than in a normal app:
- Code changes.
- Prompt changes. A one-word edit can change every answer.
- Model version changes, in config.
- Tool definitions and schemas.
- Retrieval settings: chunk size, top k, reranker, filters.
- Guardrail rules.
- Changes to the eval set itself.
Keep all of these in the repository, versioned, so a change to any of them goes through the same pipeline. See prompt versioning.
Pipeline
- Lint and unit tests: fast, no model calls.
- Regression eval suite: the golden set, mostly code checks, a few validated judges. Blocks the merge if it drops below the threshold.
- Build the container image, tagged with the commit.
- Deploy to staging, run a smoke test against the real endpoint.
- Deploy to production, gradually. See rollouts.
name: ci
on: [pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: pytest tests/unit
- run: python -m evals.run --suite regression --fail-under 0.98
env:
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
build:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: docker build -t ghcr.io/acme/support-bot:${{ github.sha }} .
Keeping evals fast and affordable in CI
- Keep the CI suite small and focused: core features, past bugs, known edge cases.
- Prefer code checks over LLM judges in CI.
- Run the large capability suite nightly, not on every pull request.
- Cache model responses for unchanged cases where your setup allows it.
- Use a separate API key with its own budget for CI.
Secrets
Never put API keys in the workflow file or the repository. Store them as repository or environment secrets, and give CI its own keys with tight limits. See secrets management.
Measuring it
- Regressions caught in CI vs found in production.
- Time from merge to production.
- CI eval cost per month.
From the floor
In the RAG project lab, CI/CD automation came after the pipeline worked locally: a GitHub Actions workflow file, which is just instructions the system follows, and a test file that checks the chunks and the pipeline, so every change, including a model update, is tested automatically before it moves forward.
In DevFix, before deploying, we stored the Langfuse API key and a GitHub personal access token as repository secrets, so the workflow could use them without putting them in the code.
In Agent Forge, the prompts lived in their own folder, one file per agent (planner, coder, reviewer and tester), with a version string we change every time a prompt changes, so traces show which prompt version produced each output.