How CI gates and production monitoring work together for LLM apps
CI evals and production monitoring both check outputs against rules, and they answer different questions. CI asks whether this change broke something you already know about, using a small curated test set on every pull request, while monitoring asks what is breaking right now that you didn't know about, using judges on a sample of live traffic. Close the loop: when monitoring finds a new failure, add examples of it to the CI set that same week.
Two questions
| CI gate | Production monitoring | |
|---|---|---|
| Question | Did this change break something we know about? | What's breaking that we didn't know about? |
| Data | Curated test set (golden set) | Sampled live traces |
| Reference answers | Often available | Usually not |
| Checks | Mostly code, a few judges | Mostly validated judges |
| When | Every pull request | Continuously |
| Result | Block or allow the merge | Dashboards, alerts, new tests |
CI test set
The evals FAQ describes CI test sets as small, often 100 or more examples, built on purpose: core features, past bugs and known edge cases. Most teams call this a golden set.
Every example in it should have a reason you can name, like "the date math bug from March" or "the EU refund window". An example nobody can explain is hard to maintain and impossible to retire.
CI runs on every pull request, so cost and speed matter. Use code checks wherever you can, and keep LLM judges for the few failures that need them.
Two suites in CI
- Regression suite: should pass at nearly 100%, and blocks the merge if it drops.
- Capability suite: starts low, tracks progress on problems you haven't solved yet, and doesn't block.
Gate on named checks. "Handoff regression suite dropped from 100% to 96%" tells an engineer what to fix. "Quality dropped 2 points" starts a meeting.
name: llm-evals
on: [pull_request]
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: python -m evals.run --suite regression --min-pass 0.98
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- run: python -m evals.run --suite capability --report-only
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
Version the golden set with your code, so every commit knows which tests it was checked against.
Production monitoring
In production, sample live traces and run evaluators on them in the background. Live traffic has no reference answers, so you lean on validated LLM judges and on signals like thumbs down, retries and handoffs.
Track confidence intervals, not single numbers. A pass rate of 91% on 50 samples and 91% on 5,000 samples are very different claims. The FAQ's rule: if the lower bound crosses your threshold, investigate.
import math
def wilson_interval(passed: int, total: int, z: float = 1.96) -> tuple[float, float]:
p = passed / total
denom = 1 + z**2 / total
centre = (p + z**2 / (2 * total)) / denom
margin = z * math.sqrt(p * (1 - p) / total + z**2 / (4 * total**2)) / denom
return centre - margin, centre + margin
threshold = 0.85
for passed, total in [(46, 50), (4550, 5000)]:
low, high = wilson_interval(passed, total)
flag = "investigate" if low < threshold else "ok"
print(f"{passed}/{total}: {passed / total:.0%} 95% CI {low:.1%} to {high:.1%} {flag}")
The sample of 50 has the higher pass rate, 92%, and still gets flagged, because with so few traces its lower bound falls below the 85% threshold.
Close the loop
When monitoring finds a new failure:
- Read the traces and name the failure mode.
- Add examples of it to the CI set that same week.
- Fix it, and watch the new test go from failing to passing.
That loop is how a CI set stays relevant, and how a failure that reached users once can't reach them again unnoticed.
A real example
Rechat's AI assistant, Lucy, hit a wall where fixing one failure caused others, which the team described as a game of whack-a-mole. The way out was layers: hundreds of fast assertions run on every code change, human and model review of logged traces, and A/B tests for mature features.
From the floor
In the RAG project lab, the step after building the pipeline was CI with GitHub Actions: a workflow file that runs on every push and a test file that checks the chunks and the pipeline, so any update, including a model update, is tested automatically before it moves forward.
In Agent Forge's production version, the stack had GitHub Actions for CI/CD and Langfuse for tracing every LLM call, so the same project had both lanes: tests before deploy and traces after.