AI system design interview: design a code review agent
Trigger on pull request events, put the job on a queue, gather the diff plus the context a reviewer would read, run static analysis and tests in a sandbox, then have the model review file by file against a strict comment schema. Post only a few high-confidence comments, because developers ignore a noisy bot. Evaluate by replaying historical pull requests where humans found real bugs.
Question
Design an agent that reviews pull requests and leaves useful comments.
Clarify
- Languages and repository size.
- What it should catch: bugs, security issues, style, missing tests?
- Whether it may push fixes, or only comment.
- How many PRs a day, and how fast comments must appear.
Architecture
| Stage | Component |
|---|---|
| Trigger | Webhook on PR opened or updated |
| Queue | Job queue, one job per PR revision |
| Context | Diff, touched files, related code via search, the project's rules file, linked issue |
| Checks | Linters, type checks and tests in a sandbox |
| Review | Model reviews each file with check results, outputs comments as JSON: file, line, severity, explanation, suggestion |
| Filter | Deduplicate, drop low-confidence comments, cap the total |
| Post | Code host API with read-only repository access plus permission to comment |
Deep dive: noise
A bot that leaves 30 comments on every PR gets muted within a week. Post only comments above a confidence threshold, prefer fewer high-severity comments, and never comment on what the linter already reports. Track which comments get resolved and which get dismissed, and use that to tune the threshold and the prompt.
Permissions and sandboxing
Running tests means running code from the PR, which may be untrusted. Run it in an isolated container with no secrets and no network, and give the agent comment permission only, so a prompt injection hidden in the code can't merge or push anything.
Evals
Replay historical PRs where human reviewers found real bugs, and measure how many the agent catches (recall) and how many of its comments were useful (precision). Precision matters more here, because false alarms destroy trust.
What interviewers listen for
Precision over volume, sandboxed execution, and evaluation on historical data.
From the floor
In the DevFix auto agent build in AI Engineer HQ, we built a CLI agent with LangGraph that writes code, runs it, checks for errors and fixes them in a loop until the tests pass, and I listed what that needs: stateful loops, sandboxing, passing tests and reflection, which are the same building blocks a review agent needs when it runs checks on someone else's code.