How to Build a Code Review AI Agent

Key takeaways

• 01

  An AI code review agent is only as good as the context pack: diff plus symbols plus the conventions file, not the hunk alone.

• 02

  Severity and a comment cap are product features; unranked nits get the bot banned.

• 03

  Repo RAG exists to fetch the invariant, the auth helper, and the last similar migration — not to dump the monorepo into the prompt.

• 04

  The agent comments; humans or deterministic CI merge.

• 05

  Evals that match senior reviewers on blockers and punish false positives are the shipping gate.

A code review agent is not a chat window on the repo. It is a batch job that fires on each pull request, reads a packed context, and writes structured comments. Authors tolerate it when it catches blockers they would have shipped and stays quiet otherwise. They mute it when it bikesheds naming.

Write the contract first. Which rules are in scope? Who may see the diff (often the same as CI secrets policy)? Does the agent suggest patches or only flag? Is it allowed to request changes, or only to comment? If that document does not exist, you will tune prompts against Slack complaints.

Send the full file for small edits and a window around each hunk for large ones, plus one hop of symbols. A changed helper needs its callers. A changed SQL string needs the wrapper that interpolates it. Language servers or ctags in CI can provide those hops without a 20-minute clone of the world.

Drop generated bundles, lockfiles, vendored third_party, and binary diffs. They waste tokens and produce false secrets findings. Respect .gitattributes and CODEOWNERS: an agent that comments on a generated protobuf file has already lost the room.

• Always: patch, file path, language, PR description, linked issue
• Usually: full file if under a size cap, plus one-hop symbols
• Retrieved: CONTRIBUTING, ADRs, auth helpers, similar past findings
• Never: the entire monorepo, secrets from CI, unrelated apps in the same git repo

Repo RAG is targeted retrieval: conventions, architecture notes, the canonical authz helper, a similar migration. Index those, not every test fixture. Query with the changed paths and a short summary of the PR. Five to ten passages is plenty. If the model still misses a local invariant, your packer missed a neighbor, not a wiki page.

Permission still applies. A public fork PR must not retrieve internal ADRs. Run retrieval as the bot identity that the forge already uses for CI, and keep customer-private repos off any vendor trainer. Traces stay in your cloud.

Each finding needs a rule id and a severity the team already uses. Blockers fail the check. Majors comment. Nits are off by default or behind a label. Cap the review at a small number of comments, highest severity first. Duplicate the same note on three hunks and you have a spam bug.

Post through the forge review API so threads update on push. Do not @-spam authors. Never grant the bot merge or admin. Deterministic tests, SAST, and CODEOWNERS remain the merge gates. The LLM pull request review is an extra reviewer with a measured false-positive rate.

Label a set of historical PRs: blockers the senior reviewer caught, nits they ignored, and comments they made that were later reverted. Score the agent on blocker recall, extra comments per PR, and whether a suggested patch applies. A model that finds 90% of blockers and files 40 nits is worse than one that finds 70% and files two.

Re-run on every prompt, model, and packer change. Sample live PRs weekly: did humans react with thumbs-up, resolve, or 'not useful'? Feed the misses back as eval cases. This is how the agent tracks the team's taste instead of a generic 'be a senior engineer' prompt.

ReinforcedX implements the packer, repo RAG, CI reporter, and the senior-review eval set in your forge and your cloud. Model-agnostic. You own the prompts, traces, labeled PRs, and runbooks. No token markup. The bot does not get merge rights.

Four weeks: contract and labeled set, packer plus comments in a shadow repo, shadow-mode on real PRs with comments hidden from authors, then handover. Thirty days on-call is for the first monorepo path exception. 98.7% QA pass rate is our delivery bar on the agreed eval set, not a claim that the agent replaces your staff engineer.

Step-by-step build

1. 1

   Define the review contract

   List the defect classes you want (authz, injections, data loss, secrets, breaking APIs), the severity labels, and a max comment count per PR.

2. 2

   Pack the diff with neighbors

   From the forge webhook, collect the patch, the full files, and one-hop definitions (callers, callees, types) so the model can see usage, not only the edit.

3. 3

   Add repo RAG for conventions and similar code

   Index CONTRIBUTING, ADRs, auth helpers, and prior review comments; retrieve a handful of passages keyed by the languages and paths in the PR.

4. 4

   Run a structured reviewer

   Force JSON findings with file, line, severity, rule id, rationale, and an optional patch; drop anything that cannot point at a line in the diff.

5. 5

   Post through CI, not as a noisy human

   Use the checks or review API, group by severity, skip generated paths, and update the same review on new pushes instead of stacking duplicates.

6. 6

   Evaluate against senior review

   On a labeled historical set, score recall on blockers, false-positive rate on nits, and time-to-first-comment; gate releases on those numbers.

Common pitfalls

The mistakes that show up in real deployments — each one costs a week if you learn it the hard way.

Frequently asked questions