TL;DR
Exploration-nudging prompt (a prompt that pushes the agent to explore) is an instruction telling the reviewer not to stop at reading the diff. The agent has to walk the repository, find where the changed code is used, and test its hypotheses by running things. In the paper this delivered up to +5.8% more bugs found for only a 10% increase in cost. A multi-agent setup (several reviewers in parallel) gives +4.7%, but costs 5.5 times more.
The headline finding: review agents have gotten good, but they don't replace a human. The best one catches about 70% of known bugs in its top-5 comments and misses the other 20–30%. It struggles most with large, sprawling PRs. Even worse are bugs you can't see from the diff, the ones that require knowing the project. On closed industrial repositories quality drops to as little as a quarter. On top of that, agents almost never run code (under 10% of steps) and trust the PR description and code comments, even when those are misleading. So a confident "no bugs found" guarantees nothing.
The practical takeaway is five rules. Match review depth to PR complexity. Give the agent your project's priorities. Put repository knowledge into an instructions file. Route different PRs to different models. Periodically re-check the reviewer's quality against your own past bugs.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Give the agent context → PR (title, description, diff) + repository + project rules file
STEP 2: Forbid taking things on faith → treat the PR description and comments as hypotheses, not facts
STEP 3: Nudge it to explore → look for call sites, adjacent modules, run code/tests
STEP 4: Demand structured output → a list of findings ranked by severity (review.md)
STEP 5: Read the top 5 → a human checks the top of the list
(all in one system prompt / agent instructions file)
ESCALATION: large PR / critical module / non-local logic → more effort + a human is mandatory
Example
Task: You're a tech lead at a food-delivery service. Claude Code writes code faster than the team can read it. Today's PR headed for production is "Partial refund on an order": money, rounding cents, orders paid with promo codes and points. A bug in this area costs real dollars, and nobody has time to read a 600-line diff.
Prompt:
You are a code reviewer at a food-delivery service. Your job is to find bugs in the PR,
not to grade style.
<Project_context>
- Money is stored in cents (int), no floats.
- An order can be paid partly with "Loyalty" points and partly by card.
- Refunds always go in reverse order: card first, then points.
- Top priority: correctness of money operations and idempotency
(a repeated webhook call from the payment provider must not trigger
a second refund).
</Project_context>
<PR>
Title: Partial refund on an order
Description: Added method refund_partial(order_id, amount). Handles promo codes.
</PR>
<Review_rules>
1. The PR description and code comments are the author's hypotheses, not facts.
Verify against the code itself.
2. Don't stop at the diff. Find every place the changed functions are called,
and the adjacent modules (payment webhook, discount calculation, points).
3. If you're unsure how something behaves, run the tests or write a short
check and run it.
4. I'm not told whether the PR contains a bug. Don't play along with the expectation
that "something will turn up": if you have no confident findings, say so.
5. Write the result to review.md: a list of findings sorted
by severity (critical / high / medium / low). For each one:
file and line, what the bug is, what inputs trigger it,
how to verify it.
</Review_rules>
Result: The agent reads not just the diff but the adjacent files too: the webhook, promo-code calculation, points accrual. It checks some hypotheses by running tests. The output is a review.md with findings, the most dangerous at the top: say, a double refund or cents lost when splitting a discount. Each finding says how to reproduce it. You read the top 5 yourself. A human decides whether to merge.
Why It Works
The weakness. A naive reviewer reads the diff and believes whatever the author wrote. If the description says "handles promo codes," it takes that as fact. Many bugs don't live in the changed lines but in how the change behaves across the rest of the project. And the agent rarely tests its guesses by running anything.
The strength. The model is good at searching code, reading other people's files, and forming hypotheses. If you explicitly allow and require it to do that, it spends its turns searching instead of jumping to a conclusion. That's why cost goes up only 10% while recall goes up several points.
Control levers:
- Project priorities (the <Project_context> block) → point attention at what's critical for you: money, security, API compatibility. Change it per module.
- Repository knowledge → the more rules and architectural caveats, the better on large closed codebases where "just the diff" isn't enough.
- Requirement to run things → drop it if your environment can't execute code. Then don't expect the gain.
- Top 5 by severity → tune the number to your team's attention: the less time people have, the more the ordering matters.
- Model choice → a cheap small model for simple PRs, an expensive large one for complex ones.
Prompt Template
The exact prompt text isn't given in the accessible part of the paper. The template below is assembled from the elements it describes: PR context, writing to review.md, ranking by severity, not telling the agent a bug exists, a no-cheating rule, and the nudge to explore.
You are a code reviewer. Task: find bugs in PR {PR_name}. Don't grade style,
naming, or nitpicks.
<Project_context>
{project_priorities: what's critical — money, security, compatibility, performance}
{architectural_rules_and_pitfalls}
</Project_context>
<PR>
Title: {title}
Description: {description}
</PR>
<Review_rules>
1. The PR description and code comments are the author's hypotheses, not facts. Verify against the code.
2. Don't stop at the diff. Find call sites of the changed functions, adjacent modules,
and the behavior of external libraries.
3. If in doubt, run {tests_or_command} or write a short check.
4. You're not told whether there's a bug. If you have no confident findings, say so.
5. Don't use the internet and don't look for ready-made solutions outside the repository.
6. Write the result to review.md: a list of findings, from most severe
to least severe (critical / high / medium / low).
For each: file and line, what the bug is, what data triggers it,
how to verify it.
</Review_rules>
What to fill in: {project_priorities} — what would be most expensive for you to break. {tests_or_command} — the real command that runs your tests. {architectural_rules_and_pitfalls} — the stuff a newcomer to the project doesn't know (better to move it into AGENTS.md or CLAUDE.md and reference the file).
🚀 Quick start — paste into a chat:
Here's a prompt template for a code-review agent. Adapt it to my project: [describe the project and stack].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about your project priorities, the non-obvious rules of the codebase, and how to run the tests. Those fields deliver the main effect: they steer the agent's attention and let it verify its guesses. It will keep the template's structure and build a prompt for your project.
Limitations
⚠️ The agent doesn't replace a human: even the best reviewer misses more than a fifth of known bugs. Fully "turning off the lights" (removing the human) isn't possible yet.
⚠️ Large, sprawling PRs: the bigger and more heterogeneous the change, the worse the findings. Split the PR or add a human.
⚠️ Bugs invisible from the diff: where you need to understand the whole project, review is noticeably weaker. On large closed repositories quality sinks to as little as a quarter of the result on open ones.
⚠️ Trusting the PR description: agents fall for misleading descriptions and comments. An automated reviewer can be lulled to sleep by the author's wording without anyone meaning to.
⚠️ The prompt gain is modest: +5.8% is not a leap. Bigger gains from more reasoning effort or a multi-agent setup are expensive: many times the cost.
⚠️ Only bugs were measured: the benchmark checks whether a known bug was found, not style, architecture, or readability. False positives (review noise) aren't evaluated in the accessible part of the paper. The accessible version cuts off in the middle of the experiments, so RQ4 details are known only from the summary in the introduction.
How They Studied It
The idea was simple: a good reviewer should at least catch bugs that humans have already confirmed and fixed. A team from Microsoft and CMU built the CRBugBench benchmark: 1000 tasks from 98 open GitHub repositories (5 languages) plus 800 tasks from closed repositories at one large company. Bugs came from two sources. The first was reviewer comments that were later fixed before merge. The second was bug-fix PRs, traced back through code history (the SZZ method) to the PR that introduced the bug. A spot check of 40 tasks showed 92.5% of them were correct.
The agent got a sandbox with the repository, the PR description, and the diff, with no hint that a bug existed. It wrote a review.md with findings by severity. Matches against the real bug were judged by three LLM judges from different model families (majority vote). Agreement with humans came out almost perfect (κ = 0.89). The metric was top-5 recall: did the agent find the bug among its first five findings, since a developer's attention is limited.
They compared large and small models from Anthropic and OpenAI. The best combination grew from 47.7% to 71.9% over five months: models are improving fast, so you need to revisit your settings. Surprisingly, agents barely run code (the best one in 9.7% of steps) and differ a lot in persistence: half of Claude's runs end by turn 35, GPT's by turn 17. Then they changed the agent's design: the exploration-nudging prompt (+5.8% for +10% cost), multi-agent (+4.7% for 5.5× cost), boosted reasoning (from 36% to 57% for 23× cost). The practical conclusion: the most cost-effective lever is the prompt text.
Adaptations and Extrapolations
🔧 Technique: add execution checks → less false confidence
Agents rarely run code, so demand it outright: "For every critical/high finding, attach a minimal reproducing test and the result of running it. If you couldn't reproduce it, mark the finding as 'unconfirmed'". Findings split into confirmed ones and hypotheses, and you don't waste time on duds.
🔧 Technique: split the review by priority → faster to read the top
If you have several priorities, don't overload one prompt. For PRs that touch money, run a separate pass with a
<Project_context>block about money only. For security, another one. This isn't the paper's multi-agent setup, it's a manual analog: two cheap passes instead of one expensive one.
Extrapolation: your own mini-benchmark. The authors recommend building your own set from your repository's bugs. For a small team that's an evening's work:
Here are 10 bugs from our project's history: {links_to_fixing_PRs}.
For each one, find the PR that introduced the bug and frame a review task:
"Here's a PR, find the bug in it." Don't reveal the bug to the agent.
Run the reviewer on each PR and show a table: bug found in top 5 — yes/no.
Run this set whenever you change the model or the prompt: that's how you'll notice the reviewer getting worse or better. This is my extrapolation of the authors' idea; the paper has no such prompt.
Resources
- Paper: "(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection" (CRBugBench)
- Authors: Chenyang Yang (Carnegie Mellon University), Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill (Microsoft), Carmen Badea, Christian Bird (Microsoft Research)
- Works mentioned in the paper: CodeReviewer, SWR-Bench, c-CRAB, CR-Bench, AACR-Bench, the SZZ method for finding the bug-introducing PR