TL;DR
Tell an agent "find and fix problems in section X" and it pulls it off about half the time. Hand it a bug description and it fixes the bug almost every time. The bottleneck isn't the fix, it's finding and recognizing the bug: you have to come up with a scenario, run it through the interface, and realize that what you're looking at is an error.
The typical picture: the agent reaches the right feature, sees a blank screen, and moves on. Or it decides "working as intended" because the existing unit tests expect exactly that. Another finding: unless explicitly asked, the agent barely opens the browser. It reads code, runs terminal commands, and fires off HTTP requests, even though a browser is installed and available.
What to do: say outright in the instructions "check the app in the browser like a real user," define the scope, and give a criterion for "how it should work." The paper's experiment confirms only the first point. The rest are observations from reviewing trajectories. They point to where to look, but weren't tested as standalone techniques.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
The funnel where agents "drop off" (in parentheses: average share of trajectories reaching the stage):
STAGE 1: reached the right feature → ~3/4
STAGE 2: ran the scenario and RECOGNIZED the bug → ~1/2 ← the main failure
STAGE 3: found the root cause → ~44%
STAGE 4: fixed it without breaking anything → ~41%
The agent instruction that follows from the findings:
1. Scope: name a product area (not "bugs in general")
2. Method: "test in the browser like a real user" ← tested
3. Criterion: what "works correctly" means (by intent, not by tests) ← observation
4. Finish: check that neighboring features didn't break ← this is how the verifier judged
All of it in one system prompt or instructions file.
Example in Practice
Task: You run an online store on an open-source engine, say Bagisto. Right before Black Friday you want an agent to walk through the cart and checkout. A classic bug of this kind: the promo code is accepted in the cart but rejected at checkout. You can't see it in the code; it only shows up in the scenario "add an item, enter a promo code, get to payment."
Prompt (an agent with browser access, e.g. Claude Code with Playwright):
Find and fix problems in this store's cart and checkout.
Test the app in the browser like a real shopper: open the store,
add items, apply a promo code, get to the last step of checkout.
Don't limit yourself to reading code and sending requests from the terminal.
If at any step what you see differs from how it should work
for the shopper (the discount disappeared, the total changed, a button doesn't respond),
treat it as a bug. Even if the existing tests expect that behavior.
For each problem: describe the reproduction steps, fix it, then rerun
the scenario. After the fixes, check that neighboring features (catalog, login,
order history) still work.
Result: The agent will launch the browser and walk the shopper's path step by step instead of just reading the cart code. The response will contain descriptions of the problems found with reproduction steps, code fixes, and a rerun of the scenario. At the end, a check of the neighboring sections. Whether it finds your specific bug isn't guaranteed: even the best agents in the study found a little over half.
Why It Works
The weakness. By default the agent takes the familiar path: read code, run commands, send requests. A browser scenario is long (in the paper's example the bug only shows up after nine steps across two accounts), and the agent won't take it on by itself. Even when it sees an anomaly, it doesn't always consider it a bug. It checks against what "the tests expect" and takes wrong behavior for normal.
The strength. When the task is clear, the agent fixes things beautifully. With a bug description it solves nearly everything, and the gap between models almost vanishes. It can drive a browser scenario when told to.
How the instruction exploits this. An explicit "test like a user" switches the agent from reading code to running scenarios, which is where bugs show up. The criterion "how it should be by intent" gives it something to recognize the error against that isn't tied to the tests.
Levers: - Scope in the request — the narrower and more specific (the cart, not "the whole site"), the better the odds of reaching the right feature. - The browser line — remove it and the agent goes back to reading code. A model at 43.5% dropped to 31.5%. - The expected-behavior criterion — add an example or a description of the desired behavior if you have one. That's the road toward the "detailed prompt," where results jumped from 35–51% to 96%. - Time limit — after roughly 40 minutes there's almost no gain, and long runs sometimes never converge at all.
Prompt Template
The authors' verbatim prompt (Appendix A) isn't included in the available text. Below is an assembly based on what the paper describes: an open-ended request with a scope plus a requirement to test in the browser.
Find and fix problems in {product_area}.
Test the app in the browser like a real {user_role}.
Don't limit yourself to reading code and sending requests from the terminal.
Expected behavior: {how_it_should_work}.
If what you see differs from this, treat it as a bug, even if
the existing tests expect something else.
For each problem: reproduction steps → fix → rerun
the scenario. At the end, check that neighboring features didn't break:
{neighboring_features_list}.
What to fill in:
- {product_area} — a section, e.g. "checkout" or "workspace invitations."
- {user_role} — shopper, admin, invited teammate.
- {how_it_should_work} — two or three sentences about the desired behavior. The authors didn't test this line separately; it's an inference from observations. If you don't know, leave "as a regular user would expect."
- {neighboring_features_list} — whatever sits nearby and shouldn't get hurt.
Limitations
⚠️ Only one line was tested: the experiment confirms only the effect of the "test in the browser" instruction. In one case the effect is small (about 2 pp), in the other noticeable (about 12 pp). The other parts of the template are inferences from trajectory review, not separately tested techniques.
⚠️ You need a browser: without the agent having access to a running app with a browser (Playwright and the like), the instruction is pointless. This is a tooling setup; a prompt alone won't cut it.
⚠️ Even with the hint, they don't find everything: the best agent found and fixed a little over half the bugs. It's a helper for pre-release checks, not a replacement for a tester.
⚠️ Narrow scope of evaluation: the tasks were only about web apps, one known bug each. Other bugs found along the way weren't scored, and extra edits weren't penalized. Design, usability, and speed weren't tested.
⚠️ More ≠ better: long runtimes and lots of browser actions aren't linked to success. The gain from extra time tops out around 40 minutes (on a small sample, not statistically significant). Some agents went off for hours with nothing to show.
⚠️ Distrust of tests is an observation: the "agent trusted the unit tests" case was analyzed on one task and four agents. It illustrates a risk, not a measured pattern.
How They Studied It
Researchers from Princeton took 200 real fixed bugs in 200 popular open-source web apps: Ghost, Uptime Kuma, Focalboard, Stirling-PDF, and others. The agent got only the code, a running app with a browser, and a single sentence like "Find and fix problems in Focalboard: board sharing and members." No bug description, no hints.
Results were checked not by a unit test suite but by a verifier agent. It opened the fixed app in a browser without seeing the code and followed step-by-step scenarios: did the target behavior work, and did neighboring features stay intact. That way any correct fix is accepted, not just one matching the reference.
They ran 15 agents. The best solved 59% of tasks; nine solved less than half. Then the researchers broke the trajectories down by stage: reached the feature, recognized the bug, found the cause, fixed it. The biggest drop was at recognizing the bug, from ~75% to 50%.
To separate "finding" from "fixing," five agents were given a detailed bug report. Results rose from 35–51% to 96%, and the gap between models nearly disappeared. Another ablation was a surprise: when the "test in the browser" line was removed for two agents, they almost stopped opening it (2 and 6 tasks out of 200). Results fell from 43.5% to 31.5% for one and from 42.0% to 39.5% for the other. Additional checks showed that working longer and taking more browser actions don't guarantee success, while a suitable harness gives a moderate boost.
Practical takeaway: the agent's instructions shape how it hunts for bugs, not just how it fixes them.
Adaptations and Extrapolations
🔧 Technique: add a regression check → catch collateral breakage
The verifier in the paper required both tests to pass: the target one and the "preserve neighboring behavior" one. In a prompt that's the line "after the fixes, rerun the scenarios from sections {A, B, C} and show that they work." The fixes themselves were mostly correct, but checking that nothing got broken looks like a natural and cheap insurance policy.
Extrapolation (my hypothesis, not tested): split finding and fixing into two requests. First request: only hunt for bugs in the browser and produce bug reports (steps, expected and actual behavior), don't touch the code. Second: fix each report separately. The logic: with a ready report, agents fix ~96% of the time, so the whole problem is getting that report to exist.
STAGE 1 (search only, don't change code):
Walk through the {area} section in the browser as a {role}. For each anomaly
record: steps, expected behavior, actual behavior.
Output a list of reports.
STAGE 2 (separate request, one report at a time):
Here's a bug report: {report}. Find the cause, fix it, repeat the steps,
and check neighboring features.
Resources
- SWEeper-Bench: Can Agents Discover Bugs in Interactive Software? Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R. Narasimhan, Zhuang Liu — Princeton University.
- Website: sweeperbench.com
- Code: github.com/zlab-princeton/sweeper-bench
- Data: huggingface.co/datasets/zlab-princeton/SWEeper-Bench
- Agent trajectories: huggingface.co/datasets/zlab-princeton/SWEeper-Bench-traces
- Related work mentioned in the paper: SWE-bench, SWT-Bench, WebProber, Browser Use, code review tools (Cursor Bugbot, GitHub Copilot code review, CodeRabbit, and others).