TL;DR
The study tests whether a coding agent benefits from a list of “suspicious spots” in the code produced by a separate tool (fault localization, FL, “finding where the bug lives”). The list was simply appended to the task description. The agent was still free to search the whole project. The result: the hint shifts where the search starts, but does not raise the share of fixed bugs. More often it even nudges that share down a little.
The headline finding: a smart bug-finding tool does not make the agent any smarter. The agent is already good at finding code on its own by searching and reading files. The hint mostly changes the first few steps: the agent reaches the right file faster. If the list is wrong, the agent checks it, realizes it was sent on a wild goose chase, and keeps searching. That burns steps and tokens. Precise locations really do save tokens (up to 39% on the repair), but the localization tool itself spends more than it saves. You end up behind on both cost and quality.
What works better: a hint that explains the cause of the bug instead of just naming a location. It delivered a gain on top of the coordinates alone. Hints matter most when the bug report has few clues about the location: no stack trace, no function or file names. If the report already contains such clues, the hint adds almost nothing.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
This is a findings paper, not a technique. Below is a blueprint of the practical takeaway: what to put in the agent's task.
STEP 1: Don't bolt a separate “bug finder” in front of the agent
→ the agent searches on its own, cheaper and no worse
STEP 2: Describe the symptom + how to reproduce → give location clues if you have them
(error, stack trace, function/file name)
STEP 3: Add the CAUSE of the bug (a hypothesis) along with the location
→ the cause gives more than the location alone
STEP 4: Label the hint as a “hypothesis to verify”
→ the agent may go beyond it
STEP 5: Definition of done: a test/reproduction
→ the agent checks its own work
All of this goes to the agent in a single message.
Example in Practice
Task: You run an online store on Django. The inventory export to Amazon occasionally sends a negative stock quantity. You fire up Claude Code or Cursor. You've already noticed the bug shows up after an order is returned.
Prompt:
Bug: the inventory export to the marketplace sometimes sends negative values.
How to reproduce: place an order for 3 units of a product, create a return
for 5 units (the return is created from the admin panel), run `python manage.py sync_stock`.
In the log: "stock=-2 sku=AMZ-48213".
My hypothesis about the cause: on a return, stock is incremented twice:
first by the post-save signal on the return, then again inside
the stock recalculation function. That's why the discrepancy accumulates,
and the export doesn't clamp the value at zero.
Where I suspect it is: the orders/signals.py and stock/services.py modules.
This is a starting point, not a boundary. If the cause is somewhere else, go there.
What I need: find the real cause, fix it, add a test that
fails before the fix and passes after. Show which files you changed and why.
Result: The agent will first check the modules you named and confirm or rule out the hypothesis. Then, if needed, it will keep digging through the project. You get a fix, a test, and an explanation of which files were touched. The cause hypothesis cuts down on blind searching, and the “not a boundary” caveat keeps the agent from getting stuck on a wrong guess.
Why It Works
Weakness: A list of locations with no explanation gives the model nothing but coordinates. If the coordinates are wrong, the agent wastes steps checking them. If they're right, it would have found them anyway by searching the project, so the gain is small. That's why the average outcome came out at zero or negative.
Strength: The agent is good at searching and double-checking. It doesn't take the hint on faith: it reads the code, cross-checks, and widens the search when things don't match. Even with completely false hints, it often still reaches the right code.
How to use it: Give the agent what it doesn't have and can't find by searching: knowledge of the cause. Your understanding of “why it breaks” saves more steps than “where it lives.” And labeling it “this is a hypothesis” preserves the agent's right to verify.
Control knobs: - The more location clues already in the bug report, the less you need a separate hint. Then the symptom and a test are enough. - The fewer clues (the user just wrote “the export is broken”), the more valuable your hypothesis about location and cause. - The phrase “a starting point, not a boundary” lowers the risk of the agent fixating on the wrong spot. - A definition of done (a test) forces the agent to verify the fix instead of stopping at a plausible-looking patch.
Prompt Template
The paper has no verbatim prompt: the authors simply appended the list of locations to the task description. The template below is built from their findings (the cause matters more than the location, the hint is a hypothesis).
Bug: {symptom, what the user sees}
How to reproduce: {steps or command}
Actual result: {log, error, stack trace}
Expected result: {what should happen}
My hypothesis about the cause: {why you think it breaks}
Where I suspect it is: {files/functions}. This is a starting point,
not a boundary. If the cause is somewhere else, go there.
What I need: find the real cause, fix it, add a test that
fails before the fix and passes after. Explain which files you changed and why.
Fill in: the symptom and reproduction steps (the more concrete, the less the agent searches blindly), the cause hypothesis (if you don't have one, honestly write “cause unknown” and don't make one up), the suspicious locations (if you don't know them, delete the whole line).
Limitations
⚠️ Not confirmed beyond bug repair: The experiments covered fixing bugs in code (Python and Java). Transfer to other agentic tasks (building features, refactoring, data analysis) was not tested in the paper.
⚠️ The cause hint was tested as “gold”: The cause descriptions were taken from the reference fixes, so they were correct by construction. Your hypothesis may be wrong, and then the gain shrinks. That's why the hint should be labeled as a hypothesis.
⚠️ No guarantees: Location hints don't do catastrophic damage, and in a few cases they even helped slightly. But almost none of the methods shows a consistent gain.
⚠️ The reference is incomplete: The locations from a single reference fix don't cover every place where successful agents actually edited code. The agent may fix the bug somewhere else, and that's not a mistake.
⚠️ Some data comes from a summary: For details of the agent-behavior analysis and the cause experiments, I'm relying on the paper's abstract. The full tables for those sections are missing from the available text.
How They Studied It
The researchers took six state-of-the-art fault localization methods. Three work from the issue description (SWE-bench Verified, 500 Python tasks), three from a failing test (Defects4J, 854 Java bugs). Each method was run with three models and the same simple agent, whose only tool was the command line. The comparison: an agent with no hint versus an agent with the same description plus a list of locations. That added up to 16,248 runs. The main metric: the share of bugs fixed on the first attempt (pass@1).
The result was a surprise: in 17 out of 18 comparisons, the hint made results worse. The spread ran from −4.45 to +0.40 percentage points. In 16 of 18 cases the difference was statistically indistinguishable from noise. Then they looked inside the runs. The hint really does change how the search starts: the agent opens the right files sooner. But after that it double-checks anyway, and with a wrong hint it goes off searching further. So the early head start doesn't turn into more fixed bugs.
Next, the authors gave the agent “gold” hints: exact locations and a description of the cause, taken from the reference fixes. They tested four kinds of tasks, including tasks with images and large multi-location edits. Conclusions: exact locations save tokens (up to 39.2%), hints matter more when the bug description is thin, and adding the cause gives a gain on top of locations alone. A separate finding: the locations from one reference fix don't cover every path to a successful repair. The takeaway for practitioners: in your task, write “why,” not “where.”
Adaptations and Extrapolations
🔧 Technique: a confidence label → controlled agent trust
Add a confidence level to the prompt:
My hypothesis about the cause: {hypothesis}. Confidence: low.
Verify it against the reproduction first, before changing any code.
Effect: the agent tests the hypothesis first instead of building a fix on top of it. This is my adaptation; the paper doesn't test it.
🔧 Technique: add a rule about hints to the agent's instructions file
In CLAUDE.md or AGENTS.md:
If the task lists “suspicious locations,” treat them as a hypothesis.
Verify against the reproduction. If it doesn't hold up, keep searching;
don't bend the fix to fit the hint.
This extrapolates the paper's finding that the agent can move past false hints but spends steps doing so. The rule makes that behavior explicit. The effect of this exact wording has not been measured.
Resources
- Paper: From Finding to Fixing: How Useful is Fault Localization for Agentic Program Repair?
- Authors: Xinyu Shi, Zhou Yang, An Ran Chen (University of Alberta, Canada)
- Benchmarks: SWE-bench Verified, Defects4J v3.0.1, SWE-bench Multimodal, SWE-bench Pro
- Agent scaffold: mini-swe-agent
- Fault localization methods tested: LocAgent, CoSIL, OrcaLoca, AutoFL, SoapFL, FlexFL