TL;DR
Modern coding agents find the buggy line almost every time just by reading the code. The conditions: the program is short, the task is clearly described, and the bug sits on a single line. The researchers compared three modes: read-only, run and debugger tools available “if you want,” and the same tools plus an order to “collect runtime evidence before answering.” The debugger adds a little, and the price for that is a multifold jump in cost.
Here is what the pain looks like. You write “make sure to run the debugger” in your prompt, and the agent burns 15–27x more tokens, takes 5–8x longer, and you pay 3–5x more. The gain? A few percentage points. The strongest of the three models gained nothing at all: it was barely making mistakes to begin with. Models read code worse when it has been obfuscated: double negatives in conditions, a loop rewritten as a manual iterator. Renamed variables and dead code barely get in the way. There is no real interpreter in the model’s “head”; it generates a trace from the text. The more tangled the conditions, the more often that trace drifts off course.
The key practical finding: when tools are available but not mandated, the agent almost never opens the step-through debugger (a few percent of runs). It just runs a small script on sample input, and that is enough to keep nearly all of the gain at a lower cost. You don’t need the debugger per se. You need a way to quickly test a hypothesis by running code.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
MODE A (no tools): task description + code only → buggy line number
MODE B (tools if you want): + script execution and debugger, no instructions → line number
MODE C (tools on order): + 2 sentences in the system prompt
“use the tools, collect runtime evidence” → line number
All modes: one prompt, one agent run.
The only difference is tool access and the add-on sentence.
Original from the Paper
System prompt (shared by all modes):
You are a Python debugging agent. Find the injected defect in the supplied
implementation. You do not have access to hidden evaluation tests.
Add-on used only in mode C:
Actively use the probe and debugger tools to gather runtime evidence before
deciding. Inspect relevant state and execution flow.
End of the task message (always):
The implementation contains one injected bug. Locate it and call `report_bug`
exactly once. Provide the single most likely affected line's 1-based line number.
The authors did not prescribe a debugging strategy. They defined the task and the answer format and left the choice of tool to the model.
Example in Practice
Task: You run a small freelance marketplace with a service-fee calculator (4% from individual clients, 6% from business clients). After an edit, one test fails: the fee on an invoice right at the cap comes out wrong. The function is short, about 15 lines, with a test and a clear description. This is the method’s sweet spot: isolated code, one bug, a reproducible failure.
Prompt (system prompt or the start of a task for an agent in Cursor / Claude Code):
You are a Python debugging agent. The function calculate_service_fee contains one injected defect.
You have no hidden tests, only the description and the code.
Description: the function takes an invoice amount in dollars and a payer type
("individual" — 4%, "business" — 6%) and returns the fee in dollars,
rounded to the cent. The annual $100,000 payout cap is not handled by this function.
Code (lines are numbered):
1 def calculate_service_fee(amount, payer_type):
2 rate = 0.04 if payer_type == "individual" else 0.06
3 if amount <= 0:
4 return 0
5 fee = amount * rate
6 return round(fee / 100, 2)
The function has exactly one defect. Find it and give the number of the SINGLE most likely
line. If you are not sure, you may run a short script on your own inputs
to test your hypothesis. Then answer: the line number and a one-sentence
explanation.
Result: The agent names one line (in our example, line 6, an extra division by 100) and briefly explains why. With a strong model, it will most likely answer straight from reading. If it does run a script, it will be a tiny check on a couple of numbers (say, a $500 invoice), not a step-through debugger session. The answer comes back fast and cheap.
Why It Works
The weakness. The model doesn’t execute code. It generates text that looks like an execution trace. On short, clean code that is enough. On tangled code (conditions with double negatives, rewritten loops) the trace “drifts” and the model blames the wrong line. Ordering it to “use the debugger” fixes this, but you pay in tokens and time.
The strength. Modern models read small chunks of code well and can whip up little scripts quickly. Running code gives you a fact instead of a guess: one line of output settles the doubt. A step-through debugger is overkill for this: lots of calls, lots of extra text in the context.
How the method sidesteps the weakness. The agent gets tools but isn’t forced to use them. The model decides for itself how confident it is. Confident? It answers right away and saves money. Unsure? It runs a check. The extra order added only a fraction of a percentage point on top, while costs went up several times over.
Control levers: - The runtime add-on (“collect runtime evidence before answering”). Turn it on for tangled code and weaker models where errors actually remain. Drop it where the model already handles the job. - Script execution access. This is the main source of value, so always keep it. You can skip the step-through debugger. - Answer format (“one line number + an explanation”). A strict format keeps the agent from spreading its answer across three hypotheses. - Round limit. The study capped tool use at 50 rounds. For simple tasks set it lower so the agent doesn’t wander off on a long hunt.
Prompt Template
You are a {language} debugging agent. The supplied code contains {defect_count}
defect(s). You have no hidden tests.
Task description: {what_the_code_should_do}
Code (lines numbered from 1):
{code_with_line_numbers}
What fails: {symptom_or_failing_test}
Find the defect and give the number of the SINGLE most likely line.
Answer with: line number + a one-sentence explanation.
Two versions to choose from:
The “if you want” version (default). Add nothing. Just make sure the agent can run code (in Claude Code / Cursor it usually can).
The “on order” version. Add this to the system prompt:
Before answering, actively use code execution and the debugger to collect runtime
evidence. Inspect variable state and execution flow at the suspicious spots.
What to fill in: {language} — the programming language; {what_the_code_should_do} — a short but precise description of the expected behavior (the more precise, the better the localization); {code_with_line_numbers} — the function with line numbers; {symptom_or_failing_test} — what exactly is wrong (a wrong value, a failing test).
Limitations
⚠️ Short code. The programs were small: 8–19 lines on average, a single file. The conclusion “you barely need a debugger” has not been tested on multi-file repositories, large functions, or complex dependencies.
⚠️ One kind of bug. Every program had exactly one single-line defect: an off-by-one, the wrong operator, the wrong variable, inverted logic, or a swapped return value. Race conditions, flaky bugs, environment issues, and data errors were not studied.
⚠️ Locating only, not fixing. They measured whether the agent pointed to the right line. Whether it fixed the bug correctly was not checked.
⚠️ Algorithmic tasks. The data comes from LeetCode. These are clearly specified problems with complete tests. Real product code with fuzzy requirements behaves differently.
⚠️ Expensive when ordered. If you do force tool use, costs go up several times over. On a strong model it barely pays off.
⚠️ Tangled code still trips models up. Double negatives in conditions and restructured loops lowered accuracy for the weaker models. So “reading the code is enough” holds mostly for clean code and strong models.
How They Studied It
The researchers took 150 LeetCode problems (50 each of easy, medium, and hard) along with hand-written reference solutions. Into each they injected one bug of five types (off-by-one, wrong arithmetic operator, condition error, wrong variable, swapped return). That produced 610 bugs. Then each program was rewritten in five behavior-preserving ways: renaming variables, splitting expressions, restructuring loops and conditions (double negatives), inserting dead code, and all of the above combined. The checks were strict: the rewritten version had to produce the same test failure, and restoring the original expression on the new line had to fix the program. The result was 3,022 rewritten programs.
Three agents (two OpenAI models and the open Qwen 27B) hunted for the bug in three modes: no tools, tools if they wanted, and tools plus an order. 65,376 runs in total.
Without tools, the agents found 98–99.9% of bugs in the original programs and 96.7–99.9% in the rewritten ones. Renaming and dead code barely mattered. Loop restructuring and double negatives hurt the most, with drops of up to 3.8 pp. Ordering debugger use eliminated 61–63% of the remaining errors for the two weaker models (+1.3 and +2.1 pp), but it cost 15–27x more tokens. The strong model stayed at 99.9% in every mode.
The surprise: when tools weren’t mandated, the step-through debugger was used in only 0.2–6.2% of runs, while scripts were run noticeably more often. The gain was +1.3–1.5 pp. The order added just 0.0 to 0.6 pp on top. The practical takeaway: the value comes from being able to run code, not from the debugger and not from the order.
Adaptations and Extrapolations
🔧 Technique: escalate to the order only when needed → savings. (My adaptation, not from the paper, but it follows from the numbers.)
First give the agent the task without the runtime add-on. If the answer comes back unsure, or it names the wrong line twice in a row, repeat the request with the add-on:
You got it wrong (or you are not sure). This time you must run a minimal script on
sample input and check which line produces the wrong value.
Only then give the line number.
That way you pay for the extended check only on the hard cases, not on every single one.
Resources
- Do Coding Agents Still Need a Debugger? Frontier-Model-Based Fault Localization at the Accuracy Ceiling. Authors: Christian Macho, Martin Pinzger (University of Klagenfurt), Christoph Daxerer, Johann Glock, Rudolf Ramler (Software Competence Center Hagenberg), Josef Pichler (University of Applied Sciences Upper Austria).
- Data: LeetCodeDataset. A replication package with the prompts, programs, and 65,376 agent traces is referenced in the paper.
- Related work: debug-gym (pdb helps reasoning models and hurts weaker ones), InspectCoder (raw pdb without an intermediate layer sharply reduced success).