TL;DR
Coding agents (Claude Code, Copilot, Cursor) with command-line access are barely fazed by the size of a task. “Find the right function among a hundred irrelevant ones” or “compute the value after 200 steps” — the agent handles these by writing a small script (grep, code parsing, running computations) instead of holding everything in its head. The same model without tools falls apart quickly on these tasks. It follows that a weak model with tools can beat a strong one without them.
Where agents break is code understanding: when they have to match up two similar pieces of code with different names and find where the logic actually diverges. Then most of the effort goes into reading and analysis, with almost no edits. The agent gets confused by renamed variables, relies on names instead of meaning and misses a one-line difference. The second weak spot: a long list of rules in a single request. That's the only type of load where agents degraded even on simple tasks. What makes a task hard for an agent isn't how many lines it has to edit, but how much it has to understand before editing.
The authors built a synthetic task generator, CABRA, with a “size” knob and turned it along four axes in turn: how many functions to edit, how many irrelevant ones, how many computation steps, how many rules. Then they added a pure-understanding task: “extract the shared logic of two classes into a parent, preserve the differences.” The paper has no ready-made prompts. What it has is a map of the agent's weak spots, from which you can derive rules for how to frame tasks.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Blueprint: what the paper showed and how to use it
The paper has no step-by-step method. Below is a map of “where the agent is reliable and where it isn't.” The right-hand column is my practical takeaway; the authors didn't test it.
THE AGENT IS RELIABLE (tools take the load off):
• finding what's needed among lots of noise → trust it, let it search on its own
• long computations / tracing → let it run code instead of “doing it in its head”
• editing many functions along a call chain
THE AGENT BREAKS:
• matching two similar pieces of code with different names
• finding a one-line divergence amid identical code
• extracting shared logic when the code is “packaged” differently
• dozens or hundreds of rules in one request
WHAT TO DO (a takeaway from the paper, not the authors' method):
understanding → a separate step with an explicit output → edit → verify with tests
Example in Practice
Task: An online store has two nearly identical shipping-cost classes: FedexDelivery and UpsDelivery. Different people wrote them, the variables are named differently, and the formulas are duplicated. You need to move the shared parts into a parent class and leave only what actually differs in the child classes, such as UPS's surcharge for packages over 20 lb. This is exactly the type of task the authors designed to be the hardest for agents.
Prompt (for Claude Code or Cursor):
Task: extract the shared logic of the FedexDelivery and UpsDelivery classes
(files delivery/fedex.py and delivery/ups.py) into a parent class BaseDelivery.
The goal is the most efficient edit: as much code as possible should end up
in the parent, and the child classes should keep only lines that change the result.
IMPORTANT: the variables in the classes are named differently. Matching names
prove nothing, and different names don't mean different logic.
Compare by what the computations mean, not by names.
Work in two stages.
STAGE 1 — UNDERSTANDING (don't touch the code yet).
Output a table: line/variable in Fedex ↔ the corresponding one in Ups,
status: “identical in meaning” / “differs.” For each “differs” pair,
write down exactly what the difference is (sign, coefficient, different variable, extra term).
Run both methods on 10 random inputs and show where the results diverge.
STAGE 2 — EDIT.
Only after the table, move the “identical” lines into BaseDelivery.
Leave only the “differs” lines in the child classes.
Then compare against the old version on 50 random inputs:
the results must match exactly.
Result: The agent will first output a correspondence table marked “identical” and “differs,” plus the results of the run on random inputs. Then it shows the edit: shared code in the parent, differences in the child classes. At the end comes the comparison with the old version. If there was a one-line difference, it's visible in the table instead of getting lost in the merge.
Why It Works
The weakness. The agent is good at reading and searching, but understanding happens inside the model, in the text of its reasoning. The more lines it has to match up, the more candidate pairs there are. If code looks different but does the same thing, the model latches onto variable names and misses one-line differences. A script can't solve this: grep won't tell you that a + a + a and 3*a are the same thing.
The strength. If a task can be reduced to mechanics, the agent writes a meta-script on its own: it searches, computes, runs things. That's why a “needle in a haystack” and long computations are nearly free for it.
How to use this. Move understanding out of the hidden process and into an explicit output (a correspondence table), and hand verification off to code execution (a run on random inputs). That shifts the task from “figure it out in your head” to “do it mechanically.” This is my adaptation of the paper's logic; the authors didn't test these techniques themselves.
The levers: - Number of rules per request: split a long list of requirements across several runs. - An explicit “correspondence table first” request turns hidden understanding into a checkable artifact. - A run on random inputs: more inputs means a more reliable check. - The phrase “compare by meaning, not by names” targets the weakness the paper found: relying on names.
Prompt Template
The template is built from the task wording in the paper (extract the shared parts into a parent, preserve the differences, most efficient edit) with an added understanding stage.
Task: extract the shared logic of {class_A} and {class_B}
(files {path_A} and {path_B}) into a parent class {parent}.
The goal is the most efficient edit: as much code as possible in the parent,
and in the child classes only what actually changes the result.
IMPORTANT: variable names in the classes may differ.
Matching names prove nothing, and different names don't mean
different logic. Compare by what the computations mean.
STAGE 1 — UNDERSTANDING (don't change the code).
Build a table: element in {class_A} ↔ element in {class_B},
status “identical in meaning” / “differ” + what the difference is.
Run both versions on {N_inputs} random inputs and show the divergences.
STAGE 2 — EDIT.
Move what's “identical” into {parent}. Leave what “differs” in the child classes.
Compare the result with the old version on {N_checks} random inputs:
the outputs must match.
What to fill in: {class_A}, {class_B} — the duplicated classes or modules. {path_A}, {path_B} — their files. {parent} — the name of the new shared class. {N_inputs} — 10 or more inputs for the initial comparison. {N_checks} — 50 or more for the final check.
🚀 Quick Start — paste into a chat or into your agent:
Here's a template for a duplicate-code merge task with a separate understanding stage.
Adapt it to my task: {describe which code is duplicated and what must stay different}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which files and classes are being compared, how they should differ and how to run the code for verification. The agent needs this to build the correspondence table and the check on random inputs.
Limitations
⚠️ The safeguards haven't been tested: The paper shows where agents break, but doesn't test techniques like “correspondence table first” or “split the rules into batches.” The template above is a conclusion drawn from the findings, not a measured method.
⚠️ Synthetic code: The tasks were generated from scratch: random math functions, not real projects. High control, low realism. On real data (SWE-bench) the relationship is weaker: correlations are low and failures are hard to predict.
⚠️ One agent harness: All agents ran in the same harness (Copilot). Claude Code or Cursor may behave differently.
⚠️ Many rules are a weak spot: Almost all agents degraded only on long rule lists. The threshold for real projects is unknown; measurements were done with dozens and hundreds of rules.
⚠️ The paper is truncated: My copy of the text lacks the final sections analyzing the causes. The conclusions about relying on names and one-line differences come from the introduction.
How They Studied It
The researchers from Microsoft Research and the University of Maryland chose not to use real repositories: there, edit size, task type and code complexity are tangled together, and you can't tell what exactly trips up the agent. Instead they generated 6,840 tasks from scratch as transformations of function call graphs. Five types of refactoring (remove dead code, add a parameter, return a value up the chain and so on) were scaled along four axes from 5 to 200: the number of functions to edit, the number of irrelevant functions, the number of computation steps and the number of instructions. The tasks were run through 8 language models without tools and 6 coding agents (GPT-5.5, Sonnet-4.6, Opus-4.7, Gemini and others), at a cost of about $25,000.
The first finding: models without tools degrade as size grows, while agents stay nearly perfect because they write scripts. Then the authors labeled the tool calls into six types (reading, analysis, search, editing, running tests and others). As the task grows, it's specifically reading and analysis that increase, while edits barely do. More than half of all reading and analysis happens before the first edit.
Next they checked against real data. On agent trajectories in SWE-bench Verified, the number of “understanding” calls predicted failure slightly better than the number of changed lines. But the correlations there are weak, around −0.2, so the authors don't claim causation. To get at causation they designed a pure-understanding task: two classes with the same logic but different names and “packaging.” The agent has to move the shared parts into a parent and keep the differences. Agent accuracy dropped, the amount of reading went up, and the number of edits barely changed. This confirmed that the bottleneck is specifically understanding. The breakdown of causes: agents can't handle matching code decompositions, latch onto names and miss one-line differences.
Adaptations and Extrapolations
🔧 Technique: get rid of “understanding in its head” → move it into a file
For your project, set up a rule in CLAUDE.md or AGENTS.md. This is my own extension; the paper has nothing like it:
## Before edits that depend on comparing several places
If the task requires matching up two or more similar code fragments
(merging, deduplication, migration, reconciliation):
1. First write a correspondence table to notes/compare.md:
“element A ↔ element B, identical/differ, what the difference is.”
2. Don't rely on matching names: compare the computations.
3. Treat a one-line difference (sign, coefficient, extra term) as
the most likely place for a mistake: double-check it separately.
4. After the edit, run the old and new versions on random inputs
and make sure the results match.
🔧 Technique: a long list of requirements → batches
In the paper, long rule lists are the only load that breaks agents even without the task itself growing. Instead of one request with 30 requirements, send 3–4 requests with 7–8 requirements each, with a check after every one. The paper has no measured threshold, so tune it yourself.
Extrapolation: combine with a check on random inputs
Perform {refactoring}. Before editing, write a
check.pyscript that runs the old version, will run the new version on 50 random inputs, and compares the results. Only then change the code. When you're done, runcheck.pyand show any divergences.
Here I'm combining the paper's conclusion (“agents are good at anything that can be scripted”) with an automatic check. Whether this works, the paper didn't test.
Resources
- Code Understanding is a Bottleneck for Coding Agents — Nishant Balepur (Microsoft Research / University of Maryland), Kiran Tomlinson, Tobias Schnabel (Microsoft Research). Preprint.
- Code: https://github.com/microsoft/CABRA
- Related work: SWE-bench (Jimenez et al., 2024), Needle in a Haystack (Kamradt, 2023), RefactorBench (Gautam et al., 2025), SWE-Agent (Yang et al., 2024).