TL;DR
The study compares two modes of running the same model on the same task. In the first, the model answers once and hands over the files right away. In the second, it works in a sandbox: it reads the input files, writes scripts, runs them, looks at the output, fixes things, and only then submits the result. The toolset is minimal: read a file, write a file, replace a line, run a shell command, submit the answer.
The main finding: models are bad at guessing the result blind. The script won’t run, the columns are named wrong, distances are computed in degrees instead of meters — and all of it only shows up after you run it. In sandbox mode, the share of successful tasks for models above 2B parameters rose by 26.7–52 percentage points. The gain was largest where intermediate files expose the error, so the agent can see it and fix it.
The core of the method: don’t ask for “the perfect answer on the first try” — give the agent a “write → run → inspect → fix” loop and an independent check of the result. The check has to look at the files, not at what the model says about them. The best configuration reached 85.3% successful tasks. A compressed (quantized) 9B-parameter model reached 69.3% and takes up about 6.5 GB of memory.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
MODE A (chat-only): task + description of required files → one answer → check
MODE B (agent-with-tools):
STEP 1: read the input files (read_file)
STEP 2: write a script or file (write_file / edit_file)
STEP 3: run it and look at the output (run_shell)
STEP 4: if there’s an error — go back to step 2
STEP 5: submit the result (submit) → the hidden check runs only after submission
All steps happen within a single agent session. They are not separate requests.
Example
Task: an analyst at a scooter-sharing company in Chicago (say, Lime) has an export of parking spots and a list of subway stations. For each spot, they need the nearest station and the distance in meters. The data has duplicates, empty names and broken coordinates. This is a direct analog of tasks T05 and T09 from the study, where models most often trip over “degrees instead of meters.”
Prompt (for Claude Code, Cursor or another agent with a terminal):
You are working in a project folder. Input files:
- data/parkings.csv (id, name, lat, lon)
- data/metro_stations.csv (station, lat, lon)
Task: for each parking spot, find the nearest subway station and the distance in METERS
(haversine formula, not the difference between coordinates in degrees).
Before computing, clean parkings.csv: remove duplicates by id, handle empty name values,
fix or drop invalid coordinates. Describe what you did in out/cleaning_report.json.
Result: out/parking_to_metro.csv with columns parking_id, station, distance_m.
How to work:
1. First read the input files and look at the actual columns and values.
2. Write a script in scripts/link.py, run it, look at the output.
3. If there’s an error or the result looks odd (distances in the hundreds of thousands of meters,
empty values) — fix it and run again.
4. Submit the result only after the script has run without errors and you have opened
the final CSV and checked 5–10 rows.
Result: the agent will look at the files first, then write and run the script. If it crashes or produces suspicious numbers, the agent will see that in the output and rewrite the code. At the end you’ll have the script, a cleaned table, a cleaning report and the final CSV. The log shows how many iterations it took.
Why it works
The weakness. In a single answer, the model writes code and files blind. It can’t see the actual column names, doesn’t know whether the script will run, and won’t notice that values have gone out of sane bounds. Any small mistake makes it into the final output.
The strength. Models are good at fixing things based on feedback. Show one an error message or a chunk of the real table, and it quickly finds the cause and the fix. Spotting an error in concrete output is easier than predicting every error up front.
How the method exploits this. The sandbox turns a “guess everything at once” task into a series of small steps with checks. The effect is strongest when intermediate results are visible: tables, JSON, logs, numbers. The study says it outright: the gain depends on what the agent can inspect and verify.
Knobs: - Visibility of intermediate files. Ask the agent to save intermediate tables and look at them. The more it can see, the more errors it catches. - Submission condition. “Submit only after a successful run and check” keeps it from stopping at the first draft. - Independent verification. A separate checker script (columns, ranges, row count) is more reliable than “check yourself.” - The model. Size matters: for models under 2B parameters, the sandbox gain wasn’t as consistent. A quantized 9B model turned out to be a reasonable memory trade-off.
Prompt Template
The available part of the paper doesn’t include the prompt texts themselves. The study describes the task structure: instructions, input files, a list of output files, five tools, and a hidden check after submission. The template below follows that structure. The wording is mine.
<Task>
{task description: what needs to be done and why}
</Task>
<Inputs>
{list of input files with a short schema: name, columns, format}
</Inputs>
<RequiredOutputs>
{list of files to create, with exact names and columns/fields}
</RequiredOutputs>
<DomainRules>
{domain rules that are easy to break: units of measurement,
time formats, what counts as junk, which formula to use}
</DomainRules>
<Workflow>
Work in the working folder. Available actions: read a file, write a file,
replace a line in a file, run a command, submit the result.
1. Read the input files and check the actual data structure.
2. Write a script and run it.
3. Read the output and the resulting files. If there’s an error, fix it and run again.
4. Submit the result only when all files from <RequiredOutputs> are created and checked.
</Workflow>
What to fill in: {task description} — one or two sentences. {input files} — real names and columns. {required files} — exact output file names and structure. {rules} — anything where the model might go wrong “by common sense”: meters or degrees, time format, what counts as a duplicate.
🚀 Quick start — paste into a chat:
Here is a task template for an agent with a working folder and a “write → run → check” loop.
Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about your input files, the output format and the domain rules. For this method, three things matter most: what exactly the agent will see as input, what a correct result looks like and where the model usually goes wrong. It will take the structure from the template and assemble a ready-to-run assignment.
Limitations
⚠️ Verifiability required: the gain is largest where the result can be checked automatically (a table, a script, JSON). For texts, the researchers only checked structure and consistency with the data. They didn’t assess prose quality, so the conclusion isn’t proven for creative or subjective tasks.
⚠️ One domain: all tasks come from transportation data and urban analytics. The authors themselves note this isn’t the whole world of data engineering. Transfer to other areas is plausible but untested.
⚠️ The whole package is measured: the “agentic gain” bundles tools, iterations, feedback and accumulated context together. The study doesn’t break down which part contributes how much.
⚠️ Local models only: the study tested open models that fit on a single 24 GB consumer GPU. Frontier cloud models weren’t tested. Model size limits reliability.
⚠️ A sandbox won’t save weak models: the conclusions about the gain apply to models above 2B parameters. The best configuration still fails roughly one task in seven, so human review is needed regardless.
⚠️ Preprint and incomplete text: the paper is still under review. The available part has no breakdown of error types, so details about where agents break aren’t covered here.
How the research was done
Researchers from the University of Alicante built 15 tasks across five stages of working with urban data: source discovery, transit-schedule processing, record enrichment, feature computation, and validation plus reporting. They deliberately planted traps in the tasks: “decoy” catalog entries, times like 25:10:00 (valid in transit schedules), distances that must be computed in meters rather than degrees, and a report that can’t make claims unsupported by the data.
Each task was checked by a hidden automatic checker. It read the output files, ran the scripts and compared the numbers against a reference. The model’s claims about its own success didn’t count. The data was frozen as local snapshots so that outages of external services wouldn’t affect the results.
Then 10 configurations of local models were run in both modes (single answer vs. sandbox with five tools), with 5 repeats per task. That came to 1,500 scored attempts. The idea is simple: if the gap between the modes is large, then “see and fix” matters more than “get it right the first time.”
The direction of the result was no surprise, but the scale is striking. The gain was tens of percentage points. It was largest where the error is visible in intermediate files. The practical takeaway: set up your agent so it can run code and inspect results, and build in an independent check on the output.
Adaptations and extrapolations
🔧 Technique: make errors visible → better odds the agent catches them
Add this to the agent’s instructions:
After each step, save the intermediate result to tmp/ and print
the first 5 rows, the row count and the value range of key columns.
If the row count suddenly changes by 2x or more, or values go beyond
sane bounds — stop and figure it out before moving on.
This is my extrapolation from the finding that “the gain is higher when intermediate files reveal errors.” The effect of this specific addition wasn’t measured in the paper.
🔧 Technique: move verification out of the agent’s head → an independent checker
Before submitting, write a separate script scripts/check.py. It must check
the output files for: presence of all columns, no empty values,
unique ids, valid ranges. Run it and attach the output.
Submit only if all checks pass.
This idea is inspired by the study: there, verification was external and hidden from the model. Here the agent builds itself a visible equivalent.