Self-Guided Reasoning: Checking Data for Errors with Code the Model Writes Itself

STEP 1: Data analysis → task type, required sections, rules (+ self-check of this analysis)
STEP 2: Insight generation → validation path → description of each function (input/output) → code → run
        └ if the code crashes → error text goes back to the model → fix → re-run
STEP 3: Report → type, location and cause of each error → corrected version of the data

You can run all the steps in a single conversation if your chat can execute code (Code Interpreter, data analysis in Claude, Claude Code, Cursor). Without code execution the method is pointless.

Task: A logistics coordinator at an online store in Chicago is preparing tomorrow's export: 80 addresses, delivery windows, order weights, 5 vans with load capacity and mileage limits. Before uploading it to the route-planning service, they need to catch the errors: a “18:00–10:00” window, an order heavier than the cargo bay, zero coordinates, an address with no phone number, a van whose shift is too short for its route.

Prompt:

Below is tomorrow's order export (80 rows) and the parameters of 5 vans.
Find every error before it goes into the route-planning service.
Don't check by eyeballing. Work in three steps.

STEP 1. Analysis.
Determine: what kind of task this is, which sections of the data are required,
which constraints must hold (weight, time windows, shift length,
mileage limit, field format). Check your analysis for contradictions
with the data itself and fix them. Output a short summary.

STEP 2. Validation with code.
Based on the summary, make a plan: where format errors are likely,
and where constraint violations are likely. For each item in the plan, describe
a function: what it does, what it takes, what it returns.
Then write Python code and run it.
For every problem found, the code must print: row number,
field, value found, which rule was violated.
If the code crashes, read the error, fix it and run it again.

STEP 3. Report.
Match the code's output against the source data. Give a table:
error type / row / what's wrong / likely cause.
Then propose a corrected version of the data. Separately show
what you changed, and flag the edits you're not sure about.

Data: {paste or attach the file}

Result: The model first gives a short summary: task type, required fields, list of rules. Then it shows the validation plan, runs the code and prints a list of violations with row numbers. If the code crashes, you'll see the fix and the re-run. At the end you get an error table with type, location and cause, plus a corrected export with the questionable edits flagged.

The model's weakness. A model generates text from patterns; it has no counter or calculator. A check like “total weight on the route is no more than 1500 kg” or “arriving at 14:20 falls inside the 14:00–14:15 window” requires exact chains of arithmetic. The more rows and rules, the more of these chains. They pile up, and the model starts missing errors. Re-running the “eyeball” check just repeats the same weakness.

The model's strength. The model understands data structure well, knows the typical rules of a domain and writes working code. Code calculates precisely and doesn't get tired at 80 or 800 rows.

How the method uses this. The model handles what to check; the code handles the checking. Step 1 locks in the rules so the code doesn't check random things. Step 2 turns the rules into functions that print details for every error found. Step 3 forces the model to rely on the code's output, not on guesses.

Control knobs: - Self-check in step 1 → drop it for simple data, keep it for unfamiliar structures. If the model misreads the rules, all the code will check the wrong thing. - The “print row, field, value, rule” requirement → without it the report will be vague. You need it to pinpoint where each error is. - The “code error → fix” loop → keep it no matter what, otherwise the first typo in the code kills the process. - Function description (input/output) → makes the code more accurate, because the model first states the task in words. - Flagging uncertain edits → add it so you don't accept fixes blindly.

The paper moves the full prompt text to an appendix, and it's not in the main text. The template below is reconstructed from the description of the three stages, with the structure preserved.

You are given a dataset: {data_description}.
Your task is to find errors before the data goes into {where_the_data_goes}.
Don't check by eyeballing: do numerical and logical checks with code.

<Stage_1_Analysis>
Determine:
- task type: {what_this_object_is}
- required sections/fields that correct data must contain
- constraints that must hold: {list_of_rules}
Check your analysis for contradictions with the contents of the data
and fix it. Output a short summary.
</Stage_1_Analysis>

<Stage_2_Insights>
1. Based on the summary, determine where format errors are likely
   (missing values, invalid values) and where rule violations are likely.
   Lay out a step-by-step validation path.
2. For each step, describe a function: what it does, what it takes,
   what it returns.
3. Write code based on these descriptions. For every problem found,
   the code prints: where (row/field), what was found, which rule was violated.
4. Run the code. If it crashes, read the error message,
   fix the code and run it again. Repeat until it completes.
5. Collect the code's output into a list of insights about the errors.
</Stage_2_Insights>

<Stage_3_Report>
Match the insights against the source data. Give a report:
error type / location / what's wrong / likely cause.
Then fix the data. Show exactly what was changed,
and flag the edits you're not sure about.
</Stage_3_Report>

Data:
{data}

Fill in: {data_description} — what the table or file is; {where_the_data_goes} — the system or process where an error is costly; {what_this_object_is} — the type of task or process; {list_of_rules} — the rules you already know yourself (the more precise, the better); {data} — the pasted text or file.

🚀 Quick start — paste this into a chat that can run code:

Here is a template for checking data with code the model writes itself.
Adapt it to my task: {your task: what data it is and where it's going}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what the data is, which rules it must satisfy and what happens if there's an error. That's so it locks in the right rules at the analysis stage; otherwise the code will check the wrong thing. It'll take the template's structure and fit it to your table.

⚠️ You need code execution: without Code Interpreter, Claude Code, Cursor or something similar, the method turns back into a plain eyeball check and loses its point.

⚠️ The code only checks what the model thought to check: if in step 1 it misunderstood the rules or missed an important constraint, the code will faithfully check the wrong thing. This is the weakest link. Better to list your rules explicitly.

⚠️ Synthetic errors in a single domain: the researchers injected the errors into the data artificially (routing and bin packing). Real-world “dirty” data can be sneakier. The authors didn't test transfer to CRMs, accounting or price lists.

⚠️ False positives aren't covered in the main text: the analysis on clean data is in an appendix that isn't included here. Test on a file you know is clean so you don't get “errors” where there are none.

⚠️ Iteration helps on simple cases, not hard ones: repeated eyeball passes get close to 100% only on simple errors. On complex combinations of rules they don't help, and that's where you need code.

⚠️ Time: the method is a bit slower than a single direct request (in the study, 6 to 17 minutes per instance; the difference is small).

The researchers took 48 variants of vehicle routing problems, from simple ones to those where up to six types of constraints apply at once. Into each variant they injected 200 errors across 20 instances: format errors (missing values, invalid values) and rule violations (capacity, time windows, mileage limits and so on). First they measured how the model does directly. Then they compared that with the three-step method.

The picture came out clear. The model catches format errors and simple constraints well, at roughly 85–95%. Time windows and mileage limits, at 0–60%. The more errors, rows and constraint types, the worse it gets: with 100 nodes the results are lower than with 50, and on the hardest variants the direct approach found only 14% to 30% of errors. Repeated iterations (“find, fix, look again”) pushed the simple variants close to 100%, but the hard ones stayed “far from ideal.”

Self-Guided pushed results to 90–100% almost everywhere. For example, the time-window variant rose from 66% to 95%, and one of the hardest went from 14% to 99%. The method was also tested on a different problem (bin packing) and on different models.

The surprise: repeated runs don't help where the model can't calculate. It's not a question of how many attempts, it's a question of approach: “looking” versus “calculating with code.” The practical takeaway: if a check requires arithmetic, don't ask the model to think harder, ask it to write the check.

🔧 Technique: add a second pass → catch what slipped through the first time

After the data is fixed, run the same checks again:

You've fixed the data. Run the same validation code on the corrected version.
Show whether any violations remain. If they do, repeat step 3.

In the study, iterations helped on simple errors. Here they verify that the fix didn't create new problems.

🔧 Technique: test on clean data → guard against false positives

First give the model a file you know for sure has no errors, and see whether it finds any “problems.” If it does, tighten the rules in step 1.

Extrapolation: a rule for your agent (CLAUDE.md / AGENTS.md)

The principle “check numbers with code, not in your head” can be baked into an agent's instructions. This is my own extension; it's not in the paper:

Data validation rule:
- Any check that requires adding, comparing ranges or calculating
  step by step must be done with a script, not by reasoning.
- For every problem, the script prints the row, field, value and the rule violated.
- If the script crashes, fix it based on the error text and run it again.
- Base your report on the script's output. If there is no output, don't claim there are no errors.

• Paper: Self-Guided LLM Reasoning for Error Diagnosis in Vehicle Routing Problems
• Authors: Ni Zhang, Zhiguang Cao (Singapore Management University), Jianan Zhou (Nanyang Technological University), Yue-Jiao Gong (South China University of Technology)
• Related work mentioned: MOID (error diagnosis via solver results), OPRO, FunSearch, ReEvo, AFL
• Full prompts are in Appendix D.1–D.3 of the paper (not included in the provided text)