TL;DR
Probe-and-Resume is a setup where the agent works undisturbed. When it declares "done," a separate critic (a second, reviewing model) receives a copy of the final state and a copy of the initial one. The critic doesn't read the log; it runs checks: commands, tests, comparisons. If it finds holes, the agent picks up from the same spot with the feedback in hand. It doesn't start over.
The main finding: evaluating an agent's work as it goes is almost useless. At early stages, the critic predicts how a task will end only slightly better than a coin flip. A critic that only reads files and the log rejects more than half of correct solutions with "insufficient evidence." The agent's log records commands and output, while the actual system state (files, processes, packages) stays off-screen. The critic can't see it and can't verify it. Afraid of being wrong, it rejects.
The method comes down to three decisions. When: the critic is called only on a "done" claim. How: it's allowed to execute commands, including destructive ones, on a disposable copy. What we provide: a copy of the initial state, so it can check requirements like "this file must not change." As a result, the share of solved tasks rose from 63% to 72%. Cost roughly doubled, but stayed at half that of an off-the-shelf competitor agent on the same model.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: The agent works alone, unaware of the critic → declares "done"
STEP 2: The critic extracts requirements from the task → requirements list
STEP 3: The critic verifies EACH requirement by running something
on a copy of the final state + a copy of the initial one → per-item verdict
STEP 4a: Everything satisfied → submit the result
STEP 4b: Failures found → the agent returns TO THE SAME state
+ receives: what's wrong, why, how it was checked
+ request: fix it OR prove with a command that the requirement is met
→ back to STEP 1 (the critic continues THE SAME conversation, rechecks the whole list)
Stop: critic accepts / 6 rounds / budget runs out
Steps 2–3 are done by a single critic in a single conversation. The copies are disposable; anything the critic breaks in them gets thrown away. Only the text of the feedback reaches the agent.
Example
Task: You own a small Amazon store. An agent in Claude Code is rewriting the nightly script that exports stock levels from QuickBooks to the Amazon Selling Partner API. Conditions: don't change the stocks.csv file format, don't touch the .env config, and if the API goes down the script must not overwrite yesterday's stock levels. The agent writes "done, everything works." Previously you took its word for it and found zeroed-out inventory in the morning.
Prompt (for a reviewer subagent you invoke once the agent says "done"):
You are the critic. The agent has claimed the task is complete. Your job is not to believe it
but to verify by running things.
The task, exactly as the agent received it:
Rewrite the sync_stocks.py script (exports stock levels from QuickBooks to the Amazon Selling Partner API).
Requirements: 1) the stocks.csv format does not change; 2) the .env file is unchanged;
3) on an API error the script does not overwrite yesterday's stock levels; 4) the script runs
from cron with no interactive input.
Where to check: a copy of the current project is /work/candidate,
a copy of the project before work began is /work/initial.
Don't be afraid to break anything: both folders are disposable.
Procedure:
1. List all requirements from the task, including implicit ones.
2. For each requirement, design a check you can RUN,
and run it. Reading the code is not enough.
Example for #3: point the API address at a nonexistent host, run the script,
see what happened to the stock file.
3. For "unchanged" requirements, compare /work/candidate with /work/initial
(hash, diff).
4. If something can't be checked, write "not verified"; don't invent a verdict.
Answer format for each requirement:
- Requirement:
- How I checked (command):
- What I got:
- Verdict: met / not met / not verified
At the end: ACCEPTED (all met) or REJECTED + a list of failures.
Result: The critic produces a numbered list of requirements. For each one you see the check command, its output and the verdict. For #3 it will most likely run the script with a "broken" API address. For #2 it will compare the hashes of .env in the two folders. If something fails, you get a rejection text that you hand to the agent in full: "fix it or prove with a command that the requirement is met." The agent continues in the same project.
Why it works
The weakness. The agent's log records only its thoughts, commands and output. The actual system state (files, processes, packages, data) never makes it into the log. A critic that reads only the log judges "by the story." It can't confirm, and when it can't confirm, it rejects. That's why it throws out more than half of correct solutions. "Halfway" evaluations don't help much either: the outcome isn't settled yet, and you can't see whose solution will make it to the end.
The strength. A model is good at writing and running checks, meaning it can solve a terminal task. The authors put it this way: evaluating a terminal task is itself a terminal task. Give the critic a terminal and it starts proving instead of guessing. False rejections drop from half to about a fifth.
How the method exploits this. It moves verification to the only moment where there's something to verify, the "done" claim. It gives the critic the right to act on a copy, so destructive checks are safe. It adds a copy of the initial state, giving the critic a reference for "preserve" and "restore" requirements. Resuming instead of restarting keeps everything the agent already got right.
Control knobs: - Critic permissions. "Read only" → "run anything on the copy." The main knob: the false-rejection rate depends on it. - Copy of the initial state. Remove it and the critic can't check "must not change." The critic itself decides when to compare, and on such tasks accuracy rises noticeably. - Round limit (6 in the paper) and the agent's move limit after a rejection (50) → lower them to cap costs. - The phrase "or prove it with a command." Lets the agent contest a mistaken rejection instead of guessing what the critic wants. - One critic conversation for all rounds. It remembers its requirements list and rechecks it in full.
Prompt Template
This is a reconstruction from the method description in §4.1. The actual prompts from the appendix weren't in the available text. I kept the structure from the paper: requirements list → verification by running on a copy → comparison with the initial state → feedback → return.
Critic prompt:
<Role>
You are the critic of an agent's work output. The agent has declared "done."
Your task is to verify that claim by running things, not by reading.
</Role>
<Task>
{original_task_verbatim}
</Task>
<Environment>
Candidate (the agent's final output): {path_to_final_copy}
Initial state (before work began): {path_to_initial_copy}
Both copies are disposable. You may delete, overwrite, install,
start and stop processes. Changes will not propagate back to the agent.
</Environment>
<Procedure>
1. List all requirements from the task: explicit and implied.
2. For each requirement, design a check you can run,
and run it. Reading code and files is not enough.
3. For requirements like "preserve / don't change / restore," compare
the candidate with the initial state.
4. If something can't be checked, write "not verified." Don't invent a verdict.
</Procedure>
<OutputFormat>
For each requirement:
- Requirement:
- Check command:
- Result:
- Verdict: met / not met / not verified
Outcome: ACCEPTED or REJECTED + a list of failures explaining
why each one failed and how you checked it.
</OutputFormat>
Message to the agent after a rejection:
Your result was rejected. The critic checked it and found:
{failures_with_explanations_and_check_commands}
You continue from the same state. Either fix the problems,
or present a command that proves the requirement is met.
When you're finished, report "done" again.
Message to the critic on a repeat round (same conversation):
The agent has revised its result. Here are the commands it ran: {agent_commands}.
Here are fresh copies: {new_paths}.
Go back to your requirements list and recheck all of it from scratch.
What to fill in: {original_task_verbatim} is the brief the agent received, not a paraphrase. The paths are two copies of the project: the final output and the state before work. In git, that's a "before" branch/commit and an "after" working copy. {agent_commands} is what the agent did after the rejection.
🚀 Quick start: paste this into the chat:
Here is a critic template for checking an agent's work with the Probe-and-Resume method.
Adapt it to my task: [describe what the agent did and what the requirements were].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what the task's requirements are, which of them are "don't change" requirements, and how to create copies of the final and initial state (git, a folder copy, a container). That's because the critic's work rests on the requirements list and the two copies: without them it can neither run checks nor compare.
Limitations
⚠️ You need copies of the environment: in the paper, the critic works on disposable forks (snapshots) of the system. In a regular chat without tools, the method can't be reproduced. In Claude Code or Cursor you can approximate it with git branches, folder copies and a separate subagent. Destructive checks only on copies: on a "live" project, the critic can break things.
⚠️ The critic stays lenient when the reference is hidden: if the correct answer can't be derived from the environment (a hidden reference value, a client's unstated expectations), the critic tends to agree with the agent. Here false acceptances were noticeably more common than on tasks where the result can be run and checked.
⚠️ Errors don't vanish: a critic that runs checks is still wrong in about a fifth of cases, including accepting wrong solutions. The method raises the success rate, but important tasks still need human review.
⚠️ Cost: token spend roughly doubled compared with an agent without verification. That's cheaper than checking at every step, but not free.
⚠️ Limits of the conclusions: the authors themselves write that they didn't prove the "done" moment is optimal. A different search policy or stronger evaluation of intermediate steps might have worked. The main experiment uses one executor model on terminal tasks.
How they tested it
The team took Terminal-Bench 2.1: 89 terminal tasks, each run 5 times. The agent ran on a single model. First they tested the obvious idea: give the agent "save/roll back state" tools and run a tree search over options (MCTS) with a critic on intermediate steps. It didn't help: the tools were almost never used (in 3 runs out of 445), and MCTS gave +0.6 points at roughly four times the cost.
Then the authors dug into why the critic fails. They built three variants: no tools (reads the log), "read only," and "can execute anything." The critics were tested on intermediate states. For each state, the agent played the task out to the end 5 times, and that was compared with the critic's assessment. Then they tested on 406 final states against the benchmark's real verification script. On intermediate steps, the critics ranked states only slightly better than random (at 25% of the trajectory, almost a coin flip), and environment access helped little. On final states it's a different story: the executing critic cut the share of wrong verdicts from 36–39% to 22%. The surprise was that even on tasks where the verification script only reads files, the "read only" critic rejected 42% of correct solutions, in 93% of cases for "insufficient evidence." It needs to compute the check, not read it.
A separate experiment showed the value of the initial-state copy. False rejections fell from 24% to 17%, and disagreement between two verdicts from the same critic fell from 18% to 9%. The critic decided on its own when to consult the initial copy (in 25% of assessments), and that's exactly where accuracy rose by almost 10 points. Verification didn't get stricter, it got more accurate.
Final result: 63.4% → 72.4% (improvement on 30 of 89 tasks, regression on 1). With other critic models the gain is comparable (+9.9 and +7.2 points). Restarting the agent from scratch after a rejection instead of resuming came out 1.6 times more expensive and no better. The same result holds on SWE-bench Pro, judging by the paper's text.
Adaptations and extrapolations
🔧 Technique: add a "don't say done" rule to the agent's instruction file → self-check before the critic
The paper tested an external critic, not self-checking: the research it cites shows that self-correction without external data can make results worse. So this adaptation is my guess, not a conclusion from the paper. In
CLAUDE.mdyou can add:Before reporting "done": list the task's requirements. For each one, run a check and show the command and its output. If something can't be checked, write "not verified," not "works."
🔧 Technique: remove the initial-state copy from the template → watch the critic start rejecting "out of caution"
A quick test on your own task: run the critic twice, with
{path_to_initial_copy}and without. If rejections on "don't change" requirements disappear when the copy is present, the critic was missing a reference, and the agent wasn't actually wrong.
Resources
- Probe-and-Resume: When and How to Evaluate Agent Execution, preprint, Columbia University.
- Authors: Jiakai Xu, Tiancheng Ge, Tujie Guan, Yusheng Li, Rohan Timmaraju, Eugene Wu, Kostis Kaffes.
- Benchmark: Terminal-Bench 2.1 (Marten et al., 2026); run via Harbor and the Terminus-2 agent.
- Branching environment: Waypoint (Xu et al., 2026).
- Related work: Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), CRITIC (Gou et al., 2023), Huang et al. (2023) on self-correction without external feedback.