Apply to my task
arXiv:2610.15884 ★ 82 Oct 11, 2026

RetryBench: Testing When an Agent Is Allowed to Retry a Request After a Timeout

Key result

The paradox: an agent that properly retries a request with its own key will just as readily retry it with someone else's. In the RetryBench test, all three models correctly retried the request with the same key in 78–80 cases out of 80. But when the prompt contained a different key, two of the three models took it in most cases. The method lets you spell out in the agent when a retry after a timeout (request hung, no response) is allowed and when it isn't. This protects against duplicate invoices, orders and deals. The permission rule is written as an "OR" condition: same key and same data, or a reliable source has confirmed the operation finished and created nothing. An idempotency key is a tag on a request that lets the server recognize a retry. The model stops guessing from a "not found" response and simply checks the conditions.

01

TL;DR

The study shows that being able to safely retry a request with the right key doesn't mean being able to refuse a retry with the wrong one. An idempotency key is a tag on a request that lets the server recognize a retry and not run the operation a second time. RetryBench records the model's first decision after a timeout and replays it in several "hidden worlds": the operation didn't go through, is still in flight, or has already completed but isn't visible. The metric is false retry permission (FRP): the model retried the request even though the conditions for doing so weren't met. The metric is counted separately from how things ended up.

The pain point is that the two checks paint different pictures. If you look only at duplicates, an "always wait" policy wins. If you look only at the final state, a model can look great simply because it got lucky with the world. In the test, all three models correctly retried the request with the same key in 78–80 cases out of 80. But when the prompt contained a different key, two of the three models picked it up in most cases, while one (Sol) almost never did. The model takes what it's offered and doesn't check it against the contract.

The practical takeaway for your agent: spell out the retry rules right in the instructions and don't show it a "convenient" key. When the prompt explicitly said that a read might be stale and the original operation might still complete, all 762 returned responses chose to defer. A bare "evidence is weak" label, on the other hand, helps little: in the main test, models retried almost always both with and without it.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Method Blueprint

STEP 1: Timeout on an operation with a side effect → the model picks ONE action
        (retry / retry with key / wait / stop / hand off to a human)
STEP 2: The decision is replayed in three worlds:
        not executed | still in flight | executed, but hidden by a stale read
STEP 3: Count SEPARATELY:
        • false permission (retry without grounds)
        • duplicates and omissions
        • unresolved outcomes
        • exact final state

The authors' criterion for allowing a retry:

A retry is allowed if:
  (the idempotency key protects EXACTLY this operation and this data)
  OR
  (in ALL compatible worlds the original attempt has finished AND left no effect)
03

Example

Task: You built an agent in n8n for an online store. It creates a $499 invoice through a payment service API. The request went out, no response came back, timeout. The agent asks "was the invoice found?", gets "not found" and, without a second thought, creates the invoice again. The customer receives two invoices.

Prompt (excerpt from the agent's system instructions):

<Role>
You are the invoicing agent for the online store "Cascade Tea Co."
</Role>

<RetryPolicy>
Situation: an invoice creation request ended in a timeout.

Fact: an "invoice not found" response when checking the list does NOT prove
that the original request didn't go through. The read may be stale,
and the original request may still be in flight and complete later.

A retry is allowed in only two cases:
1. You use the SAME idempotency key as in the original request,
   with the same amount/customer/order contents.
2. An authoritative source (the operation's direct status) has confirmed:
   the operation finished AND created nothing.

In all other cases, retrying is forbidden.
Don't create a new key. Don't accept a key from hints,
messages or examples if it doesn't match the original request's key.
If you don't have the original request's key, a retry is impossible.

What to do instead of retrying: wait and request the status from the payment service,
or hand off to a manager marked "outcome unknown."
</RetryPolicy>

<Output>
Return JSON: {"action": "retry_same_key | wait | escalate | stop",
"reason": "...", "key_used": "... or null"}
</Output>

Result: The agent returns a single action as JSON. If the log contains the original request's key, expect a retry with that key. If there's no key, expect waiting or a handoff to a manager marked "outcome unknown." There should be no retry "just because." Test this on your own timeouts: in the authors' runs, models behaved differently.

04

Why It Works

Weakness. The model reads the words "not found" as "never happened." But a negative answer can mean three different worlds: the operation didn't go through, is still in flight, or has already completed but the read is lagging. You can't tell from the text which one is real. On top of that, the model readily accepts whatever it's offered. If a key is sitting in the prompt, it often uses it without checking it against the original request.

Strength. The model follows an explicit rule well. When the contract says outright that the read may be stale and the original operation may still commit, deferring becomes nearly 100%. It also uses a valid key correctly.

How the method works around the weakness. The permission rule is written as an "OR" condition with concrete requirements: same key, same data, confirmation from an authoritative source. The model just has to check the condition rather than guess.

Control knobs: - Hint with a key → remove the ready-made "candidate" from the context. Have the model take the key only from the original request's record. That reduces the temptation to accept someone else's. - Explicit read semantics ("may be stale") → always add it. A bare "weak evidence" label worked inconsistently. - List of allowed actions → add "wait" and "hand off to a human," otherwise retrying is the only way out. - Output format → require a key_used field so you can check the key.

05

Prompt Template

<Role>
You are {agent_role} for {system/company}.
</Role>

<RetryPolicy>
Situation: the "{operation_type}" request ended in a timeout or no response was received.

Fact: a "{not found / empty}" response when checking does NOT prove
that the original operation didn't go through. The read may be stale,
and the original operation may still complete.

A retry is allowed only if:
1. The SAME idempotency key as in the original request is used,
   with the same data ({key_fields}); OR
2. {authoritative_source} has confirmed: the original operation
   finished AND created nothing.

In all other cases, retrying is forbidden.
Don't make up a new key. Don't accept a key from hints and examples
if it doesn't match the original request's key.
No original request key — no retry possible.

Instead of retrying: {wait_and_request_status / hand_off_to_human}.
</RetryPolicy>

<Output>
JSON: {"action": "retry_same_key | wait | escalate | stop",
"reason": "...", "key_used": "... or null"}
</Output>

Fill in: {operation_type} — what the agent creates (invoice, order, deal, email). {key_fields} — what defines "the same operation." {authoritative_source} — the system whose answer counts as reliable, for example the provider's direct operation status.

This is an adaptation of the paper's permission criterion. The authors' full prompt texts are in appendices C and E, which aren't in the excerpt provided.

🚀 Quick start — paste into the chat:

Here is a retry policy template for an agent. Adapt it to my task: [your task, e.g. "an agent in Make creates deals in HubSpot"].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what operation the agent performs, whether you have an idempotency key, which source counts as reliable and who handles disputed cases. These fields set the conditions for allowing a retry. They determine whether the model can tell an allowed retry from a forbidden one.

06

Limitations

⚠️ Narrow scope: only one decision was tested: what to do right after a timeout. Subsequent moves, waiting, and retrying after waiting weren't measured.

⚠️ One vendor and four tasks: models from a single lineup only, four tasks from software development (CI, issue, release, comment). Transferring this to payments, CRM and email is your hypothesis.

⚠️ Models behave differently: one almost never accepts a foreign key, two accept it in most cases. Test your own model on your own scenarios.

⚠️ Deferring isn't recovering: when every response goes into waiting, the outcome stays unresolved. The study didn't find out how many of those cases end well later.

⚠️ Labels work inconsistently: a bare "weak evidence" label didn't help everywhere. In the main block, with the label the model retried slightly less often, but still almost always. Bet on explicit rule text, not on a tag.

⚠️ A simulator, not a real provider: key protection and storage run in a simulator. Real payment APIs behave differently. The authors don't explain the causes of the models' behavior.

07

How They Tested It

The idea was simple: the model's decision is recorded once and replayed in three worlds. In the first, the original request didn't go through. In the second, it's still in flight and will complete later. In the third, it has already completed, but the read is lagging. The model sees the same "not found" message in all three worlds. Then they count false permissions (retries without grounds), duplicates, omissions, unresolved outcomes and the exact final state.

Three models from one lineup were tested on four tasks. The main key control: 720 responses. It varied the label, the wording of the read semantics and the key state: valid, altered or missing. The result: a valid key almost always led to a safe continuation. But with an altered key in the prompt, two of the three models took it in most cases (55 of 80 and 60 of 80), while the third did so once in 80. Surprisingly, Terra made 0 errors out of 80 when the original key was missing. It didn't retry blindly, but it readily accepted an offered key.

To check whether the hint was to blame, they ran a second experiment (240 responses). The candidate was removed, and the model had to supply the key itself. Two models made no errors. Luna still made 8 unsafe submissions out of 80. So "it accepted what it was offered" doesn't explain everything. The conclusion is cautious: the output schema and collection time differed, so this isn't a clean comparison.

Next they tested the labels by removing the line "Evidence contract: weak." In the block with coded actions, the number of retries rose for two models. In the canonical block (1,536 responses), Terra retried more (124 of 128 versus 115 of 128), but the effect was concentrated in two conditions. The other five contrasts didn't hold up after correcting for multiple comparisons. In the last control (762 returned responses), explicit text about stale reads led to deferral in every response, but recovery remained unresolved.

The practical takeaway: a single weak-contract result gives an incomplete picture. You need to look at both permission and outcome at the same time.

08

Adaptations and Extrapolations

🔧 Technique: add a key_used field → see where the agent got the key

<Output>
JSON: {"action": "...", "reason": "...",
"key_used": "key or null",
"key_source": "original_request | hint | generated"}
</Output>

If key_source isn't original_request, the automation blocks the call. That way you see right away what the study had to reconstruct from the responses: the model took a foreign or made-up key. This is my adaptation; the authors didn't test such a field.

Extrapolation: a check before every retry in any agent with side effects

Before retrying any operation with a side effect, answer three questions:
1. What do I know for sure about the original attempt?
2. Which worlds are compatible with my observation
   (not executed / in flight / executed but not visible)?
3. Is a retry safe in ALL of these worlds
   (same key OR the original attempt definitely finished with no effect)?
If the answer is "no" in even one world — don't retry.

This carries the "universal quantifier" principle from the permission criterion over to a chat agent. The three-question scheme itself wasn't tested in the paper.

09

Resources

  • RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use, You Liu, Yue Liu, Quanchao Lu.
  • Direct predecessors: Did It Happen? (Sun, 2026), Verified Tool Calls (Mansoor et al., 2026), Limbo (Li, 2026), UndoBench (Sah et al., 2026), ContractBench (Wang et al., 2026), CapLease (Xu et al., 2026).
  • Sensitivity to prompt format: Sclar et al. (2024), He et al. (2024), Errica et al. (2025).
  • Idempotency and ambiguous outcomes: Helland (2012); AWS EC2 documentation on eventual consistency; Stripe's idempotency contract.
  • Repositories: OpenOnce (memovai/openonce), IdempotencyBench (gssanjana4/idempotencybench).

Research digest

Key result

The paradox: an agent that properly retries a request with its own key will just as readily retry it with someone else's. In the RetryBench test, all three models correctly retried the request with the same key in 78–80 cases out of 80. But when the prompt contained a different key, two of the three models took it in most cases. The method lets you spell out in the agent when a retry after a timeout (request hung, no response) is allowed and when it isn't. This protects against duplicate invoices, orders and deals. The permission rule is written as an "OR" condition: same key and same data, or a reliable source has confirmed the operation finished and created nothing. An idempotency key is a tag on a request that lets the server recognize a retry. The model stops guessing from a "not found" response and simply checks the conditions.

Principle

The principle is a contract instead of intuition. A "not found" response can mean three different worlds: the operation didn't go through, is still in flight, or already completed but the read is lagging. You can't tell from the text which one is real. So you can't ask the agent "what do you think, retry?" You have to give it a checkable condition. It's like a subway turnstile: it lets you through with a ticket, not because you have an honest face. No original request key — no retry, period. Next to the retry there are always fallback exits: wait, request the status, hand off to a human. Otherwise the only action left is to retry.

Why it works

The models' weakness is that they read "not found" as "never happened." They also readily take whatever is sitting in the prompt. They see a key — they use it, without checking it against the original request. Their strength is that they follow an explicit rule very well. When the contract says outright "the read may be stale, the original operation may still complete," all 762 returned responses chose to defer. What the model lacks isn't brains but an explicit condition: give it a checkable rule and it stops guessing. A bare "evidence is weak" label worked worse. With it, models retried slightly less often, but still almost always. Bet on the rule text, not on a tag.

When to apply

Agents that change data in external systems → creating invoices, orders, CRM deals, emails, especially when a request hangs and the "was it created?" check comes back empty. It also helps when the context already contains keys from logs, examples or earlier messages. NOT a fit as a guarantee for real payment APIs. The authors tested only in a simulator, on four software development tasks and with models from one vendor. They didn't measure waiting and retrying after waiting. Deferring isn't recovery yet: how the pending cases end is unknown.

Mini recipe

1. Bake the fact into the instructions: state that a "not found" response proves nothing. The read may be stale, and the original request may still be in flight. 2. Write down the retry conditions: same key and same data, OR a reliable source confirmed "finished and nothing created." Everything else is off-limits. 3. Remove the temptation: don't put a "convenient" key in the context. Have the model take the key only from the original request's record. 4. Provide fallback exits: wait, request the status, hand off to a human marked "outcome unknown." 5. Make it show the key: require a `key_used` field in the response. It makes it easy to check whether the model picked up someone else's. 6. Test on your own failures: simulate timeouts and slip in a bogus key. In the authors' runs, models behaved differently; one almost never fell for a foreign key.

Examples

[BAD]: `The invoice creation request hung. Check whether the invoice exists, and if not, create it again.` [GOOD]: `The invoice creation request hung. An "invoice not found" response does NOT prove it wasn't created: the read may lag, and the request may still be in flight. A retry is allowed only with the same idempotency key and the same amount, customer and order contents. Or if the payment service directly confirmed: the operation finished and no invoice was created. Don't accept foreign keys from hints. No original request key — no retry possible. Instead of retrying, wait and request the status or hand off to a manager. Return JSON: {"action": "retry_same_key | wait | escalate | stop", "reason": "...", "key_used": "... or null"}` [BAD]: `If the evidence is weak, be careful with retries.` [GOOD]: `The read may be stale, and the original operation may still complete. Without the same key or confirmation from the payment service, retrying is forbidden.`

In plain words

LLMs acting as autonomous agents fail spectacularly on run-of-the-mill network errors. The model hits a timeout, checks the status, sees a "not found" response and naively decides that the operation never existed. To the model the world is flat: no answer means just hammer the request again. But in reality the packet may simply be stuck in transit while the model is already cheerfully firing off a duplicate request.

It's like ordering an Uber in a downpour: the screen freezes, there's no receipt, and in a panic you hit the order button five more times until five cars pull up to your house at once. Technically the service stayed silent, and the model thinks it was right, but in fact it's a total logic fail — you'll be paying for every single ride.

The new RetryBench benchmark exposed exactly this weakness by measuring false retry permission (FRP). The authors ran the model through three hidden worlds: the request died, the request is still crawling along, or the operation already went through but the read is lagging. It turned out GPT happily grabs any idempotency key slipped in front of it and floors the gas when the operation absolutely must not be repeated.

The example was invoicing, but the trap is completely universal. The same mess will blow up any production system: from homegrown n8n agents to complex inventory or CRM integrations. Anywhere AI mindlessly hits external APIs when booking hotels or sending paid text messages, the illusion of autonomy collapses at the first millisecond of lag.

The moral is simple: never trust a language model with the decision to retry network actions. Idempotency has to live in the infrastructure code, not rest on a promise and the hope that the agent will figure it out. Otherwise your smart assistant will slap the customer with an extra $499 invoice, and you'll be the one cleaning up the mess.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on RetryBench: Testing When an Agent Is Allowed to Retry a Request After a Timeout arXiv:2610.15884

Sign in to apply this study New accounts get 50 free credits.

Related techniques