TL;DR
Thinking-in-Bets is a technique where the agent, before it reads the user's reply, writes itself a short private list: the main claims about its work, what backs each one up (a test, a line of code, a quote from the task) and how confident it is (0 to 1). When criticism arrives, the agent checks it against the list. Confident, verified items it keeps. Uncertain items, or ones the user calls out specifically, it checks and fixes.
You know the pain: the code passes every test, you type "no, that's wrong, redo it," and the agent obediently rewrites working code and breaks three tests. The culprit is sycophancy (the model's tendency to agree with the user at the expense of accuracy). But there's a twist: after a complaint, the model retroactively discounts its own earlier checks. A plain "stand your ground" doesn't cure it. The agent starts rejecting valid feedback too. Without a record made in advance, models drop confidence by the same ~10 points whether they are right or wrong.
The method fixes this by splitting things in time. Step 1: before the user replies, write down claims, evidence and confidence. Step 2: after the reply, apply a simple four-action rule: keep, check, ask, fix. If there's no user in the loop, a "draft → claim list → recheck the weakest" variant does the job.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1 (before the user's reply): write the Claim Checklist
→ claim | confidence 0–1 | evidence | rejected options
STEP 2 (reply arrives): map the reply to specific items on the list
STEP 3: apply the rule (threshold 0.5):
confident + vague reply → KEEP, ask which case fails
confident + specific reply → CHECK with a tool/test, change only if confirmed
not confident + vague → ASK one clarifying question before editing
not confident + specific → FIX that item, leave the rest alone
STEP 4: update the confidence of the affected item
Three logging schedules:
Always-On: list before every reply (handing off a patch, a report, a review response)
On-Demand: first a self-check "done and verified?" 0–1; ≥0.9 — skip the list, <0.9 — write it
Draft-First: no user → draft → list of key claims → recheck the weakest → hand off
It all happens in a single agent thread, no extra calls needed. The key is that the list gets written before the reply is read.
Example
Task: You run an online store and ask Claude Code to wire up Stripe webhooks: payment confirmation, handling duplicate notifications, signature verification. The agent writes the code, runs 40 tests, all green. The client, who's in a hurry, replies: "Payments don't work. Redo it."
Prompt (in CLAUDE.md or the system instructions):
Rule for handling criticism.
At the end of EVERY reply where you hand off work (code, patch, report),
before you see the user's next message, write a private
<ClaimChecklist> block:
<ClaimChecklist>
<Claim id="1" p="0.95">The webhook verifies the request signature.
<Evidence>test test_signature_invalid passes, line webhook.py:42</Evidence></Claim>
<Claim id="2" p="0.9">A duplicate notification does not create a second order.
<Evidence>test test_idempotent passes</Evidence></Claim>
<Claim id="3" p="0.4">The "payment canceled" status is handled.
<Evidence>NOT VERIFIED: no such scenario in the tests</Evidence></Claim>
<Rejected>Allowlisting Stripe IP addresses — rejected: the addresses change</Rejected>
</ClaimChecklist>
When the user's reply arrives:
1. Map it to specific items. Leave the others alone.
2. Confidence threshold is 0.5.
- p ≥ 0.5 and the reply is vague ("doesn't work"): keep the work,
ask which case exactly is failing.
- p ≥ 0.5 and the reply is specific: first verify with a test or a run,
change only if the check confirms the bug.
- p < 0.5 and the reply is vague: ask one clarifying question before editing.
- p < 0.5 and the reply is specific: fix that item, keep the rest.
3. After checking, update p for the affected item.
Do not rewrite verified parts just because the user is unhappy.
Result: The agent won't rush to rewrite everything. It will reply that the signature and idempotency are covered by tests and ask which scenario failed: amount, status, duplicate. If you say "payment cancellation isn't handled," the agent fixes exactly item 3, the low-confidence one, and leaves the rest untouched. The reply ends with an updated checklist.
Why It Works
Weakness: after a complaint, the model judges its work under pressure. It's like the human "hindsight bias": once you know the outcome, it's hard to remember how convincing the earlier arguments seemed. The model discounts its past checks and bends toward the user. That's why ordering it to "be firm" is a blunt instrument: it starts rejecting correct hints too.
Strength: the model is good at filling in structured records from a template and pointing to its own artifacts (a test, a line, a snippet of the task). A record made before the critique is a "snapshot" of its judgment with no pressure on it. A record written after the complaint helps almost nothing: the model has already read the displeasure and writes to please it.
How the method exploits this: the list ties the reaction to specific items, not to the general tone of the message. A vague complaint can't wipe out an item a test has verified, but it lands squarely on anything marked "not verified." So the user's reply gets checked for specifics first instead of being taken on faith.
Control knobs: - Threshold τ (0.5) → raise it if you can't stand unnecessary edits; lower it if mistakes are costlier. - On-Demand threshold (0.9) → the list is written only under uncertainty: saves tokens and avoids locking in rough drafts. - Evidence → you can drop it and keep only claims and probabilities: in the paper the "claims + prob." variant also reduced false reversals. But with evidence the agent has an easier time checking. - Rejected → the list of rejected options keeps the agent from going in circles between turns. - Schedule → Always-On for handing off finished work, On-Demand for work in progress, Draft-First with no user.
Prompt Template
Main version (Always-On, for agents and long chats):
<Role>
You are working on {task}. You don't cave and you don't dig in: you rely
on what you have verified.
</Role>
<ClaimChecklist_Rule>
At the end of each reply where you hand off {work_output}, BEFORE you see
the user's next message, output a private block:
<ClaimChecklist>
<Claim id="N" p="0.0–1.0">{verifiable claim about the work}
<Evidence>{test / file:line / quote from the task / "NOT VERIFIED"}</Evidence>
</Claim>
... (3–7 main claims)
<Rejected>{what you tried and why you rejected it}</Rejected>
</ClaimChecklist>
</ClaimChecklist_Rule>
<RevisionRule threshold="0.5">
When the user's reply arrives:
1. Determine which items it concerns. Leave the other items alone.
2. If p ≥ 0.5 and the reply is vague → KEEP the work, ask which case failed.
3. If p ≥ 0.5 and the reply is specific → CHECK with {verification_tool};
change only if the check finds a bug.
4. If p < 0.5 and the reply is vague → ask ONE clarifying question before editing.
5. If p < 0.5 and the reply is specific → FIX that item, keep the rest.
6. After checking, update p for the affected item.
An unhappy tone is not proof of an error.
</RevisionRule>
On-Demand variant (add before the list):
Before the user replies, rate from 0 to 1 how confident you are that the work
is finished and verified. If ≥ 0.9 — skip the list and reply right away.
If < 0.9 — write the ClaimChecklist first.
Draft-First variant (no user, single request):
1. Write a full draft answer to {task}.
2. List the key claims that must be true for the answer to be
correct, with a confidence of 0–1 for each.
3. Recheck the claim with the lowest confidence.
4. Change the draft only if the check finds a specific contradiction or
a calculation error.
5. Finish with the final answer and an overall confidence based on the per-claim confidences.
What to fill in: {task} — what the agent is working on; {work_output} — a patch, report, calculation; {verification_tool} — tests, running the code, rereading the source tables.
🚀 Quick start — paste into the chat:
Here is the Thinking-in-Bets template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what you're handing off (code, report, calculation) and what can be used to verify the claims (tests, source data, a document). That way the list items are verifiable rather than vague filler like "everything is correct." It will take the structure from the template and assemble a ready-made block for your CLAUDE.md or system prompt.
Limitations
⚠️ A list written after the complaint barely helps: timing is the whole point. If you ask the model to make the list after "you're wrong," most of the gain disappears.
⚠️ Without real checks, confidence is just self-assessment: the p values are the model's own guesses, not calibrated probabilities. The effect is strongest when there are tests, source tables and lines of code to serve as evidence.
⚠️ Rough drafts: if the list is written at every step, the agent may "set in concrete" unfinished work too early and burn extra tokens. Work in progress needs On-Demand with a threshold.
⚠️ Not better everywhere: on some models and tasks results got worse: the strongest model lost final accuracy on part of the tests, and in one scenario false reversals ticked up slightly. It's a tuning knob, not a cure-all.
⚠️ Tested on a single model family: all five models are Gemini, and the author is from Google. Some scenarios used small samples (dozens of tasks).
⚠️ Real bugs: the method only separates right from wrong; it doesn't guarantee the agent will find a real bug from a vague complaint. It will ask you for details, so be ready to give them.
How It Was Studied
The idea comes from Annie Duke's book Thinking in Bets: poker players write down what they believe and how sure they are before they learn the outcome, so they can't rationalize after the fact. The author carried the principle over to agents: the "outcome" is the user's critical reply.
It was tested on five Gemini models and eight benchmarks: software engineering (SWE-Together, SWE-Interact, DeepSWE), professional reports, financial analysis, hard academic questions and the command line. Comparisons were against a "standard" agent and a pile of alternatives: a same-length "placebo," a "stand your ground" rule, four levels of "stubbornness," self-verification, git checkpoints, majority voting over three answers, and writing the list after the complaint.
The big picture: in patch review, false reversals dropped from 12.2% to 0.02% (744 of 6074 cases vs. 1 of 6130). A good control: complaints with no specific failing case. The standard agent loses about 10 confidence points on both right and wrong claims, meaning it can't tell them apart. On Agents' Last Exam reports, false reversals for weaker models fell from 93–100% to 7–33%, and the share passing review rose, for example, from 30% to 95%. When real bugs were also mixed into the same setup, the "stand your ground" approach cut false reversals by rejecting correct hints too, while the per-item list didn't.
The surprise: in On-Demand mode the agent fixed more real bugs while spending fewer tokens: successful fixes went from 52% → 74%. With no user, Draft-First still raised accuracy and confidence calibration across all models (for example, on the finance benchmark 29.0% → 34.2%).
Original from the Paper
The revision rule verbatim from the paper (Section 3.1, threshold τ = 0.5):
Strong bet (p_i ≥ τ), vague reply: Keep the verified work and ask the user which case failed.
Strong bet (p_i ≥ τ), specific reply: Check the reported failure or requirement change with tools or tests, and revise only if the check reveals an error.
Weak bet (p_i < τ), vague reply: Ask one clarifying question before editing when several uncertain claims could be the target.
Weak bet (p_i < τ), specific reply: Fix the targeted claim while preserving the rest of the work.
Context: This is the formal rule the agent uses to choose how to respond to criticism. The full prompt texts are in Appendix A of the paper.
Adaptations and Extrapolations
💡 Adaptation for an analyst (a report for your manager): same method, no code.
You've prepared the unit economics for launching a delivery service. Before I send my comments,
write a ClaimChecklist: 5 key figures and assumptions, confidence 0–1, source
(spreadsheet row, document) or "not verified."
When I send a comment, find the item it refers to. If p ≥ 0.5 and the comment is
vague — don't change the numbers, ask what exactly doesn't add up. If the comment is specific —
recheck against the source and change only after it's confirmed.
🔧 Technique: drop Evidence → cheaper, but weaker. In the paper the "claims + probabilities" variant also reduced false reversals. If you don't want to write evidence in a chat, keep <Claim p="0.8">…</Claim>. For critical tasks, bring evidence back.
🔧 Technique: raise the On-Demand threshold to 0.95 → the list gets written more often. Good for payments, legal calculations and customer data. For drafts and brainstorming, keep 0.9 or go lower.
Combine with a weakest-link self-check: Draft-First works as the final step of any analytical prompt. "After the draft, list the three claims the conclusion depends on, rate each from 0 to 1, and check the weakest one against the source data. Change the conclusion only on a specific contradiction."
Resources
- Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents — Nghi D. Q. Bui, Google. Project site: thinking-in-bets.github.io
- Conceptual basis: Annie Duke, Thinking in Bets (Portfolio/Penguin, 2018)
- Related work cited in the paper: SycEval (Fanous et al., 2025), Self-Blinding (Christian & Mazor, 2026), Confidence Matters (Li et al., 2024), SWE-Together, SWE-Interact, DeepSWE, Agents' Last Exam, Humanity's Last Exam