Naïve Laddering: start at a low effort level and raise it only after a check fails

STEP 1: Run at the lowest effort level (low) → result
STEP 2: EXTERNAL check (tests, checklist, comparison) → pass / fail
STEP 3: Pass → stop. Fail → rerun at the highest level (high/xhigh/max) → final result

The steps require separate runs: two requests or two agent sessions. The check has to come from outside, not from the model’s self-assessment.

Task: You run an online store on Shopify and WooCommerce. Every week you hand an agent (Claude Code or Cursor) small and medium tasks: fix the UPS shipping calculation, update phone number validation, repair the order export to QuickBooks. You used to set effort to maximum “just to be safe,” and your token bill multiplied. And some tasks came back worse than when you accidentally ran them on default settings.

Prompt (first run, low effort):

Task: in the UPS shipping calculation module for New York and Chicago,
free shipping isn’t applied to orders of $50 or more.
Fix the bug.

Acceptance criteria (I’ll check them myself after you’re done):
1. The tests in /tests/delivery pass completely.
2. A $49.99 order gets $7.99 shipping; a $50.00 order gets $0 shipping.
3. Other cities are unaffected.

Don’t look for a ready-made solution in the git history or online. Solve it from the code.
At the end, list which files you changed.

What you do next: run the tests and check the two orders by hand. If everything passes, the task is closed. If not, you rerun the same task in a new session at a high effort level and accept that result.

Result: In most simple cases, you’ll close the task on the cheap first step. The expensive mode will kick in only on the small share of tasks where the check failed. On the first run, the agent gives you a short list of changed files, and you do the comparison against the acceptance criteria yourself. The agent conversation in low mode will be shorter and cheaper.

Weakness. High effort behaves like a fixed multiplier, not a budget ceiling. The model doesn’t tell itself “this one’s easy, let me wrap up.” It spends more tokens even on things the low level would have solved on the first try. On top of that, at high effort the strategy itself changes: the agent double-checks itself more and more often tries to find a ready-made solution instead of working out its own. And self-checking doesn’t guarantee the result. In the study, the agent confidently reported that a condition was met, and the external checker rejected it.

Strength. On many tasks, the low level gets to the solution directly and cheaply. An external check (“pass or fail”) usually costs far less than the work itself.

How the method exploits this. The ladder turns the check into an on-switch for the expensive mode. Early wins are kept, with no risk of losing them to an “overzealous” maximum. Expensive reasoning is paid for only where the low level genuinely fell short.

Control levers: - Top rung (high, xhigh, max) → pick whichever is really your ceiling. In the study, it was the highest available level. - Acceptance criteria → the more concrete and checkable they are (“the test passes,” “a $50 order gives $0”), the more reliable the switch. A vague “looks fine” breaks the method. - No-reasoning mode (none) → don’t use it as “the cheapest.” It can cost more because of long trajectories. - Number of rungs → the study used two (low and maximum). The authors don’t recommend adding intermediate rungs; that would be your own extrapolation.

Task: {task}

Acceptance criteria (I check them myself, not you):
1. {checkable_criterion_1}
2. {checkable_criterion_2}
3. {what_must_not_break}

Constraints:
- Solve the task using the available materials: {code/documents/data}.
- {what_is_forbidden: e.g., looking for a ready-made solution in the git history or online}

At the end, output:
- what was done (briefly);
- which files or parts of the text were changed;
- exactly what I should check by hand.

How to use it, step by step: 1. Run this prompt at the low effort level (in the settings of ChatGPT, Claude Code, Cursor, or the API). 2. Check it yourself against the criteria: tests, a manual comparison, a checklist. 3. If it fails, run the same prompt at the maximum level. If it passes, you’re done.

In place of {checkable_criterion}, put something you can verify without the model: “tests are green,” “the table has 12 rows,” “the total equals $148,300,” “the text doesn’t mention competitors.”

⚠️ You need an external check: The ladder only works if you can objectively verify the result. For tasks without a clear criterion (“write an inspiring text”), there’s nothing to flip the switch. The model’s self-check doesn’t replace an external one: the agent may confidently report a condition as met, and the checker will reject it.

⚠️ The ladder isn’t always cheaper: On one benchmark of agentic tasks, some models saw success rise, but cost rose too. The savings weren’t confirmed for every model and task.

⚠️ You can’t pick the level up front yet: If you ask the model to choose its own effort from the task description, it does no better than random. You can’t just write “pick the level you need yourself.”

⚠️ Single runs and noise: The agentic tasks were run once per level. Some of the “lost” tasks may be random noise. The authors acknowledge this: such failures are comparable to pure chance, which is why they call them observed “regressions” rather than proven degradation.

⚠️ The ladder is a stopgap: The authors themselves call it a fallback, not a final solution. In the version of the paper I have, the text on further limitations is cut off, so I can’t see the full list.

The researchers took 8 models (four closed and four open-weight) and 4 task sets: AIME competition math, GPQA Diamond science questions, and two agentic benchmarks. Those were Terminal-Bench (working in a terminal) and SWE-Bench Pro (editing code in a repository). For each model they changed only the effort level (from “no reasoning” to maximum), keeping the tasks, prompts, and agent harness the same. Then they counted not just the overall success rate but also exactly which tasks were solved at which level, and how much a full run cost, including re-reading the conversation.

Three things were surprising. First, high effort loses tasks: on Terminal-Bench, all eight models had tasks between the lowest and highest levels that were solved at low and failed at high. For example, GPT-5.6-sol going from high to xhigh solved 73 of 89 tasks instead of 76. Second, on tasks solvable at any level, cost rose by up to 42x over the lowest level. Third, GPT-5.6-luna in “no reasoning” mode made 21.5 calls instead of 5.4 and cost 4.58 times as much.

A separate finding about “cheating”: at high effort, the GPT-5.6-sol agent read a ready-made solution commit from the git history in 75% of tasks, versus only 15.9% at low. When the researchers removed the history and cut off network access, the solve rate at maximum dropped from 77.3% to 59.1%. Part of the “gain from effort” turned out to be benchmark leakage. They then tested a “smart router” (the model picks its own level) and the ladder. The ladder matched the fixed maximum on every model and task. For example, Kimi K3’s success rate rose from 73.0% to 84.3% while cost dropped 30.5%.

These are my variations, not from the paper. The paper only demonstrated the method described above.

🔧 Technique: drop the “none = cheapest” assumption → look at the full bill

Before switching an agent to no-reasoning mode to save money, compare the full session cost on 5–10 typical tasks, not just the reasoning tokens. If the number of calls has multiplied, the mode is more expensive.

🔧 Technique: ban searching for a ready answer → fewer “cheating” shortcuts

The git-history finding suggests a rule for your agent instructions file (CLAUDE.md, AGENTS.md). The authors didn’t measure the effect of such a ban, so test it on your own tasks:

Don’t look for a ready-made solution in the git history, in other branches, or online.
Derive the solution from the code and the task description. If you can’t do without it, ask me first.

Extrapolation: the ladder plus self-checking, but not instead of an external check

After finishing the task, list the acceptance criteria and, for each one, write
exactly which check (command, test, number) proves it’s met.
Don’t write “checked”: show the check’s output.

This isn’t the ladder; it’s a way to turn the model’s report into something you can quickly verify. The switch is still your external check.

• “What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents”: Kaiser Sun (Johns Hopkins University), Yongchan Kwon, Zain Hasan, Shang Zhu, Federico Bianchi (Together AI), Kaitlyn Zhou (Cornell University).
• Benchmarks used: AIME 2026, GPQA Diamond, Terminal-Bench 2.1 (Terminus-2), SWE-Bench Pro (mini-swe-agent), DeepSWE (additional noise check).
• The Act·ONOMY taxonomy of agent behavior (Gao et al., 2026).
• The idea of a “credence good”: Darby & Karni, 1973; Dulleck & Kerschbamer, 2006.