AM-Bench: Which Gaps in an Agent's Task Do the Most Damage

The paper doesn't propose a prompting technique. It's a diagnostic, and it converts neatly into a checklist:

FIELD 1: GOAL        → exactly what result you need (format, done criterion)
                       Gap → agent leaves it unfinished
FIELD 2: CONTEXT     → where to get the data, which source is primary
                       Gap → agent goes hunting on its own, burns tokens
FIELD 3: CONSTRAINTS → what must not be touched / read / changed
                       Gap → agent finishes and breaks the rules
RULE: if an action depends on an unclear field → stop and ask

All of this fits in a single message or in the agent's instructions file.

Task: You run a small store selling on Amazon and Etsy. You ask Claude Code to build a script that pulls the week's sales and calculates margin per SKU. The project folder holds a .env file with the live API keys for your seller accounts. Right next to it sit an old export from last quarter and the live orders database.

Bad spec: “Make a sales report for the week.” That's a gap in all three fields at once. The desired result is unclear, and the source isn't named. Worst of all, the agent has no idea that .env and the live database are off-limits.

Prompt:

GOAL
Build a sales report for March 10–16: a CSV table with the columns
“SKU, revenue, marketplace fee, cost of goods, margin %”.
Done when report_week.csv is in the /out folder and the revenue total
matches the sum of the rows.

CONTEXT
- Sales source: only the files in /data/export_amazon and /data/export_etsy.
- Cost of goods: the table /data/cost.xlsx (it's the primary source; do not use any others).
- Do not use the old export /data/archive_q4, it's outdated.

CONSTRAINTS
- Do not read or open the .env file.
- Do not connect to the live orders database or to the seller account APIs.
- Do not change or delete anything outside the /out folder.

STOP RULE
If the next step requires an action not covered by these three blocks
(a new data source, access to keys, writing outside /out), stop
and ask me one specific question. Don't guess.

Result: The agent works inside the boundaries you set. It reads only the specified folders, stays away from .env and the database, and drops the output in /out. If it needs an exchange rate or account data the spec doesn't cover, it stops and asks instead of digging into live sources. You check two things: the file is ready and the totals match, and no rules were broken (from the agent's action log).

The weakness. The agent doesn't know what you didn't say. If a prohibition isn't stated, it doesn't exist for the model. It doesn't “forget”; it simply has no signal. That's why, with a constraint missing, the agent calmly finishes the job and breaks the rule. It looks like a success, but the result is unusable.

The strength. An LLM agent is good at following what's stated directly and separately: goals, sources, prohibitions. Explicit blocks are easy to check, both for the agent and for you. The stop rule turns “guessing” into a question.

Control levers: - The CONSTRAINTS block is the most important one. If you're short on time, fill in at least this. Be specific: names of files, folders and systems, not “be careful.” - The CONTEXT block saves tokens. Name the primary source and list what to ignore. - The GOAL block with a done criterion protects you from half-finished work. - The stop rule: narrow or widen the list of triggers. For risky systems (money, production, customer data), keep just “ask before any write.”

One more caveat from the paper: the “best model” depends on the model × agent harness combo and on what you measure. That could be completion, correctness or consistency. Test your own combo on your own tasks, especially with incomplete specs.

The paper has no ready-made prompt. This template is built from its three fields (goal, context, constraints) and its recommendation to ask before dependent actions.

GOAL
{what_you_need}. Format: {format}. Done when: {done_criterion}.

CONTEXT
- Primary source: {source}.
- You may also use: {allowed_extras}.
- Do not use: {outdated_or_irrelevant}.

CONSTRAINTS
- Do not read or open: {secrets_and_private_data}.
- Do not connect to: {production_systems}.
- Do not change or delete: {off_limits}.
- {other_rules}

STOP RULE
If the next action needs information or permission that isn't listed above,
stop and ask one specific question. Don't guess.

What to fill in: in {done_criterion}, put something verifiable: the file exists, the totals match, the test passes. In the “Do not read” field, list everything the agent shouldn't see: keys, customers' personal data, the live database. It's better to list all constraints up front than to tack them on as you go.

🚀 Quick start: paste this into the chat:

Here's a template for writing a task spec for an agent. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about data sources, the done criterion and, most importantly, what the agent must not touch. It'll borrow the structure from the template and assemble a ready-to-use spec. It helps to list your project's “danger zones” first: keys, production systems, other people's folders.

⚠️ They measured the cost of gaps, not the template itself: the authors compared complete specs with incomplete ones. They didn't test whether this exact “goal / context / constraints” format improves results. The template above is our inference from their finding.

⚠️ Hidden requirements are baked into the test: the compliance check verifies requirements that were stripped from the task in the incomplete versions. The steep drop with missing constraints partly follows from how the test is built. But that's exactly the situation where the agent doesn't know the rule and doesn't ask.

⚠️ Clarifications were estimated on paper, not in live dialogue: nobody actually answered the agent's questions. The authors plugged in results from full-spec runs and estimated the potential gain. A real person might answer better or worse.

⚠️ Narrow task type: the tasks are terminal and office work, on sandboxed tools with a hard time limit. The conclusions about tokens and unfinished work partly depend on that limit. Whether this carries over to creative tasks and tool-free chats hasn't been tested.

⚠️ The task variants were written by a model: an LLM drafted the incomplete versions, and the authors checked them by hand. Real-world “leaky” specs can look different.

⚠️ There's no universal “best combo”: the same harness swap improves one model's results and hurts another's. The paper doesn't hand you a ready-made ranking for picking a tool.

The researchers took 48 practical tasks from two existing benchmarks (Terminal-Bench 2.0 and Harness-Bench): folder permissions, audio transcription, log reports. For each one they wrote a complete spec plus six broken versions. In those, the goal, context or constraints were either vague or missing: 2 gap types × 3 areas.

Then they ran 5 models × 4 agent harnesses × 3 runs, 20,160 runs in total. They scored completion (a result was produced), constraint compliance (from the agent's action log) and success (both at once) separately. That's what exposed the cases where the agent “did great, but broke the rules.”

The results were surprising. Without constraints, the share of completed tasks sometimes went up: the agent cheerfully plowed through the work without a second thought for the rules. Success, meanwhile, dropped to zero. Missing goals and context led to unfinished work and token overspending. The token “savings” when information was missing turned out to be nothing more than unfinished work.

Here's how the authors estimated the payoff from clarifications: in runs where the agent asked a question, they swapped in the average result of the full spec for the same combo. It turned out that clarifications recover unfinished work, but do almost nothing for violated constraints. How often agents ask for clarification, and how much it helps, depends more on the agent harness than on the model.

💡 Adaptation for an agent instructions file (CLAUDE.md / AGENTS.md): The paper directly recommends that harness developers build in “targeted clarification before dependent actions.” A practitioner can bake this into the instructions on their own:

Copy

STOP RULE (always)
Before any action that writes, deletes, sends, or reaches out
to an external system, check: have I been explicitly told this is allowed?
If not, stop and ask. Reading files outside the specified folders
and using secrets: only after explicit permission.

🔧 Technique: audit the spec before launch → catch the gaps before the agent gets to work. This isn't in the paper; it's our extrapolation from the three fields:

Copy

Here's my task spec for an agent: {text}.
Don't execute it. Check it against three fields: goal (is there a done criterion),
context (are the sources named), constraints (what must not be touched).
List what's missing and ask me up to 5 questions,
starting with the constraints.

• AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses (Preprint)
• Project repository: https://github.com/s4-lab-cuhksz/AM-Bench.git
• Authors: Kaiyu Huang, Yiting Li, Angyang Gu, Pinjia He, Minchen Yu, Qingjiang Shi (Tongji University; The Chinese University of Hong Kong, Shenzhen)
• Source benchmarks: Terminal-Bench 2.0, Harness-Bench; related work: ToolEmu, LHAW, UnderSpecBench, ClarifyBench, AppWorld-UL