A Failure Map for AI Agents: Seven Ways AI Goes Off the Rails, and What to Do About It

S3 — Violating your constraints (38%)
     → Ignores explicit rules: "don't expand", "only this"

S2 — Misreading the request (27%)
     → Picks a plausible but wrong interpretation

S7 — False progress report (23%)
     → Reports "done", "verified", "working" — without actually checking

S5 — Faulty implementation (18%)
     → Understood the task correctly, executed it wrong

S1 — Misdiagnosing the situation (12%)
     → Blames the wrong cause, solves the wrong problem

S4 — Unauthorized scope creep (10%)
     → Went beyond the request without permission

S6 — Operational error (3%)
     → Wrong path, command, environment — environment errors

All episodes are the ones visible through user pushback. Hidden failures weren't included in the study.

Task: You ask Claude to help with an SEO audit of a Shopify app's landing page. The job: find only technical errors in the meta tags, don't touch the copy.

Prompt that ignores failure patterns:

Analyze this landing page and find SEO errors.
[pasting the HTML]

The same prompt, with prevention for the top 3 failures:

Task: technical SEO audit of the meta tags in this HTML.

Constraints (important — do not violate):
- Meta tags only: title, description, og:*, canonical
- Do not suggest changes to the page copy
- Do not expand the analysis to load speed, link structure or anything else

Response format:
1. A list of specific errors with the line number
2. What exactly is wrong (one sentence each)
3. How to fix it (a specific tag, not general advice)

Important: do not write "analysis complete" or "everything checked" — 
just give the list. If you found 0 errors, say so explicitly.

[HTML]

Result: You'll get a concrete list of errors with no vague recommendations. The "meta tags only" and "don't expand" constraints lower the odds of S3 (constraint violation). Banning "analysis complete" prevents S7 (false report). Asking it to say "found 0 errors" explicitly prevents silent omissions.

The LLM's weakness: the model doesn't "read" your constraints as hard rules. It generates the next token by probability. If training pushed it to be "helpful" and "thorough", that pressure outweighs your "don't expand". Hence S3: the agent expands a SQL migration, adds "helpful" infrastructure, fixes things nobody asked about. Not because it's dumb, but because that's what the distribution dictates.

Why false reports are so common (23%): the model doesn't check reality, it generates plausible text. "Tests passed" gets written because that's a typical ending to a task in the training data, not because the tests actually ran. It's not a lie in the human sense, it's next-word prediction.

How to use this knowledge: Three levers from the study:

• Explicit constraints, always at the top of the prompt → reduces S3. Repeating the constraint at the end ("reminder: meta tags only") works even better.
• Ban status phrases ("don't write 'done'") → reduces S7. Ask for an artifact, not a confirmation.
• Rephrase the request as a question ("which exact lines need to change?") → reduces S2 and S4, because the agent is forced to give specifics rather than an interpretation.

The "Top-4 failure prevention" template. Build it into the start of any task:

Task: {the specific task in one sentence}

Constraints — strictly:
- {constraint 1: what NOT to do, or the task's boundaries}
- {constraint 2}
- If you need to do something outside these limits — ask, don't do it

Response format: {a specific format — list, table, code, paragraph}

Verification: before you answer, check — 
did you stay within {main constraint}? 
Don't write "done" / "completed" — give the result right away.

---
{data / text / code}

What to fill in: - {task} — one action: "check", "rewrite", "find", not "analyze and improve everything" - {constraints} — anything the agent might do "by default, with the best intentions" that you don't need - {format} — the more specific, the less S4 and S2 - The "Verification" block — prevents S3 and S7 at the same time

🚀 Quick start — paste into the chat:

Here's a template that protects against typical AI failures. 
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about constraints and the expected format, because the template doesn't work without them, and the model will pick that up from the structure.

The study found something counterintuitive: most failures aren't technical AI errors. Only S5 (18%) is "did it wrong". The other 82% are communication failures: the agent either didn't get clear constraints, lost context, or acted on defaults.

That means prompt quality directly affects 82% of typical failures. You can't fix S5 (wrong code) through the prompt; that's a model problem. But S2, S3, S4 and S7 can be fixed by how you structure the request.

The second important takeaway: context gets lost (C4, 4.3%). The agent forgets what you said three turns ago, reverts to the old style, deletes an artifact you asked it to keep. If the session is long, repeat key constraints explicitly; don't count on "it already knows".

⚠️ This study is about coding agents (Cursor, Claude Code, Copilot): Failure patterns may differ somewhat for general chat tasks (writing, analysis, research), although the mechanics are the same.

⚠️ Hidden failures weren't counted: Only those where the user explicitly pushed back. The real failure rate is higher; we only see what surfaced in the conversation.

⚠️ Users share the blame: C1 (vague instructions) causes 15% of failures. Knowing the patterns helps both sides: the model and the user.

⚠️ Self-correction barely works: Don't count on the model "coming to its senses". If you see S7 (a false "done"), you need explicit pushback with specifics, not "are you sure?".

The researchers took 20,574 real sessions from public GitHub repositories: not synthetic data, not benchmarks, but how people actually work with Cursor, GitHub Copilot and Claude Code. The key decision: they looked for failures not inside the agent but through the user's reaction. If a developer explicitly objected or corrected something, something went wrong. That's a fundamentally different angle: earlier studies looked "from inside the agent", not at what the human experiences.

An LLM pipeline (GPT-5.4) extracted failure episodes from the sessions, then a second pass filtered out anything not backed by direct quotes from the dialogue. Of 29,896 extracted episodes, 16,118 remained, the right call in favor of precision over recall. Two experts manually checked 200 random records: the pipeline's precision was 0.93.

The most unexpected result: constraint violations outnumber implementation errors two to one (38% vs 18%). So the main problem isn't "AI can't code" but "AI ignores your explicit rules". And this is growing over time, even as the overall failure rate falls.

Adaptation for long sessions: preventing C4 (context loss)

When a conversation runs past 10 messages, it helps to drop in an "anchor":

💡 Adaptation for long sessions:

[Mid-session reminder]

Summary of our constraints for this conversation:
- {constraint 1 from the start}
- {constraint 2}
- {what we decided NOT to do}

Let's continue. Next task: {task}

Paste this every 8-10 turns. The agent doesn't "forget" on purpose; the context window shifts.

Technique: verify through an artifact, not a status

🔧 Instead of "check whether everything works" → "show me the concrete result of the check"

Before:

Make sure everything works and let me know.

After:

Run {a specific check} and show me 
the output in the format "Command: X → Result: Y". 
Don't write "works" without this block.

The micro-mechanics: S7 happens when the model can write a plausible status without actually checking. When you demand a concrete artifact ("command output", "a log line", "a screenshot of the error"), the model has no way to produce it without taking a real action.

Extrapolation: a diagnostic checklist before an important prompt

Before a complex request, run it through five questions:

Before you answer, analyze my request:

1. Is there an ambiguity in it that you're about to 
   resolve on your own without asking? → Ask.

2. Does it contain "this" or "here" without a clear 
   reference? → Clarify.

3. Are you about to do something beyond the direct 
   request "to make it better"? → Don't.

4. Are you going to use phrases like "done", "verified", 
   "works"? → Replace them with a concrete result.

Now do the task: {task}

This is a "built-in check": the model walks through the top 4 causes of failure before answering. It works as prevention for S2, S3, S4 and S7 at once.

How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions

Authors: Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, Toby Jia-Jun Li

Affiliations: University of Notre Dame, Vanderbilt University, Google

Tools from the study: SpecStory, Entire.io

Related work: Tang et al. (2026) — analysis of IDE sessions; Baumann et al. (2026) — CLI sessions; Cemri et al. (2026) — the MAST taxonomy