TL;DR
CtxGap is a study that compares two things: what a coding agent actually did and saw (the full session log) and what it told you in its reports. Both "readers" got the same questions about how the work went: which attempts failed, which workarounds were used, which risks remain. Read only the reports, and you answer roughly half the questions correctly. Read the full context, and you get almost all of them.
The headline finding: the agent knows more than it says. Ask it directly and it answers nearly everything. But in a typical "done, tests are green" report, some of the failures, workarounds and risks just never make it in. And you can't ask about what you don't know exists. The nastiest part: most often the event is mentioned in the report, but without details. "Added a test" - but which one, what it checks and how it turned out goes unsaid. Simply adding a mention doesn't help. What helps is concrete supporting details: file names, commands, the outcome, the reason.
The practical takeaway: don't mistake "done" for the full picture. Ask the agent for a report broken into categories (what broke, what was worked around, what risks remain), with evidence. Before you sign off, ask it direct questions, because it answers those well. A report rewritten with the full context in view becomes noticeably more informative, even at the same length.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
The paper doesn't offer a single "method". It's a measurement plus a set of tested interventions. Together they add up to a practical procedure:
STEP 1: Don't trust "done" → treat the report as incomplete by default
STEP 2: Ask for a report by category (failures, workarounds, risks, decisions, environment)
→ every item backed by evidence: file / command / outcome
STEP 3: Before sign-off - ask the agent direct questions about the work
(it answers more accurately than it writes in reports)
Do steps 2 and 3 in the same chat or session, while the agent still "remembers" the work.
Example in Practice
Task: You run a small online store built on Django and asked Claude Code to hook up Stripe payments. The agent reports back: "Integration is done, tests pass." You're about to ship to production and want to know what got swept under the rug.
Prompt:
Before I accept this work, give me a detailed report on the session.
Break it into sections:
1. FAILURES: what you tried that didn't work (command/file, what happened, why you dropped the approach)
2. WORKAROUNDS: where you deviated from the task or took a shortcut (what exactly, why, what the risk is)
3. UNRESOLVED RISKS: what could break in production (the condition that breaks it, what you saw in the logs)
4. CHECKS: which tests and scripts you ran or wrote (file name, what it checks, result)
5. ENVIRONMENT: what you changed in configs, environment variables, dependencies
Rules:
- Don't write "added a test" or "accounted for the risk". Write which test exactly, what it checks, how it turned out.
- If nothing happened in a section, write "none". Don't skip the section.
- Back every item with specifics: file name, command, error text.
Result: The model will produce a five-section report with specific files, commands and the conditions under which something could break. Chances are, things will surface that weren't in the short "done": abandoned approaches, temporary stubs, uncovered cases. If a section is empty, the model will say so, and you'll see it explicitly.
Why It Works
The weakness. The agent writes reports "for a human": short, focused on the outcome and the highlights. Failures, workarounds and doubts don't fit that format. Some events get a single sentence with no details. The reader can't reconstruct what actually happened.
The strength. The session context is right there in front of the agent, and it answers direct questions almost flawlessly. So the information exists; what's missing is the format and the requests that pull it out.
How the gap gets closed. A category-based report structure forces the agent to look where it doesn't look by default. Requiring "evidence for every item" stops it from getting away with a passing mention. In the study, simply adding a mention produced no gain at all. Only concrete details moved the needle.
Control knobs: - Categories - drop the ones you don't need and add your own (security, performance, compatibility). In the paper, the biggest share of lost information came from facts, failures and progress, followed by environment state. - The evidence requirement - the main lever. Without it, the agent slides right back into vague generalities. - Length - at the same total length, a report rewritten with the full context in view is more informative. It's about content, not volume. - "If nothing happened, write "none"" - separates "nothing happened" from "forgot to mention it".
Prompt Template
Two versions: a one-off request at the end of a session, and a standing rule in the agent's instructions file (CLAUDE.md, AGENTS.md).
Version 1. Requesting a report before sign-off:
Give me a detailed report on the session, by section:
1. FAILURES: {what you tried that didn't work}
2. WORKAROUNDS: {where you deviated from the task or took a shortcut}
3. UNRESOLVED RISKS: {what could break and under what conditions}
4. CHECKS: {which tests and scripts you ran; what each one checks; result}
5. {your own section}: {for example, changes to configs and dependencies}
Rules:
- Don't write in generalities ("added a test", "accounted for the risk"). Be specific: file name, command, error text, result.
- Don't skip an empty section: write "none".
- If you're not sure about a fact, mark it "not sure" and explain why.
Version 2. A standing rule for the agent's instructions file:
## Reporting rules
When you report that a task or stage is complete:
- Don't stop at "done". Include these sections: Failures, Workarounds, Unresolved risks, Checks, Environment changes.
- Describe every event with supporting evidence: file, command, result.
- A mention without details counts as an incomplete report.
- If nothing happened in a section, write "none".
- Don't report success while unresolved risks remain. List them.
What to fill in: you can keep the section names and descriptions as they are or tailor them to your task. For software work, tests, configs and dependencies fit well. For analytics, use data sources and assumptions.
⚠️ This is assembled from the paper's findings. The exact prompt wording the authors used to test the "structured" and "targeted" additions isn't given in the text.
Limitations
⚠️ The headline number doesn't cover every kind of task: It was measured on SWE-bench tasks (bug fixes in open-source repositories, mostly Django-like projects). How it plays out on other types of work wasn't tested directly.
⚠️ Report details aren't a cure-all: Rewritten reports improve information recovery, but a gap remains. Even the full terminal output doesn't bring the reader up to the level of the full context.
⚠️ Details have a cost: The most informative "targeted" additions were noticeably longer. Such a report takes longer to read, and that's a trade-off in its own right.
⚠️ A special format gave no edge: At the same total length, the "targeted" report (built around the needed categories) showed no significant advantage over a plain "natural" rewrite. What matters is the level of detail itself and grounding in the full context, not a clever format.
⚠️ Not every missing fact matters: The questions were generated by a model, not by a human supervisor. Models were also used to judge "importance". The gap held up on questions selected as important, but the measurement is still model-based.
⚠️ A detailed report isn't the same as the truth: The study measures how much information can be recovered from a report. It doesn't check whether the agent is telling the truth. A detailed report still needs to be double-checked.
How They Studied It
The idea is simple: if the agent knows everything but writes only part of it in the report, let's measure exactly which part gets lost. The researchers took 37 tasks from SWE-bench Verified, one session per task, and paused each session at several points. From the log they pulled out "atoms" (individual facts: a failure, progress, a risk, a decision). For each atom they wrote a question with a reference answer.
Then the same "observer" model was asked to answer those questions in different ways. Option one: only the reports and the user's instructions. Option two: the full terminal. Option three: the entire context, including reasoning. A separate judge did the scoring. In total, about 4,800 questions were collected.
The results: the full context recovers 92.7%, the reports only 52.0%. The full terminal gets 73.6%. The agent itself, asked directly, answers correctly 93.6% of the time. The gap persists even after successful fixes (where the tests passed): at the final checkpoints of 30 tasks it stayed at around 35 points.
Things got interesting with the manual annotation. Two people went through why the information wasn't recovered. It turned out that most often the fact was mentioned in the report, but not in enough detail (55–59%), and only about a third to two-fifths of cases were "not mentioned at all". They also tested the "just add a mention" hypothesis: zero gain. Adding supporting details, on the other hand, raised recovery by 19 points.
Reports rewritten with the full context in view, at the same total length, gave roughly +8 points. For facts that can't be inferred from the rest of the text, the gain is largest: around 0.02–0.03 for plain additions, 0.28 for structured ones, and 0.60 for "targeted" category-based ones (but those are longer).
The result was checked on three agent harnesses (Claude Code, Pi, DSH), on an independently run DeepSeek model, and on a completely separate measurement pipeline. The gap reproduced everywhere. The practical conclusion: the problem isn't one model or one tool.
Adaptations and Extrapolations
🔧 Technique: direct questions instead of reading the report → get what the agent keeps quiet about
Since the agent answers direct questions well, add a short interrogation before sign-off:
Answer these questions about this session honestly and specifically: 1. Which approaches did you try and abandon, and why? 2. Where did you deviate from my original task? 3. What in the solution rests on an assumption you didn't verify? 4. What will break first if the load or the data differ from what was in the tests? 5. Which files did you change that I didn't ask you to change?
Extrapolation (not from the paper): checking the report with a fresh reader
This mirrors the study's logic in a lightweight form. Open a new chat with no context and give it only the agent's report.
Here is the agent's report on its work: {report}. You know nothing about how the work went beyond this text. Ask five questions that the report can't answer, but whose answers are needed to decide whether to ship.Pass those questions back to the original agent. It's a test of how self-contained the report is. The paper doesn't include this procedure, so validate it on your own tasks.
Resources
- Paper: What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports (CtxGap)
- Authors: Guoyang Xu, Rui Zhang, Xiuyu Wu, Hao Chen — Tencent
- Code and benchmark: https://github.com/Kaguraaaaaa/Ctxgap
- Environments and data mentioned: SWE-bench Verified, Claude Code, Pi, DSH (DeepSeek)