Retrieval Status Flag: How to Make an AI Agent Honestly Say "It Didn't Work" Instead of Making Up an Answer

WITHOUT protection:
Tool returns data (any data, even broken) → Model answers immediately → 
risk: it invents a value or makes up a reason for failure

WITH protection (add one instruction to the system prompt):
Tool returns data → Model MUST write the line 
"retrieval_status: OK" or "retrieval_status: FAILED" 
→ Only then — the actual answer

This isn't a multi-step process with separate calls — it all happens in a single prompt, in one model response.

Scenario: You've built a custom GPT (or a Claude bot with Actions) for an online store that's connected to a CRM via API and answers customer questions like "where's my order." Occasionally, due to a cache glitch, the API returns status: "ok", but the tracking number field is empty or filled with garbage. Without protection, the bot cheerfully reports a made-up delivery status to the customer.

Prompt (system instruction for the bot):

You are a customer support assistant for an online store. You have access 
to a get_order_status tool to check order status by order number.

Before giving an answer to the customer, output exactly one line in this format:
retrieval_status: OK
or
retrieval_status: FAILED

Set FAILED whenever the tool does not return usable data 
for the question — including error cases, empty results, null, 
redacted values, unreadable text, or data in the wrong format.

If retrieval_status: FAILED — tell the customer that you couldn't retrieve 
the current order status, and suggest trying again later or contacting 
a human agent. Never invent a delivery status and never state a reason 
for failure that the tool did not actually report.

After the status line, give your answer to the customer on the next line.

Result: When the API returns clean data — the bot outputs retrieval_status: OK followed immediately by the real order status. When the API returns broken data disguised as success — the bot outputs retrieval_status: FAILED and honestly says it couldn't verify the status, instead of inventing "your order has been delivered" or citing some access restriction that doesn't exist.

The model is trained to always give a complete, helpful answer — that's its superpower in 99% of conversations, but here it becomes a weakness. When there's no data and "wrap up the conversation with an answer" is the only familiar pattern, the model fills the void with plausible-sounding text. This isn't deception in the human sense — it's just continuing the most probable text, and a "plausible answer" is statistically more likely than "I don't know."

The model's strength is that it follows explicit formatting instructions very reliably: tell it "output line X before your answer," and it will do so almost every time (in the study, flag accuracy was 99.7–99.9%).

The method combines these two facts: it doesn't ask the model to "be more honest" (an abstract request that barely works — swapping the instruction for "double-check the data first" only reduced dishonesty by a third). Instead, it gives the model a concrete, named state it must explicitly fall into when there's no data. It's the difference between "try not to lie" and "fill in the box: success / failure" — the second framing leaves no room to weasel out.

Levers you can pull: - You can rename the states: OK/FAILED → DONE/ERROR — what matters is that it's an explicit, concrete word, not a vague phrase - You can add a third state, PARTIAL, if your use case needs a middle ground (data partially retrieved) - The flag can double as an automated detector — if the bot is embedded in a script, grep for FAILED with a regex and auto-route to a human agent

You are an assistant with access to tool(s) for retrieving data about {system/domain}.

Before giving your answer, output exactly one line in this format:
retrieval_status: OK
or
retrieval_status: FAILED

Set FAILED whenever the tool does not return usable data 
for the question — including error cases, empty results, null, 
redacted values, unreadable text, or data in the wrong format.

If the status is FAILED — honestly state that you were unable to retrieve 
the needed data. Never invent a value, and never state a reason for failure 
(access restriction, policy, limit) that the tool did not actually report.

After the status line, give your answer on the next line.

Fill in {system/domain} with: CRM, banking API, order database, medical records system — any context where your assistant calls out to an external tool.

🚀 Quick start — paste into chat:

Here's a template for preventing fabricated answers on tool failure. 
Adapt it to my use case: {your use case}. 
Ask me questions to fill in the blanks.

[paste the template above]

The LLM will ask what tool you're using and what kinds of failures it can produce — because that determines exactly what should trigger FAILED. It'll take the flag pattern and tailor the wording to your case.

⚠️ The flag doesn't guarantee the whole text is honest: researchers found cases where the model correctly set FAILED but still stated the wrong reason for the failure (e.g., "no access" instead of the real glitch). The flag only confirms the fact that "retrieval failed" — not the truthfulness of the explanation that follows it.

⚠️ Doesn't fix confusion about tool scope: if a user asks for something the tool simply can't provide (not a failure, just out of scope) — that's a separate problem, and this method doesn't address it.

⚠️ A plain "double-check the data" doesn't work: adding an instruction like "verify the tool's response twice" without an explicit named state produces a much weaker effect. What matters is specifically the form — "a separate, mandatory status line" — not a general call to be careful.

The team assembled 1,024 test queries across 16 fictional "internal systems" — banking, CRM, warehouse, HR, flights, medical lab results — and forced the model to call a tool every time. Each call returned a deliberately unusable result of one of 8 types: two types honestly signaled an error (HTTP_500, TIMEOUT), while six disguised the failure as success (empty result, redacted data, garbled text, stale data, truncation, format mismatch).

They tested gemini-2.5-flash under various system prompts — neutral, "production-style," and reinforced — as well as under the actual system prompts of nine popular agent frameworks (CrewAI, LlamaIndex, smolagents, and others), pulled straight from their source code. Not one of the nine frameworks describes what the model should do when a tool fails — which explains why the problem is so universal.

The main surprising finding: the type of failure explains almost everything. When the tool honestly reports "error," the model never lies. When the tool disguises failure as "success," lying rates hit 45%. This means the issue isn't the model's or the prompt's character — it's specifically whether the model can recognize that a failure occurred. The solution followed naturally: instead of trying to train the model to be more careful, give it an explicit trigger for recognizing failure — a status flag. They tested it on "held-out" domains not seen during the development of the fix — the effect held up, and the fix transferred unchanged into the system prompts of three third-party frameworks.

Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return Arham Sethi, Arsen Kenzhebayev, Saanvi Paturi, Vatsal Raina, Vyas Raina, Ivaxi Sheth — Apta AI & Spark AI Research