AIDA: an independent adversary and judge instead of self-checking — how to stop an agent from closing a case on an empty result

STEP 1 (Strategist): determines what evidence is needed → list of queries
STEP 2 (Executor): runs the searches, handles failures → results into the ledger
         (an empty result is recorded as "empty", not as "clean")
STEP 3 (Strategist): locks in a thesis — "escalate / close" + the evidence it rests on
STEP 4 (Adversary, SEPARATE context): objection if the evidence contradicts
         the thesis or required data is missing → objection into the ledger
STEP 5 (Judge): weighs thesis + objection against the evidence
         → decision OR a new round specifying what to collect
         (the evidence bar is higher for closing)
Repeat 2–5 with the same ledger until the Judge reaches a decision

Steps 1–5 are separate model calls, not a single prompt. The Adversary and the Judge each get their own context.

Task: You own an online store built on Shopify. Your monitoring system sends alerts: "admin login from Brazil at 3:12 a.m." Almost all alerts are harmless: a contractor, a VPN or a business trip. But one of them could be a breach, and nobody will ever reopen a closed alert. You need an agent (Claude Code or a custom GPT with log access) that checks the alert on its own and decides whether to close it.

Prompt:

<Role>
You orchestrate the investigation of security alerts for an online store.
Nobody will ever re-check a closed alert, so missing an attack is the worst possible error.
Work through four roles and an investigation ledger.
</Role>

<Ledger>
The investigation ledger is append-only; entries are never edited or deleted.
Entry format: [round] [role] [type: query / result / thesis / objection / decision] [content]
</Ledger>

<Protocol>
Until the Judge reaches a decision:

1. Strategist: determine what evidence is needed to tell whether the login is legitimate.
   For example: this user's other logins, password changes, admin actions after the login,
   logins from this IP on other accounts, the employee's location history.
   Output a list of queries for the Executor.

2. Executor: run the queries through the tools. Mark query failures as "ERROR".
   If a query found nothing, write "EMPTY: query X for period Y returned nothing".
   An empty result is not proof of safety.

3. Strategist: once there's enough evidence or the round's query limit ({query_limit}) is used up,
   write a THESIS: "escalate" or "close" + which ledger entries support it.

4. Adversary: work in a separate context; you see only the ledger and the thesis,
   not the Strategist's reasoning. Write an OBJECTION if:
   - the evidence contradicts the thesis, or
   - data is missing without which the alert can't be resolved.
   If you have no objections, write "no objections" and explain exactly what you checked.

5. Judge: weigh the thesis and the objection against the evidence in the ledger.
   - For "escalate", plausible evidence is enough.
   - For "close", positive proof of legitimacy is required
     (for example, the login is confirmed by a known VPN or the work schedule).
     The absence of suspicious findings is not proof.
   - If important data is missing, request a new round and list exactly what to collect.
   - Otherwise, issue a decision with a rationale.
</Protocol>

<Alert>
Login to the Shopify admin panel, user {login}, IP {IP}, Brazil, 3:12 a.m. Eastern.
</Alert>

Result: The agent will run one or more rounds. The ledger will show the queries and results, with empty searches clearly flagged. Then the thesis, the Adversary's objection and the Judge's decision citing the evidence. If there's no confirmation of legitimacy, the alert gets escalated or goes back for another round with a specific list of missing data.

The weakness. The model can't tell "didn't find it" from "it isn't there". It reads an empty tool response as confirmation of whatever version it was already leaning toward. When the model checks itself in the same context, it conforms to its own earlier text. That's why, in the study, self-checking broke several times more correct decisions than it fixed wrong ones.

The strength. Give the model someone else's concrete thesis and ask it to poke holes in it, and it critiques well. It also works well as a judge when it has to weigh two claims against recorded evidence. Criticism is sharper when the model doesn't carry the baggage of its own reasoning in context.

How the method uses this. The thesis is locked in before criticism, which gives the Adversary a target. A separate context removes the conformity. The asymmetric bar (closing requires positive proof) corrects the bias of "closing is cheaper than escalating". An append-only ledger keeps round 3 from quietly rewriting the conclusions of round 1.

Control knobs: - {query_limit} per round → raise it for complex cases, lower it to save tokens. - Bar for closing → tighten or loosen the evidence requirements to match the cost of an error. - Number of rounds → set a ceiling after which the case goes to a human. - The Judge role → without it, F1 in the ablation fell from 0.952 to 0.895.

<Role>
You orchestrate the process of "{what_we_decide}". The decision "{irreversible_decision}" is irreversible
and nobody re-checks it later, so missing {critical_error} is the worst possible error.
Work through four roles and a ledger.
</Role>

<Ledger>
The ledger is append-only. Entry format:
[round] [role] [type: query / result / thesis / objection / decision] [content]
</Ledger>

<Protocol>
Until the Judge reaches a decision:

1. Strategist: determine what evidence is needed to resolve {case}.
   Output a list of queries for the Executor.

2. Executor: run the queries through {tools_or_sources}.
   Failure → "ERROR". Nothing found → "EMPTY: what was searched, where, for what period".
   An empty result is not proof.

3. Strategist: once there's enough evidence or the limit ({query_limit}) is used up,
   write a THESIS: "{option_A}" or "{option_B}" + the ledger entries it rests on.

4. Adversary: work in a separate context; you see only the ledger and the thesis.
   Write an OBJECTION if the evidence contradicts the thesis or data needed for a decision is missing.
   If you have no objections, state what you checked.

5. Judge: weigh the thesis and the objection against the ledger.
   - For "{safe_option_A}", {soft_bar} is enough.
   - For "{irreversible_option_B}", positive proof of {what_kind} is required.
     The absence of anything suspicious is not proof.
   - If important data is missing, request a new round and list what to collect.
   - Otherwise, issue a decision with a rationale.
</Protocol>

<Input>
{case}
</Input>

What to fill in: {irreversible_decision}: the thing that can't be undone (closing a ticket, approving a payment, letting an order through). {critical_error}: the most expensive error in your field. {soft_bar} and {what_kind}: the difference between "suspicion is enough" and "confirmation is required".

🚀 Quick start: paste into the chat:

Here is the AIDA template (four roles + an investigation ledger). Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which of your decisions is irreversible, which error costs more and what evidence can be gathered. The Judge's asymmetric bar depends on exactly that, and it's the central part of the method. It will take the structure from the template and tailor it to your field.

ℹ️ What in the template comes from the paper and what is reconstructed. The available part of the paper doesn't include the agents' prompt texts. The roles, the order of steps, the ledger, the thesis-before-objection rule, the Adversary's separate context and the higher bar for closing are all described by the authors. The wording in the template was assembled from that description.

⚠️ About chat. In a single chat window, the Adversary has no separate context. To preserve the essence of the method, run the Adversary and the Judge in a new chat (or as subagents in Claude Code), passing them only the ledger.

⚠️ The price is more escalations: AIDA sends noticeably more cases to analysts (18.4% vs. 8.5–14%). That's the cost of missing almost no attacks. If human attention is scarce for you, this is the bottleneck.

⚠️ Tested on one task: this is a single attack scenario (out of eight possible) on lab data, and almost all the attacks are web reconnaissance. Rare phases (privilege escalation, service scanning) were represented by a handful of cases.

⚠️ Tools and evidence are required: the method works when the agent can gather data. If the decision is made from text alone with no sources, the Adversary and the Judge have nothing to stand on.

⚠️ Not for simple decisions: for reversible, cheap decisions, four roles and several rounds are overkill.

⚠️ The authors' implementation is code: in the paper it's an orchestrator with an MCP interface to a SIEM. Reproducing it with chats and agents takes manual assembly: separate contexts, passing the ledger along.

The researchers built ALERT-BENCH. They took recorded telemetry from a corporate network with a labeled multi-day attack and "replayed" it through a live Wazuh SIEM system. The agent gets one alert and decides on its own what to look for around it. Choosing the right evidence is up to the agent, just like a human analyst. The correct answers are hidden until scoring. The sample included 1,247 alerts: 225 attacks and 1,022 benign ones.

Five popular approaches (direct prompt, ReAct, Self-Consistency, Self-Refine, Chain-of-Verification) were run on the same model (GPT-4.1) with the same tools. Each one missed at least 40.4% of attacks. An analysis of the recorded steps revealed three recurring behaviors: empty search → close, "revising" instead of fixing (Self-Refine fixed 16 errors and broke 79 correct decisions; CoVe, 25 and 132), and the same depth of checking for closing as for escalating.

AIDA was built as a direct answer to these three holes. The result: F1 of 0.958 vs. 0.371–0.744, and the share of missed attacks fell from 40.4% to 3.1%. The Judge corrected 31 wrong theses without reversing a single correct one. Without it, F1 fell from 0.952 to 0.895. The 9-billion-parameter Qwen3.5 model with AIDA reached an F1 of 0.952: structure gives more than model size.

The unexpected part: self-checking, which is commonly recommended, actually did harm here. The practical takeaway: check with a different context and against new evidence, rather than asking the same model to "reread" its work.

🔧 Technique: an empty-results rule → drop it into any agent prompt

The cheapest part of the method, which you can take on its own if you don't have time for the full scheme:

Copy

An empty search result is not proof of absence. Record it as
"EMPTY: what was searched, where, for what period". A case may only be closed
on positive proof of legitimacy, not because nothing suspicious was found.

The paper tested the full scheme as a whole, not this snippet on its own. But empty results were the main cause of misses, so it's a sensible place to start.

🔧 Technique: a faceless Adversary → a concrete persona

Instead of Adversary, specify Adversary: a paranoid head of security who will be fired for a missed breach. The criticism will get sharper, but the number of unnecessary escalations may rise. This is my own idea; it wasn't tested in the paper.

Extrapolation: the same framework for support tickets or order review. Plug in {irreversible_decision} = "close the customer's ticket as resolved", {critical_error} = "the customer was left without help". The bar for closing: "the customer confirmed the problem is gone". Not tested in the paper, but the logic (the decision is irreversible and rarely re-checked) is the same.

• Paper: From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage
• Authors: Danfeng (Daphne) Yao (North Carolina State University), Peng Gao, Saimon Amanuel Tsegai (Virginia Tech), Alex Kantchelian (Google)
• Benchmark: ALERT-BENCH on AIT-LDSv2 data with the AIT-ADS labeling procedure, Wazuh SIEM, MCP interface
• Approaches compared: Direct-Prompt, ReAct, Self-Consistency, Self-Refine, Chain-of-Verification