TL;DR
A simple harness (the program around a model that feeds it data and executes its actions) with no scripted playbook turned out to be more accurate than elaborate agents. The model decides for itself which data to open and how to compare it. The authors tested three setups: a single call with a pre-assembled evidence pack; a loop of SQL queries over raw files; and the Codex coding agent. Codex gets a data folder, a task and just six guardrail rules, and the investigation steps are never spelled out.
The headline finding: when a program or an instruction decides up front what to read, it can miss the decisive clue. In the case they dissected, an agent with a ready-made procedure (SOP) picked a template that had no "read the logs" step. It blamed the wrong service instead of the one that actually failed. Another agent followed the call chain from the alert and missed a service that had stopped receiving calls. When calls simply vanish, there is nothing in the chain to read.
The method boils down to not scripting the steps. You give the model a task, access to the data and hard boundaries (what is off-limits, when to stop, what report to hand in). It picks the course of the investigation on its own. Bonus findings: a stronger model mostly buys you consistency, while a multi-step mode buys you capability to crack the problem at all. Cranking up reasoning effort adds a few points of accuracy at best, while taking up to seven times longer.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
OPTION A (single call, chat):
Assemble a numbered evidence pack → model returns up to 5 candidates + cites evidence numbers
OPTION B (tool-using agent, like Codex):
Give a data folder + task + 6 guardrail rules
→ the model writes its OWN commands/code, inspects, narrows down → final report
OPTION C (query loop):
Tools: list files, schema, SQL, "submit answer" (100 calls max)
What NOT to do: fixed stages, SOPs like "if failure type is X, read Y", expert roles with voting
Example in Practice
Task: On Tuesday evening a grocery delivery service saw orders drop by 30%. The export folder holds per-minute orders, application logs, deploy events and payment gateway responses. You need to find the cause without burning a whole day on it. You launch Claude Code or Codex in that folder.
Prompt:
Task: on Tuesday from 19:10 to 21:00, orders dropped by roughly 30% compared
to the previous Tuesday. Find the root cause.
Data: folder ./incident_tue — orders_per_minute.csv, app_logs/, deploy_events.json,
payment_gateway.csv. The "before" period is Monday 19:00–21:00, the "during" period is Tuesday 19:00–21:00.
Rules:
1. Read only the files in this folder.
2. No network access, don't install anything.
3. Don't modify the source data.
4. Spend no more than 30 minutes on this.
5. Always finish with a report, even if you're not confident.
How to work is up to you: which files to open and how to compare them.
Output format: up to 5 root-cause candidates, ranked by confidence.
For each: 1–2 sentences and a pointer to specific evidence (file, line or timestamp).
Result: The agent will come up with its own plan: list the files, write code comparing "before" vs "during" and narrow down the suspects. Then it will pull logs and events for the candidates it found. At the end you get a report: a list of 1–5 causes, each with a pointer to the evidence. There are no pre-scripted stages. The model itself drives the work.
Why It Works
The weakness of rigid procedures. Any playbook ("metrics first, then logs, then the experts vote") is written for the typical case. In an atypical one it keeps the model away from where the answer lives. The model didn't botch its reasoning: it simply never got shown the key clue. In the paper that's exactly what happened: the procedure that was needed had no log-reading step.
The model's strength. A modern model can write its own queries and code, quickly compare "before vs during" across every entity at once and narrow the field after a broad sweep. In the dissected case, Codex first compared everything against everything, spotted the right line in the output and reached the answer in seven commands.
How the method exploits this. You don't steer the course of the investigation; you set the frame: boundaries, answer format and a stopping condition. The levers: - Guardrail rules (folder, network, no data changes, time limit) → safety and predictability without giving up freedom. - Output format (up to 5 candidates + evidence pointer) → an answer you can verify at a glance. - A single call instead of an agent → faster and cheaper. But you still pick the evidence pack yourself, which is the same "what to show" trap. - Reasoning effort → don't chase the max: the gain is small, the time multiplies.
Prompt Template
Option A — a single call in a regular chat:
You are investigating an incident: {incident description and time}.
Below is an evidence pack. Each item is numbered. The "before" period is {period_before},
the "during" period is {period_during}.
<evidence>
{E1: description of the list of entities and their dependencies}
{E2: metrics that changed — raw "before" and "during" values}
{E3: per-minute request statistics}
{E4: log templates with "before" / "during" counts}
{E5: events (restarts, deploys, errors)}
</evidence>
Reason freely. Which evidence matters and how to compare it is up to you.
Answer: up to 5 root-cause candidates, ranked by confidence.
For each: a brief justification and the evidence numbers you rely on.
Option B — for an agent with folder access (Codex, Claude Code and the like):
Task: {incident description and time}. Find the root cause.
Data: folder {path}. "Before" period: {period_before}. "During" period: {period_during}.
Rules:
1. Read only the files in this folder.
2. No network access, don't install anything.
3. Don't modify the source data.
4. Always finish with a final report.
5. Spend no more than {limit} on this.
How to investigate is up to you.
Output: up to 5 root-cause candidates, ranked by confidence, each with a pointer to the evidence.
Fill in: {incident description} — what happened and when; {period_before} and {period_during} — a clean baseline period and the incident window; {path} — the folder with your exports; {limit} — how much time you're willing to spend. In the original, Codex had one more rule: "don't try to figure out which dataset the case comes from." That one only matters for the benchmark, so you can drop it in real work.
Limitations
⚠️ One domain: all conclusions come from root-cause analysis of microservice failures. Carrying them over to other investigations (sales, support, finance) makes sense, but the authors themselves did not test it.
⚠️ Freedom isn't free: a coding agent costs time and money, and each case can take minutes. The cheapest option (a single call) is noticeably weaker than the other simple ones.
⚠️ A single call depends on evidence selection: a program or a person decides what goes into the pack. It's the same trap as rigid SOPs, just milder.
⚠️ Hard cases: on cases with all four data types available, the gap between the simple methods and the published agents nearly vanished. One of the published agents caught up with the best simple one there.
⚠️ Weak model, shaky results: a strong model mostly buys consistency. If you're using a weaker model, run it several times and cross-check the answers.
⚠️ Fine-grained levels: every LLM method lost accuracy at finer cause levels (a specific pod or node rather than a service). That's where the rigid procedures of the published agents dropped the most.
⚠️ Some third-party agents were rebuilt from their paper descriptions rather than their original code. On top of that, the cases had the failure-type hint removed, which the original works do include. Part of the gap may come from that.
How They Studied It
The authors took 8 public datasets of microservice failures and unified them into a single format. Of 542 cases reviewed, 348 passed the filter: no contradictory labels, actual traces of the failure in the data and a clean "before" period. From these they assembled a set of 90 cases (RCAtlas). They compared three groups: 3 simple harnesses, 5 published LLM agents (with SOPs, roles, voting) and 7 classic non-LLM methods (statistics, causal graphs). All of it ran on 6 models and 3 reasoning effort levels with one shared accuracy metric (PR@1 — the share of cases where the very first cause is correct).
The results: simple harnesses on three mid-tier models scored 65–77%, published agents 16–53%, non-LLM methods 6–39%. Random guessing landed around 5%. Even the weakest simple method (a single call) beat the best published agent by 12 points. The surprise: plain NSigma statistics outperformed two of the published LLM agents.
To understand why, the authors walked through one case step by step: a service stopped receiving calls while its neighbor was throwing "Connection refused." The single call with an evidence pack immediately spotted the spike in those errors in the logs. The SQL agent and Codex found the right line on their own. The SOP agent picked a procedure with no log-reading step, and the call-chain-walking agent didn't notice that the culprit service had simply stopped receiving requests. The statistical methods missed that the metric had dropped: they only looked for increases. The practical takeaway: the problem is a program deciding in advance where to look. The reasoning logic of these agents isn't to blame.
The final test: on the accuracy-vs-cost/time curve, among all methods more accurate than the best non-LLM one, every optimal point belongs to a simple harness.
Adaptations and Extrapolations
🔧 Technique: add a "broad sweep → narrow down" step → fewer blind spots
In the walkthrough, Codex first compared all entities "before vs during" in one piece of code and only then narrowed the field. You can ask for this directly without spelling out the details:
First compare ALL entities and metrics "before" vs "during" in a single pass. Then narrow it down to 3–5 suspects and examine them in more detail.
🔧 Technique: look for disappearances too → catch causes that go "silent"
In the paper's case, a method that ranked only increases missed the drop in requests to the culprit service. If your data works the same way, add:
Look not only for what went up, but also for what went down or disappeared (requests, orders, events). The culprit service may show no errors at all and simply stop receiving calls.This is an inference drawn from the case walkthrough; the authors did not test the trick itself.
Resources
- Paper: Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
- Authors: Tingzhu Bi, Xun Zhang, Ping Wang, Meng Ma (Peking University); Jinglin Li (Beijing University of Posts and Telecommunications)
- Benchmark: RCAtlas (case list and code: https://ufile.io/5ww9iyfb)
- Mentioned: OpenRCA 2.0, RCAEval, Codex CLI, Claude Code; published agents FoA, RCAgent, RCA-Agent, mABC, DecompRCA