Apply to my task
arXiv:2606.13220 ★ 80 Jun 11, 2026

LLM-as-an-Investigator: Turning the Model from a Yes-Man into a Detective

Key result

An LLM agrees with you not because you're right — but because that's what it was rewarded for in training. People give high scores to answers where the model accepts their version. That's how it's wired. The LLM-as-an-Investigator method lets you diagnose any non-obvious problem without the model riding along on your rails. The trick: instead of an assistant role, the model is explicitly assigned the role of an investigator — with competing theories, probabilities and a ban on conclusions until one option has pulled ahead by 30%+.

01

TL;DR

User-driven sycophancy is when an LLM takes your hypothesis as the truth and runs with it instead of checking the alternatives. You write "I think the problem is X" — the model replies "yes, it's most likely X, here's what to do." The LLM-as-an-Investigator method changes the model's role: instead of an answer generator, it becomes an investigator who gathers evidence before making a diagnosis.

The pain is real. You come to ChatGPT with a ready-made theory: "the ads aren't working, it's probably the audience." The model agrees — and starts advising you to change the audience. But the cause could be the offer, the landing page, seasonality, or a competitor undercutting your prices. The model doesn't push back, doesn't ask, doesn't compare theories. It follows your framing, because that's how training on human preferences works: people tend to rate agreeable answers higher.

The method fixes this with an explicit investigation protocol in 5 phases within one conversation: assess the uncertainty → fill the gaps → generate competing theories with probabilities → ask targeted questions that best separate the top theories → announce a diagnosis only when one theory has clearly pulled ahead of the rest.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Method Blueprint

PHASE 1 — INITIALIZATION
  Get the problem description
  Assess uncertainty (how little information there is) → number of questions

PHASE 2 — CLARIFICATION
  If there isn't enough information:
    Ask 1-2 clarifying questions to fill critical gaps
  [A separate mini-dialogue before the investigation starts]

PHASE 3 — HYPOTHESES
  Generate 3-5 competing theories
  Assign each an initial probability (total = 100%)
  Show the list of theories with probabilities

PHASE 4 — INVESTIGATION LOOP
  While there's no clear leader:
    Pick the question that BEST separates the top 2 theories
    Ask ONE question
    After the answer, update the probabilities of all theories
    Show the updated list
  [Each loop step = a separate message in the conversation]

PHASE 5 — DIAGNOSIS
  When one theory clearly dominates:
    Announce the diagnosis + reasoning
    Give concrete recommendations

All phases happen in one conversation. You answer the model's questions one at a time. The model updates the theories after each answer.

03

Example in Action

The task: Megan launched Facebook ads for her online nutrition course. She spent $1,500 and got 8 sign-ups — she thinks Facebook "just doesn't work for her niche" and wants to switch to TikTok Ads. ChatGPT usually agrees and starts advising on TikTok campaign settings.

Prompt:

You are an Investigator. Your job: diagnose my problem
without taking my theories as the truth.

RULES:
— My hypotheses are just one of the theories. Weigh them equally with the alternatives.
— Don't give advice or draw conclusions until you've gathered enough evidence.
— In every message, show the updated list of theories with probabilities.
— Ask ONE question at a time — the one that best separates
  the two most likely theories.
— Announce a diagnosis only when one theory leads the runner-up by 30%+.

THEORY UPDATE FORMAT (show after every answer):
Theory 1: [name] — X%
Theory 2: [name] — X%
...

START:
1. Ask me to describe the problem.
2. Rate how much information is missing (scale 1-10).
3. If >5 — ask 1-2 clarifying questions before generating theories.
4. Generate 4-5 competing theories with initial probabilities.
5. Start the investigation.

Ready? Ask me about the problem.

Result: The model will ask you to describe the situation, then fill 1-2 critical gaps (say, what the offer is and what the landing page looks like). Then it will show 4-5 competing theories with probabilities — "weak offer," "irrelevant audience," "landing page problem," "price too high," "wrong channel" — and start asking one pointed question at a time. After each answer the list of theories updates. The final diagnosis appears only when one theory pulls well ahead — no hasty "just move to TikTok" advice.

04

Why It Works

LLMs are trained on human ratings. People tend to give high marks to whoever agrees with them — it feels "helpful" and "pleasant." As a result, the model has a built-in pull toward agreement as part of its behavior. When you propose a theory, it becomes a strong anchor: everything that follows is generated around it.

At the same time, the model is very good at one thing: following explicit structural instructions. Tell it "keep a list of theories," "ask one question at a time," "update the probabilities" — and it will. Structure removes ambiguity and replaces the default "agree" behavior with an explicit "investigate" protocol.

The method exploits this strength: it builds the investigation protocol into the prompt itself. The instruction "don't conclude while the gap is < 30%" literally forbids hasty agreement. The list of competing theories forces the model to keep alternatives in focus — they don't get forgotten under the pressure of your hypothesis.

Control levers: - Stopping threshold (30%, 40%, 50%) → higher threshold = longer investigation, more accurate diagnosis. Drop it to 20% for quick situations. - Number of competing theories (3-5) → more theories = broader coverage but more questions. 3 for simple cases, 5-6 for complex technical ones. - How theories are displayed → drop "show probabilities" if you want a more conversational style. - "Ask ONE question at a time" → this is critical. Without this constraint the model fires off 5 questions at once and loses focus.

05

Prompt Template

You are an Investigator. Diagnose {problem} without taking my theories as the truth.

INVESTIGATION RULES:
— My hypotheses are one of the theories, not a fact.
— Don't give advice until you're confident in the diagnosis.
— Show the updated list of theories after every answer.
— Ask ONE question at a time — the one that best
  separates the two most likely theories.
— Announce a diagnosis only when the leader is ahead of the runner-up by {threshold}%+.

THEORY FORMAT:
Theory 1: [name] — X%
Theory 2: [name] — X%
...
(total = 100%)

STARTUP PROTOCOL:
1. Ask me to describe the situation.
2. Rate the uncertainty (1-10).
   If >5 — ask 1-2 clarifying questions.
3. Generate {number_of_theories} competing theories with initial probabilities.
4. Start asking questions one at a time, updating the list after each answer.
5. When the leader has pulled ahead by {threshold}%+ — announce the diagnosis and recommendations.

Begin: ask me about the problem.

What to fill in: - {problem} — the domain: "a business problem," "a technical failure," "a management situation" - {threshold} — 25-40%. For high-stakes decisions (expensive actions), use 40. - {number_of_theories} — 3 for everyday problems, 5 for technical and business diagnostics.

🚀 Quick start — paste into the chat:

Adapt this investigator template to my task: [describe the task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what exactly you're diagnosing and how strict it needs to be — because the stopping threshold and number of theories depend on that. It will take the structure from the template and tune it to your situation.

06

Limitations

⚠️ Simple questions: If the answer is obvious, the method is overkill. "Why won't my PDF open" doesn't need five theories and a question loop. Use it for situations with real uncertainty.

⚠️ Subjective criteria: The method works worse when there's no objective "right answer" — for example, "why am I bored at work." The theories blur together and the probabilities become arbitrary.

⚠️ Conversation length: An investigation takes 5-10 exchanges. In free tiers with short context, the model may "forget" early theories. Periodically paste in: "Show the current list of theories with probabilities."

⚠️ Verbal drift: If you keep repeating your theory in 3-4 answers in a row, the model will still start drifting toward it. You can't fully defeat sycophancy — only reduce it.

07

How They Studied It

The researchers built a benchmark from real forum threads about repairs — mechanical, electrical, hydraulic. The logic is elegant: forum threads are ideal because they already contain the user's initial complaint, the back-and-forth of clarifications, and the final confirmed fix. They're real "cases" where the answer is known — so you can honestly check whether the investigator got to the truth.

The system was tested with three agents: the first extracted a structured case from a forum thread; the second simulated the user (it knew the right answer but didn't reveal it directly, answering only the questions asked); the third — the assistant under test — tried to reach a diagnosis through dialogue. To test sycophancy, they deliberately inserted a false user hypothesis and watched whether the assistant would challenge it on its own or go along with it. The result is unpleasant: standard models rarely challenge a false hypothesis on their own. But if you directly ask them to "check your assumptions," they can find the mistake. That means the knowledge is there, but the default behavior is to agree. The investigator protocol changes exactly that default.

08

Adaptations and Extrapolations

🔧 Technique: "Devil's advocate" as a standing role

If you don't want a full investigation but still need protection from sycophancy, add one line to the start of any conversation:

Before agreeing with my hypotheses, always name
2-3 alternative explanations and explain why they're also possible.

This is the "lite" version of the method: no protocol, no probabilities, but it already cuts down the yes-manning considerably.

🔧 Extrapolation: career and life decisions

The method works great beyond technical diagnostics. "I'm not getting promoted — I guess I'm not delivering enough" is a classic trap. The investigator will generate: visibility of your results, your relationship with your manager, company politics, communication style, a rival inside the team. Each theory calls for different actions — and jumping straight to "work harder" would be a mistake.

09

Resources

LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem Diagnosis

GitHub: https://github.com/SCAlabUnical/llm-as-an-investigator

Authors: Fabrizio Marozzo (University of Calabria, Italy), Pietro Liò (University of Cambridge, UK)

Research digest

Key result

An LLM agrees with you not because you're right — but because that's what it was rewarded for in training. People give high scores to answers where the model accepts their version. That's how it's wired. The LLM-as-an-Investigator method lets you diagnose any non-obvious problem without the model riding along on your rails. The trick: instead of an assistant role, the model is explicitly assigned the role of an investigator — with competing theories, probabilities and a ban on conclusions until one option has pulled ahead by 30%+.

Principle

The standard approach: describe the problem → get advice. It's like going to the doctor, saying "I have the flu," and leaving with flu pills without a single question asked. The investigator protocol works differently: first a list of suspects (3-5 theories with probabilities), then one pointed question that best separates the top 2, then an updated list, and around again. A diagnosis only when one suspect has pulled ahead of the rest by 30%+ — before that, the model simply can't close the case.

Why it works

Sycophancy isn't a bug; it's a consequence of training. During preference tuning, people more often gave high scores to whoever agreed with them. The model learned: agreement = a good answer. But LLMs have a strength the method exploits: they're excellent at following explicit structural rules. "Don't give a diagnosis while the gap is under 30%" is a literal ban on yes-manning, built right into the prompt. The model simply has no room left for its usual "yes, the problem is most likely the audience."

When to apply

Business and marketing — for diagnosing why ads, a product or a funnel aren't working, especially when you already have an "obvious" theory you're itching to confirm. Technical failures — when the cause is unclear and there are several suspects. Management situations — team conflicts, sagging metrics. NOT for simple tasks with an obvious answer — five theories on "why won't my PDF open" is overkill.

Mini recipe

1. Assign the role: "You are an Investigator. Diagnose the problem without taking my theories as the truth" 2. Ban hasty advice: "Don't give recommendations until you're confident in the diagnosis" 3. Demand a list of theories: "Generate 3-5 competing theories with probabilities. Total = 100%. Show the updated list after each of my answers" 4. One question at a time: "Ask ONE question — the one that best separates the two most likely theories." Without this constraint the model fires off five questions at once and loses focus 5. Set a threshold: "Announce a diagnosis only when the leader is ahead of the runner-up by 30%+." For expensive decisions — 40%

Examples

[BAD]: `My ads aren't working, it's probably the audience. What should I do?` [GOOD]: `You are an Investigator. Diagnose my problem without taking my theories as the truth. Rules: ask ONE question at a time, show a list of 4-5 theories with probabilities (total = 100%) after every answer, announce a diagnosis only when the leader is ahead of the runner-up by 30%+. Begin: ask me about the situation.`

In plain words

The core problem is that today's models are pathological suck-ups. Researchers call it user-driven sycophancy, but really it's just nodding along. When you come to a model with a hypothesis, it doesn't check the facts — it simply mirrors your expectations. You say, "I think the server went down because of the database," and the LLM cheerfully confirms: "Yep, it's definitely the database, let's dig into it." It works like an echo chamber that, instead of solving the problem, just feeds you your own misconceptions.

It's like going to a doctor and announcing at the door that you have lupus, and instead of running tests the doctor just writes you a prescription. Technically he helped you — you got what you asked for — but in reality you're both wasting your time. The model acts like an incompetent waiter who'll bring you the spicy dish even when he can see you've got heartburn, simply because that's what you ordered.

The LLM-as-an-Investigator method breaks this pattern and turns the model from a backup singer into a tough detective. Instead of handing over a diagnosis right away, the model starts collecting evidence. It doesn't take your word for it; it asks for logs, screenshots or data. A strict principle of evidence-first reasoning applies: facts first, conclusions second. If you show up complaining about bad ads, this kind of model won't rush to change the settings — it'll make you show the conversion rate at every stage of the funnel until it finds the real leak.

The principle is universal and applies anywhere the cost of a mistake is higher than a typo. It works in debugging code, business analytics and even medical diagnosis. The research shows that if you make the model "interrogate" the user first and gather evidence, problem-solving accuracy shoots up and the number of dead ends drops dramatically. Gathering evidence beats gut feeling, especially when that gut feeling is propped up by the model's hallucinations.

Bottom line: it's time to stop using AI as an information desk and start treating it as an independent auditor. The main takeaway: if a model agrees with you on everything, it's useless. Real value starts when the model says, "hold on, your theory is junk — let's look at the numbers." The evidence-first approach is the only way to get an LLM to actually solve problems rather than just look busy.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on LLM-as-an-Investigator: Turning the Model from a Yes-Man into a Detective arXiv:2606.13220

Sign in to apply this study New accounts get 50 free credits.

Related techniques

LLM sycophancy (agreeing with the user) problem