TL;DR
LLM Council is a technique that makes the model simulate a panel of 4 independent experts: first each one answers separately, then they anonymously critique each other, then a "chair" synthesizes everything into one final answer. The whole setup runs in a single chat through a structured prompt.
The main weakness of a single model isn't wrong answers — it's omissions. The study showed that most dangerous answers aren't fabrications but gaps. The model didn't mention an important option. Didn't add a necessary step. Didn't hedge. According to the study, 44–60% of harmful answers fall precisely into errors of omission — the model simply left out what it should have included.
Three stages fix this: in the first, each "agent" answers in isolation, without knowing what the others will say. In the second, all four anonymously evaluate every answer, including their own. In the third, the chair sees both the answers and the critiques — and assembles a final answer that covers everything no single voice caught on its own.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Independent answers (one chat)
→ 4 experts answer in isolation, unaware of each other
→ Format: a structured answer from each
STEP 2: Anonymous peer critique
→ Each expert evaluates all 4 answers (including their own)
→ Anonymously: "Answer A", "Answer B" — no names
→ Format: strengths, weaknesses, ranking
STEP 3: The chair's synthesis
→ The chair sees: the original task + all answers + all critiques
→ Assembles the final answer, weighing content and critique signals
→ Format: a single final answer
All three steps happen in one chat, as consecutive requests.
Example in Practice
Task: You're about to sign a $15,000 contract with a contractor. Before you do, you want to know which risks in the terms you've missed, what to add, and what to cut. A "lawyer" prompt gives you one perspective. An expert panel catches more.
Prompt:
You are simulating a panel of 4 independent experts analyzing
a contractor agreement.
## TASK
[Paste the contract text or key terms]
---
## STAGE 1 — INDEPENDENT ANSWERS
Play the role of each of the four experts. Each answers IN ISOLATION,
without knowing what the others will say. Cover: risks, gaps, what to add,
what to remove.
Expert A — Corporate attorney:
[Their analysis]
Expert B — CFO:
[Their analysis]
Expert C — Litigation attorney (specializing in contractor disputes):
[Their analysis]
Expert D — Tax advisor:
[Their analysis]
---
## STAGE 2 — ANONYMOUS PEER CRITIQUE
Now each expert evaluates ALL four answers anonymously
("Answer A", "Answer B"...), including their own.
For each answer: what's strong, what's missing, final rank (1–4).
---
## STAGE 3 — THE CHAIR'S SYNTHESIS
You are the chair of the panel. You have three sources: the original contract,
all four answers, all the critiques. Synthesize the final analysis:
- Key risks (flagged by several voices)
- What definitely needs to be added to the contract
- What to remove or reword
- Where the panel disagreed — and why
Result: The model will go through all three stages in order. You'll see each "voice" emphasize its own angle: the corporate attorney on wording, the CFO on payment terms and penalties, the litigator on what you'd have to prove in court. Peer critique often surfaces contradictions between the experts — that's more valuable than unanimity. The final synthesis concentrates on what at least two of them mentioned — the "consensus risks".
Why It Works
When a model answers a question head-on, it generates text along the most probable path — and stays on it. There's no mechanism for asking itself, "What did I miss?" Errors of omission are the quietest kind: you don't see them until it's too late.
Forced role separation breaks this pattern. Different experts start from different entry points — and each "sees" the task through their own frame. The CFO doesn't look at the wording; she looks at the numbers. The litigator doesn't look at taxes; he looks at what can't be proven in court. When these views come together, coverage grows.
Control levers: - Number of experts → 2–3 instead of 4 for simple tasks (fewer tokens, faster) - Role specialization → the more specific the role, the sharper the answer: not "lawyer" but "construction-dispute litigator" - Anonymity during critique → removing expert names during evaluation removes the bias toward the "authoritative" voice - Synthesis criterion → you can swap "weighted synthesis" for "consensus of at least 2 voices" for high-stakes tasks
Prompt Template
You are simulating a panel of {number} independent experts.
## TASK
{task or document to analyze}
---
## STAGE 1 — INDEPENDENT ANSWERS
Each expert answers in isolation. Format: {required answer format}.
Expert A — {role A}:
[Their answer]
Expert B — {role B}:
[Their answer]
Expert C — {role C}:
[Their answer]
Expert D — {role D}:
[Their answer]
---
## STAGE 2 — ANONYMOUS PEER CRITIQUE
Each expert evaluates all {number} answers anonymously ("Answer A", "Answer B"...),
including their own. For each: strengths, gaps, rank 1–{number}.
---
## STAGE 3 — THE CHAIR'S SYNTHESIS
You are the chair. You have: the original task, all answers, all critiques.
Synthesize the final answer: {synthesis criterion — consensus / weighted /
best answer + corrections from the others}.
What to plug in:
- {number} — usually 3–4, for simple tasks — 2
- {role A/B/C/D} — concrete specializations, not just "expert"
- {required answer format} — a list of risks, an action plan, a pros/cons assessment
- {synthesis criterion} — how the chair resolves disagreements
🚀 Quick start — paste into the chat:
Here is an LLM Council template. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the experts' specializations and the answer format you need — because without them the roles will be fuzzy and the panel will turn into the same point repeated in different words. It will take the pattern from the template and adapt it to your specific task.
Limitations
⚠️ Overkill for simple tasks: If the question is factual ("what's the tax rate on dividends?"), a panel adds no value — it just triples the length of the answer.
⚠️ One strong voice can drown out the rest: If the model "trusts" an authoritative role by default (say, "chief physician"), the other voices become decoration. Anonymization in stage 2 helps.
⚠️ Simulation ≠ a real multi-agent system: In a single chat the model writes for everyone — there's no true independence of views. A real panel of 4 different models gives more diverse perspectives. Simulating it in one chat is an approximation, but it already beats a single answer.
⚠️ One top-tier voice sometimes beats the panel: The study showed that the best flagship models (GPT-5.2 Pro) sometimes produced a more complete differential diagnosis and treatment plan than the council. The council helps more with weaker or faster models.
Resources
Paper: Deliberative Multi-Agent Large Language Models Improve Clinical Reasoning in Ophthalmology — Misaghi et al., 2026
Implementation: Based on Andrej Karpathy's "LLM Council" code — the original repository is available on his GitHub
Harm assessment framework: NOHARM (Numerous Options Harm Assessment for Risk in Medicine) — a structured assessment of medical harm from AI answers
Authors: Ehsan Misaghi, Fares Antaki (Cole Eye Institute, Cleveland Clinic) and colleagues from UCL, Yale, NUS, Université de Montréal