TL;DR
MAScope is a way to analyze multi-agent system logs with an LLM judge. Along with the log, the judge gets the "shape" of the system: its topology (who exchanges information with whom), the frequencies of typical failures for that topology, and a short description of how it usually breaks. The judge walks through a list of 14 failure types and decides yes or no for each one.
The headline finding: every topology has its own signature failures. In an "agent → agent → agent" chain, facts get lost along the way. In a "manager and workers" setup, the manager's check gradually drifts away from the workers' task. On a shared "board", context piles up and each new reader silently reinterprets it their own way. In logs with peer agents you get circles of mutual waiting and role confusion. Without this knowledge the judge sees the same symptoms ("the result is wrong", "there is no check") and cannot tell where exactly the handoff of information broke. A cheap model without a hint rarely guesses right.
The method boils down to two steps. First, identify the topology. If you know the setup in advance, you can skip this: one topology covers thousands of runs. Then give the judge the log, the topology, the failure frequencies and the description of failure patterns. With that hint, a cheap model caught up with a strong one that saw only the log.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1 (once per system): identify the topology
From the orchestration config OR reconstruct it from the log → one of 4 labels:
Layered / Centralized / Decentralized / Shared Message Pool
STEP 2 (for every log): diagnosis in ONE prompt
Input: log + topology label + failure frequencies for it
+ a short description of "how this topology usually breaks"
Instruction: check each failure type INDEPENDENTLY against the text of the log
Output: JSON — one 0/1 for each of the 14 failure types
Automatic topology reconstruction (called TSE in the paper) runs in four stages: compressing the log and anonymizing the participants, extracting links with pointers to quotes, verifying and repairing the links, and assigning a label. You only need it when the configuration is not at hand.
Example in Practice
Task: You sell on Amazon and built a chain of three agents in n8n: "Review Analyst → Listing Copywriter → Editor". The brief has a hard requirement: do not promise "health benefits" and do not mention prices. Yet every so often the output still says "helps with insomnia". You export the run log and ask a model to find the cause.
Prompt:
<Role>
You are a failure diagnostician for multi-agent systems. You check a run log
and determine which failure types are present in it.
</Role>
<Topology>
Layered (a layer-by-layer chain): Review Analyst → Listing Copywriter → Editor.
Information flows only forward; there is no feedback loop.
</Topology>
<FailurePrior>
The most common failures in layered systems:
- information withholding (an agent did not pass an important fact along) — high frequency
- ignoring input from the previous agent — high frequency
- all other types — below average
</FailurePrior>
<FailureSignature>
In layered chains, facts and constraints get lost on the way forward:
an upstream stage knew the requirement, a downstream stage never received it or ignored it.
"Clarify — rewrite" loops between neighboring roles are possible.
</FailureSignature>
<Instruction>
Check EACH failure type from the list below independently, based on quotes from the log.
The frequencies are only a soft hint about where to look more closely;
do not put "1" without evidence in the log.
List of failure types: {failure_type_list}
Return JSON: for each type — 0 or 1 and a short evidence quote.
</Instruction>
<Trace>
{run_log}
</Trace>
Result: The model returns JSON with a flag for each failure type and quotes from the log. The most likely findings for this topology: the "no health claims" constraint was lost at the Analyst → Copywriter handoff, or the Editor ignored its input. You will see at which handoff the requirement disappeared and can fix the transfer: add the constraint to the Copywriter's prompt explicitly.
Why It Works
The LLM's weakness. Without structure, a judge model treats the log as one long conversation and grades local coherence. The last exchange "looks fine". That one of the requirements got lost three turns ago and nobody ran the check is something it barely notices. Identical symptoms in different systems have different causes, and the log does not spell that out.
The LLM's strength. It reads context well and checks claims against quotes, as long as it has been told where to look. A short description of "how this structure usually breaks" works like a map: the judge checks specific points where information changes hands instead of the whole log in sequence.
How the method uses this. The topology turns the general question "what went wrong?" into a narrower one: "how did information get lost in a layered chain?". The frequencies set where to look first. The instruction "check each type independently" keeps the frequencies from outweighing the evidence in the log.
Control levers: - Topology and signature → swap in your own setup. This is the main lever. - Frequencies → take them from your own labeled runs. The closer to your system, the better. - Strictness of the instruction ("do not put 1 without a quote") → tighten it if the judge sees failures everywhere. - List of failure types → cut it down to the ones that matter to you. To start, 4 groups are enough: information loss, coordination stall, semantic drift, verification failure.
Prompt Template
<Role>
You are a failure diagnostician for multi-agent systems. You read a run log
and determine which failure types are present in it.
</Role>
<Topology>
{topology}: {description_of_who_hands_off_to_whom}
</Topology>
<FailurePrior>
Frequency of each failure type in systems with this topology:
{frequencies_by_type}
</FailurePrior>
<FailureSignature>
{short_description_of_how_this_topology_usually_breaks}
</FailureSignature>
<Instruction>
Check EACH failure type independently, based on quotes from the log.
The frequencies are a soft hint, not evidence.
Failure types: {failure_type_list}
Return a single JSON: for each type — 0/1 and an evidence quote.
</Instruction>
<Trace>
{log}
</Trace>
What to fill in:
- {topology} — one of four:
- Layered: a chain, information flows only forward.
- Centralized: a hub hands out tasks and checks the results.
- Decentralized: peer agents talk to each other directly.
- Shared Message Pool: a shared board, everyone reads and writes.
- {frequencies_by_type} — the share of your runs with that failure. In the paper they were computed from 400 labeled logs and frozen.
- {short_description_of_how_this_topology_usually_breaks} — 2–3 sentences. Samples from the paper:
- Layered: facts lost moving forward down the chain, "clarify — rewrite" loops.
- Centralized: the hub's verification criterion drifts from the workers' task, the hub becomes a bottleneck.
- Decentralized: mutual waiting, role confusion, drift in mutual critique.
- Shared Message Pool: context with blurry boundaries, later readers silently reinterpret earlier intentions, the verification duty belongs to "nobody".
- {failure_type_list} — in the paper these are the 14 types of the MAST taxonomy.
🚀 Quick start — paste into the chat:
Here is a topology-aware failure diagnostician template for a multi-agent system.
Adapt it to my system: [describe the agents and who hands work to whom].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask how the agents pass information to each other, who checks the result and which failures you have already seen, because the topology, signature and frequencies depend on that. It will take the pattern from the template and assemble a ready-to-use judge prompt for your chain.
Limitations
⚠️ Diagnosis quality is still modest: the best test result is around 0.35–0.40 Macro-F1 (accuracy averaged over the 14 failure types). Treat it as a hint for log analysis, not a verdict. Double-check the judge's conclusions against the quotes.
⚠️ Strong models barely benefit: for a strong model the gain from the topology is small (about 7%). The method mainly brings a cheap model up to the level of an expensive one.
⚠️ Noisy labels, topology tied to framework: in the main test set each framework has a single topology. The authors checked this with a control corpus built on one codebase, but some conclusions still partly lean on "framework fingerprints". There are no decentralized systems in the main test set; those were tested on synthetically injected failures.
⚠️ The automatic part needs code: reconstructing the topology from a log (TSE) is a pipeline with per-framework adapters. You cannot replicate it in a chat. But if you know your setup, you do not need this step.
⚠️ The savings are a projection: the "6% of the cost" figure was calculated for a fixed topology, not measured in real-world operation.
⚠️ Frequencies need labeled data: without your own labeled logs there is nowhere to get frequencies from. How much each component of the hint (label, frequencies, description) contributes on its own is not shown in the accessible part of the paper.
How They Tested It
The researchers collected 2220 traces of multi-agent runs into three corpora. The first is 851 "clean" traces from the MAST dataset with ready-made expert failure labels. The second is 961 traces with artificially injected failures; it covers decentralized systems. The third is 408 traces on the same LangGraph codebase, where only the links between agents were changed.
First they checked whether the failure type depends on topology. An independence test (χ² = 409.9) showed that the link exists, and it is very strong. The positive deviations for each topology fall on different failure types. The control corpus on a single codebase was there so the link could not be written off as "framework style".
Then they compared a judge without the topology to a judge with it. For the cheap gpt-mini, Macro-F1 rose from 0.173 to 0.350, a twofold jump. With the topology the system reconstructed on its own, it reached 0.346. The strong gpt-5.4, seeing only the log, scored 0.372. gpt-5.4 with the topology scored 0.399. The surprise was that a cheap model with the right context nearly caught up with an expensive one without it. In practice this means structural knowledge about the system replaces part of the model's horsepower. If the topology does not change, you identify it once, and diagnosing 1000 logs costs about 6% of repeatedly calling the strong model. Feeding knowledge through the prompt instead of training the judge makes the diagnosis transfer more robustly to new frameworks. In a "leave one framework out" check, the method kept 82% of its quality.
Adaptations and Extrapolations
These are my own variations. The paper did not test them. The exact method comes first above; here are just ideas.
🔧 Technique: from diagnosis to prevention → bake the "weak-spot signature" into the agents' instructions. The paper shows where each topology breaks. You can turn that into rules for the agents themselves:
# For a chain (Layered) — in the prompt of every stage:
At the end of your answer, output an "Original requirements" block, copying
the constraints from the brief word for word. The next stage must respect them.
# For a hub (Centralized) — in the orchestrator's prompt:
Before accepting the work, compare your verification criterion with the original brief
the workers received. If they differ, fix the criterion.
# For a shared board (Shared Message Pool) — in every agent's prompt:
Every entry on the board contains: the author, which requirement it relates to,
and who is responsible for checking it. No check may belong to nobody.
# For peer agents (Decentralized):
If you trade critique for two turns in a row without a new result,
stop and hand the decision to the user.
Extrapolation: judge + self-repair. Add an instruction at the end of the judge prompt: "For every failure found, propose an edit to the agent's instructions that would eliminate this failure type". You get a "log → diagnosis → system prompt edit" loop.
Resources
- Paper: Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failures — Xinwen Liu, Jiawei Zhang, Zhuocheng Pan, Xudong Liu, Isabella Zhu, Tianyu Wo; Beihang University, Beijing.
- Foundations: the MAST failure taxonomy (14 types); the Guo et al. topology classification (layered, centralized, decentralized, shared message pool); Who&When, A2P.
- Data: MAScope-Bench on Hugging Face (
mascope/mascope-bench); code in an anonymous repository, link in the paper.