Direct Judge: Comparing Texts with 6 Error-Type Examples Beats Multi-Agent Systems

When you need to check whether meaning got lost in translation or paraphrasing of a technical text, a single LLM call with 6 examples of different error types works more accurately than an elaborate system of eight "agents" that take turns translating, extracting structure, cross-checking, and critiquing each other. Researchers compared 6 different architectures — from simple rule-based matching to a full multi-agent system — and the simplest version with examples beat all the others.

The pain point is very concrete: when translating technical requirements, meaning can "drift" without anyone noticing. The phrase "must respond within 50 milliseconds" can easily turn into "should ideally respond within 50 microseconds" — the grammar is clean, it reads smoothly, but the requirement just changed by a factor of a thousand and shifted modality (mandatory → optional). Standard text-similarity metrics (like BLEU) don't catch this kind of swap, because the words look similar.

The method is simple: give the model both texts at once plus 6 carefully picked examples that show different kinds of "meaning drift" — a harmless paraphrase, a number change, a polarity flip (must not → must), an object swap, a weakened obligation, a dropped condition. In a single call, the model outputs: is there drift or not, a confidence level, and an explanation — no multi-step architecture required.

Method Blueprint

ONE LLM CALL:
Input: [text A] + [text B] + [6 examples of different types of meaning divergence]
Output: drift yes/no → confidence (0-1) → explanation of which field changed

All 6 examples go straight into the prompt as few-shot examples (shown before the task itself). No separate calls, no roles, no stages — just one request.

Example in Practice

Task: A European dev studio sent a technical spec to an overseas freelance contractor, machine-translated into English. The team needs to check whether critical numbers and wording — deadlines, limits, mandatory language — survived the translation.

Prompt:

Compare two versions of a technical requirement — the original and the translation.
Determine whether the technical meaning changed (not the style, but the actual meaning:
numbers, units of measurement, obligation level, conditions).

Here are 6 examples of what counts as a divergence and what doesn't:

1. NO DIVERGENCE (just a paraphrase):
Original: "The system must save data every 5 minutes"
Translation: "Every 5 minutes, the system is required to perform a data save"
→ Drift: NO. The meaning is identical, only word order changed.

2. DIVERGENCE (number):
Original: "Response time must be under 50 ms"
Translation: "Response should be under 50 seconds"
→ Drift: YES. The unit changed from milliseconds to seconds — critical.

3. DIVERGENCE (polarity):
Original: "The function must not send data to third parties"
Translation: "The function may send data to third parties"
→ Drift: YES. "Must not" became a permission — the opposite meaning.

4. DIVERGENCE (object swap):
Original: "The authorization module validates the user token"
Translation: "The logging module validates the user token"
→ Drift: YES. "Authorization module" was replaced with "logging module."

5. DIVERGENCE (weakened obligation):
Original: "Backups are mandatory and must be performed daily"
Translation: "Backup is recommended to be performed daily"
→ Drift: YES. "Mandatory" became "recommended" — the requirement was weakened.

6. DIVERGENCE (dropped condition):
Original: "If the limit of 100 requests is exceeded, the system blocks the IP for an hour"
Translation: "The system blocks the IP address for an hour"
→ Drift: YES. The condition "if the limit of 100 requests is exceeded" was dropped.

Now compare these two requirements:

Original: {source text}
Translation: {translated text}

Answer:
1) Drift: YES/NO
2) Confidence (low/medium/high)
3) What exactly changed (if anything)

Result: The model returns a short verdict on three points — whether there's a divergence or not, how confident it is, and, if there is a divergence, exactly which field "drifted" (a number, an obligation level, an object, a condition). To check dozens of items in a spec, you can run this prompt line by line.

Why This Works

An LLM struggles with this task when asked point-blank and abstractly: "is there a difference in meaning?" Without a reference point, the model tends to either fixate on style or miss subtle meaning shifts, because it doesn't know which specific categories of change actually matter.

But LLMs are excellent at generalizing from a pattern. Show it 6 distinct error types explicitly, and the model picks up the pattern of "what counts as a divergence" and applies it to a new pair of texts — even if the specific case doesn't literally match any of the examples.

Multi-agent systems (translate → extract structure → cross-check → critique → final verdict) look like a logical way to improve accuracy, but in practice each stage can introduce its own error, and intermediate steps lose context that would have been visible in a direct, whole-text comparison. An error at one stage gets carried down the chain — and the specialization of the agents doesn't make up for that loss.

Levers you can pull: - Number and variety of examples — if your task isn't about technical requirements but, say, legal language or marketing copy, swap in your own divergence categories instead of the 6 shown here (for contracts: money, deadlines, liability; for marketing: numbers, promises, guarantees). - Ask for an explanation, not just a verdict — this forces the model to state explicitly WHAT changed, instead of just guessing "yes/no." - Don't take the confidence score literally — in the study, the model systematically overstated its confidence. It's better to ask for a word-based rating ("low/medium/high") than a number, or to explicitly ask what's causing any doubt.

Prompt Template

Compare two texts — the original and the {translation/paraphrase/version}.
Determine whether the meaning changed in the aspects that matter to me: {what matters — numbers, obligation level, conditions, objects}.

Here are examples of what counts as a divergence and what doesn't:

1. NO divergence: [example of a paraphrase with no meaning lost]
2. DIVERGENCE ({type 1}): [example]
3. DIVERGENCE ({type 2}): [example]
4. DIVERGENCE ({type 3}): [example]
5. DIVERGENCE ({type 4}): [example]
6. DIVERGENCE ({type 5}): [example]

Now compare:
Text A: {original}
Text B: {version to check}

Answer:
1) Divergence: YES/NO
2) Confidence: low/medium/high
3) What changed (if anything)

Fill in {type 1-5} with the divergence categories that matter for your task (numbers, obligation, polarity, objects, conditions — or your own, if you're comparing something other than technical text).

🚀 Quick start — paste this into the chat:

Here's a template for checking whether meaning changed during translation/paraphrasing of a text.
Adapt it to my task: {describe what texts you're comparing and what shouldn't be lost}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which types of divergence matter most for your specific task (legal text, technical documentation, marketing copy) — because without domain-specific examples, the method won't be nearly as accurate.

Limitations

⚠️ Doesn't work well on general-purpose text: the method was trained around technical patterns (numbers, units, obligation level). On ordinary text without clear formal markers (literary translation, for instance), accuracy drops sharply.

⚠️ Doesn't distinguish direction of meaning: if the task isn't "did the meaning change or not" but rather "does text B logically follow from text A," the method gets confused. It's good at spotting "these texts are different," but it doesn't always grasp which direction the difference matters in.

⚠️ The confidence score isn't reliable on its own: the model almost always overstates its confidence. Don't trust a bare "confidence: 92%" without an additional check.

⚠️ For adversarial cases (texts deliberately made to look similar but differ in meaning), pure few-shot performs slightly worse than an explicit structural breakdown into components — if the texts are specifically designed to mask a difference behind surface similarity, decomposing them into parts before comparing helps.

How It Was Studied

The researchers created 300 technical requirements across 10 engineering domains (from medical devices to cybersecurity) and generated 890 variants with controlled distortions — deliberately corrupting numbers, polarity, obligation level, and so on. They then ran 6 different architectures: from simple rule-based matching to a full system with eight specialized "agents" (translator, structure extractor, critic, judge, and so on).

The simplest version — a single call with 6 examples — outperformed everything else by a wide margin, while taking 17 minutes instead of 181 for the complex alternative. This surprised the researchers: they expected agent specialization to yield better results, but errors accumulating at intermediate stages hurt the final verdict more than specialization helped.

The method was also tested on two external datasets — one with adversarial paraphrases (PAWS-X) and one with logical entailment (XNLI). On the adversarial examples, structural decomposition edged out direct comparison — meaning that for "tricky" similar-looking texts, explicitly breaking them into components does help. On the entailment dataset, all methods performed poorly — because that task is fundamentally different (directional logic rather than mutual equivalence).