A Multi-Agent Framework for Evaluation: Splitting a Hard Judgment Across Independent Agents

STEP 1: Preprocessing Agent

Input: raw text (with noise, filler words, repetition)
Task: clean it up and pull out the substance of the answer
Output: clean text, ready to be scored

STEP 2: Scoring Agents (a separate prompt for each criterion)

Input: clean text + rubric for one criterion + 3 examples (low/mid/high)
Task: score ONLY this criterion on the scale
Output: score + rationale

9 criteria = 9 separate calls. Each agent sees only its own task.

Task: Score startup pitches on three criteria: (1) Problem clarity, (2) Feasibility of the solution, (3) Market understanding. You have 15 pitches from an accelerator cohort, need to give each a 1-7 score, and decide who gets invited to the next round.

Prompt 1 (Preprocessing):

Here is a transcript of a startup pitch from Demo Day. Remove filler words ("um", "like", "you know"), repetition and tangents. Keep only the substance: what problem they solve, how they solve it, and what they know about the market and competitors.

TRANSCRIPT:
[paste the raw pitch transcript]

Return a clean version focused on content, no fluff.

Prompt 2 (Scoring Agent, criterion "Problem clarity"):

Rate how clearly the problem is stated in this pitch on a 1-7 scale:
1 = the problem isn't named or makes no sense
4 = there is a problem, but it's fuzzy
7 = a crystal-clear problem with examples and real customer pain

Here are 3 calibration examples:

EXAMPLE (score 2): [low score: a pitch where the problem is not clear at all]
EXAMPLE (score 4): [medium score: the problem is there, but vague]
EXAMPLE (score 7): [high score: the problem is sharply stated]

Now rate this pitch:
[paste the cleaned text from Prompt 1]

Return: a score (1-7) + a short rationale.

Prompts 3 and 4: Same thing for "Feasibility of the solution" and "Market understanding": separate calls with DIFFERENT calibration examples.

Result: You get 3 separate scores with rationales. The model won't blend the criteria: a crisp problem statement won't pull up the feasibility score. The calibration examples stop the model from handing out high marks to everyone, because it can see where the scale ends.

Models are bad at keeping criteria independent inside one prompt. Ask for 5+ dimensions at once and you get the halo effect: a high score on one criterion pulls the others up. Plus the model can't juggle 5 different scales at the same time, so it starts averaging or falls back on a general impression.

Models are great at a narrow, well-defined task: "rate ONLY empathy, here are examples of the boundaries." Semantic representations (embeddings) capture topics, not context: "walked over to my friend" and "walked away from my friend" are semantically close (movement + friend), yet mean opposite things when you're scoring empathy. A model with a rubric and examples sees the implication of the action, not just the topic.

The multi-agent framework relies on divided attention: each agent focuses on one job and isn't distracted by other criteria. Preprocessing removes the noise that makes it hard to extract the signal (filler, repetition). 3-shot calibration with low/mid/high examples sets the ends of the scale: the model sees "this is what a 2 looks like, here's a 4, here's a 7" instead of making it up.

Control knobs:

Number of examples (3-shot): The research found 3 to be the sweet spot. Fewer (0-shot, 1-shot) means no calibration, and the model guesses where the boundaries are. More (4+) triggers recency bias: the model overfits to the last example and ignores the rest. If your task is simple (yes/no, fact/not a fact), you can drop to 1-shot. If it's complex and subjective, keep 3.

Example percentiles (low/mid/high): Pick examples from both ends and the middle of the scale. Only high examples and the model inflates everything. Only low ones and it doesn't understand the upper bound. A balanced set is critical for calibration. If you don't have a ready-made pool, ask an LLM to generate synthetic examples for each level.

Preprocessing: If the text is already clean (an article, an email), you can skip it. If it's noisy (a transcript, a chat, social media), preprocessing is a must. The alternative is telling the scoring agent to "ignore filler words," but a separate step is more reliable.

Independent agents vs grouped: If criteria are correlated (writing style + grammar), you can group them in one prompt. If they're different (empathy + logic), keep them separate: the extra tokens for independent scoring pay for themselves in accuracy.

Preprocessing Agent

Here is a text to be evaluated. Remove the noise: filler words, repetition, off-topic tangents. Keep only the substance that relates to {task description}.

ORIGINAL TEXT:
{raw_text}

Return a clean version focused on content.

Scoring Agent (one per criterion)

Rate {criterion_name} on a {min}-{max} scale:
{min} = {description of the low end}
{midpoint} = {description of the middle of the scale}
{max} = {description of the high end}

Here are 3 calibration examples:

EXAMPLE (score {low}):
{low_score_example_text}

EXAMPLE (score {mid}):
{mid_score_example_text}

EXAMPLE (score {high}):
{high_score_example_text}

Now rate this text on the criterion "{criterion_name}":
{cleaned_text_from_preprocessing}

Return: a score ({min}-{max}) + a short rationale.

How to fill it in: - {criterion_name}: what you're scoring, e.g. "empathy," "strength of argument," "problem clarity" - {min}, {max}: the ends of the scale (1-5, 1-7, 1-10) - {boundary descriptions}: what each level means, in plain words - {examples}: real or synthetic texts with scores from both ends and the middle of the scale

If you don't have examples on hand, ask an LLM to generate them: "Generate 3 sample answers on {topic}: with low/medium/high {criterion}. Explain what makes each one weak/average/strong."

🚀 Quick start: paste this into the chat:

Here is a Multi-Agent Framework for scoring complex criteria. Adapt it to my task: {description of your evaluation task}.

I want to rate {what you're evaluating} on the criteria {list of criteria} using a {which scale} scale.

Ask me questions first:
- Which examples should we use for calibration?
- Do we need preprocessing, or is the text already clean?
- How should we phrase the scale boundaries for each criterion?

Then build me a set of prompts: 1 preprocessing prompt (if needed) + a separate scoring prompt for each criterion.

[paste both templates above]

The LLM will ask about the specifics of your task (which criteria, whether you have past scores, what scale) because it needs to pick relevant calibration examples and phrase the scale boundaries clearly. It will take the structure from the template and fill it in for your case.

Problem 1: Cross-criterion interference

When a model scores several criteria in one prompt, it doesn't keep them independent. Picture this: you ask it to rate a pitch on 5 criteria at once. The model generates its answer sequentially: first "Problem clarity: 7," then "Feasibility: ...". By the time it reaches the second criterion, it already "knows" the first score was high, and that colors its judgment. If the problem is stated brilliantly (7), the model's "mind" (its weights and attention) drifts toward a positive context and nudges the next scores up. That's the halo effect, a cognitive bias baked into the architecture of autoregressive generation.

Fix: Separate calls = a context reset. Each scoring agent never sees the other criteria's scores and works in isolation.

Problem 2: Calibration drift

Without examples, models invent the scale on their own, and often get it wrong. Especially on imbalanced data: if in reality 85% of scores are 4-6 (as in the MMI), the model gravitates to that range and misses the nuances. Or the opposite: it inflates everything because the instruction sounds upbeat. Calibration = anchoring: low/mid/high examples show the model "this is a 2, this is a 5, this is a 7" and set the boundaries.

The research showed: give only high examples and the model hands out 6-7 to everyone. Only low ones and it doesn't grasp the upper bound, so everyone gets 3-4. A balanced set (low/mid/high) is critical.

Fix: 3-shot with low/mid/high percentiles gives the model reference points across the whole scale.

Problem 3: Semantic similarity ≠ evaluative similarity

Embeddings (BERT, Sentence-BERT) group texts by topic, not by quality. "I walked over to my friend" and "I walked away from my friend" are semantically close: both are about movement, both mention a friend. The model clusters them together as "actions involving a friend." But in the context of empathy, they're opposite actions: walking over = support, walking away = avoidance.

This is a fundamental limitation of embeddings: they capture co-occurrence (which words show up together), not implication (what an action means for the criterion). "Speaking quietly" and "ignoring rude people" land in the cluster "behavior in a public situation," but the first is de-escalation (good) and the second is avoidance (bad for a criterion like "taking a stand").

Fix: A model with a rubric and examples works through reasoning: instead of grouping by topic, it interprets the meaning of the action for the criterion. "Walking over to a friend" = showing empathy (earns points), "walking away" = avoidance (costs points). That requires understanding the task context, not just word semantics.

Knob: Criterion-specific examples

Give every agent the same examples and they all calibrate the same way, losing what's specific to each criterion. The researchers picked examples for each criterion separately: low/mid/high on that criterion, not on the overall score. This matters: a pitch can be strong on "problem clarity" (7) and weak on "market understanding" (3). If you reuse that text as the 7-point example for "market understanding," calibration breaks.

How to apply it: If you have a pool of past scores, sort it by each criterion separately and pick percentiles. If you don't, generate synthetic examples with an LLM: "Create a sample pitch with weak market understanding (score 2), average (5), and strong (7). Explain what defines each level."

⚠️ Tokens and calls: The method needs N+1 calls for N criteria (1 preprocessing + N scoring agents). For 9 criteria that's 10 calls. Scoring 100 texts on 9 criteria with Llama 4 Maverick gets expensive. For 2-3 criteria on a small batch, it's fine. At industrial scale you'll need a budget or some API optimization.

⚠️ Picking examples: The method is sensitive to the quality of the calibration examples. If your low/mid/high examples don't represent the scale well (say, your "low" is actually a middling score), the model calibrates wrong. You need either an existing pool of scores or time to generate and validate synthetic examples. For brand-new criteria with no history, getting started is harder.

⚠️ Overkill for simple tasks: The method shines on complex, subjective criteria (empathy, creativity, ethics). For simple, checkable facts (grammar, text length, presence of keywords) it's overkill; one prompt will do the job faster. Multi-agent makes sense when the criteria are genuinely different and abstract.

⚠️ Fine-tuning may win with detailed rubrics: The research found that on the ASAP dataset (specific, detailed rubrics for each essay), a fine-tuned SOTA model beat multi-agent on some tasks. Multi-agent wins where rubrics are vague and generic (one set of criteria across different scenarios). If you have a clear, detailed rubric and a big training set, fine-tuning may be more effective.

The team took 1001 transcripts of virtual MMI interviews for medical school admissions. Candidates responded to 4 scenarios: comforting a grieving friend, handling professional incompetence, resolving a public ethical conflict, and settling a team dispute. Experts scored each answer on 9 soft-skill criteria (empathy, ethics, communication) on a 1-7 scale.

First they checked whether Sentence-BERT embeddings + clustering could tell good answers from bad ones. Result: a flop. The embeddings grouped phrases by topic ("moving toward a friend" landed in the same cluster as "walking away from a friend") but missed the evaluative meaning: for empathy those are opposite actions. Lesson: semantic similarity ≠ quality for subjective criteria.

Next they adapted a SOTA method from AES: Rationale-based Multiple Trait Scoring (RMTS). In the original, a large model generates rationales (justifications for scores), which are then used as features to fine-tune a small model. On the ASAP dataset (essays) this beats humans (QWK higher than human-to-human). On the MMI it failed: QWK 0.31 versus 0.62 for multi-agent.

Why? Rubrics. ASAP provides detailed, concrete criterion descriptions for each essay type: "Ideas must include X, Y, Z." The MMI uses one vague rubric for every scenario: "Empathy: the ability to understand another person's feelings." A rationale for "Ideas" is a checkable fact (X is there or it isn't). A rationale for "Empathy" is yet another subjective interpretation that adds noise instead of signal. The fine-tuned models showed high uncertainty: wide error spread, unstable predictions.

So they built the multi-agent framework: preprocessing (cleaning the transcripts) + 9 separate scoring agents with 3-shot calibration. Each agent got low/mid/high examples for its own criterion (not shared ones). They tested it on 4 SOTA models. Llama 4 Maverick: QWK 0.621, MSE 0.871, agreement on par with the experts.

To check transferability, they ran the same framework on the ASAP benchmark (school essays). They took two prompts: Q7 (7th grade, ethical dilemmas) and Q8 (10th grade, the meaning of laughter). Result: Q7 hit QWK 0.638 (slightly below the SOTA 0.749), Q8 hit QWK 0.663 (above the SOTA 0.658). And Q8 is harder: longer texts, more scale points. So multi-agent wins where the context is richer and the criteria are more abstract. No fine-tuning on ASAP, just the structure applied.

They ran an ablation: removing the 3-shot examples (zero-shot) dropped QWK from 0.663 to 0.503. Swapping ASAP's detailed rubrics for generic ones (MMI-style) also hurt. That confirmed it: calibration through examples is critical even for standard AES tasks.

Key insight: Fine-tuning thrives on checkable signals from detailed rubrics. Prompting thrives on flexible interpretation through reasoning. Where the rubric is vague and generic, structured prompting with task decomposition beats data-hungry fine-tuning. That changes how you should think about when to use which approach.

🔧 Technique: Single-pass grouping for correlated criteria

If your criteria are strongly correlated (say, "Grammar" and "Writing style"), you can group them in one prompt. Independence is critical when criteria are different (empathy vs logic). For similar ones, saving tokens is justified.

Modified prompt:

Rate the text on two related criteria:
1. Grammar (spelling, punctuation)
2. Writing style (readability, structure)

Use a 1-5 scale for each criterion.

[provide 3 examples where both criteria are scored]

Return: a Grammar score, a Style score, and a rationale.

You save calls but lose full independence. Use it when the criteria are logically linked.

🔧 Technique: Reverse calibration

If you don't have a pool of scored texts but do have texts, ask an LLM to generate the calibration examples.

Generation prompt:

Create 3 examples of {content type} with different levels of {criterion}:

1. LOW level (score 2/7): {what makes the example weak}
2. MEDIUM level (score 4-5/7): {what makes the example average}
3. HIGH level (score 7/7): {what makes the example strong}

Write realistic texts and show concrete differences in how the criterion shows up.

Use the generated examples in your scoring agents. This is synthetic calibration, and it works when you have no real data.

🔧 Technique: A meta-agent for consensus

If you want a second look at borderline cases, add a meta-agent after scoring:

Here are the scores on 9 criteria:
[paste the scores from all agents]

Check them for consistency:
- Are there contradictions? (high empathy + low communication = odd)
- Which scores look questionable?

If there are contradictions, suggest which specific criteria to re-score.

The meta-agent doesn't re-score; it flags suspicious patterns for human review. Useful for high-stakes decisions (hiring, admissions).

Automated Multiple Mini Interview (MMI) Scoring: Ryan Huynh, Frank Guerin, Alison Callwood (University of Surrey)

Key references: - ASAP dataset (Automated Student Assessment Prize): a public AES benchmark - RMTS framework (Rationale-based Multiple Trait Scoring): Chu et al., 2025 - ArTS model (Autoregressive Trait Scoring): Do et al., 2024 - Eva et al., 2004: research on the MMI as a selection method