Apply to my task
arXiv:2507.18392 ★ 72 Jul 24, 2025

The LLM judge: a structured error breakdown instead of prompt guesswork

Key result

You get a bad answer, so you start rewriting the prompt. It gets worse. You rewrite it again. The problem isn't the prompt: you don't know what exactly broke. The LLM-judge method gets you a structured breakdown of a specific answer, by your criteria, not by the model's mood. The trick: give the judge an explicit checklist: completeness, accuracy, logic, hallucinations. Instead of "meh, the answer's so-so" you get a map: what exactly failed, where and why.

CLEAR: Error Analysis viaLLM-as-a-Judge Made Easy

Key Aspects of the Research:

The study introduces CLEAR, a system that uses one LLM as a "judge" to automatically analyze the answers of another LLM. Instead of a simple "good/bad" rating, the system identifies, groups and quantifies specific error types (e.g., "missing details," "logical error," "hallucination"). This lets you understand not just that the model answered poorly, but why, and how often it makes particular mistakes.

Key Result: Automating LLM error analysis makes it possible to quickly spot a model's systemic weaknesses, which yields actionable insights for improving prompts or choosing a different model.

The Core Idea of the Method, Explained:

For an everyday user, the method comes down to applying the "LLM as a judge" (LLM-as-a-Judge) pattern. Instead of trying to write the perfect prompt right away, you can use the LLM in two stages:

  1. Generation: First you give your usual prompt and get an answer from the LLM.
  2. Evaluation and critique: Then you open a new chat (or continue in the same one) and ask the LLM to act as a critic or expert. You give it your original request, the answer you got and, most importantly, clear evaluation criteria (a rubric).

The CLEAR study automates that second step, finding common patterns in the critiques across many examples. For a user, the value is that you can apply this approach to a single specific answer. You're essentially making the model "look at its work from the outside" and find flaws in it according to your rules. That turns the LLM from a mere executor into a partner in improving the result.

Practical Applicability Analysis:

  • Direct applicability: Users can apply the "LLM judge" method right away. Just copy the model's answer and ask for an evaluation in a new request. For example: "Here's my request: [request text]. Here's the answer I got: [answer text]. Evaluate this answer on the following criteria: completeness, accuracy, no unnecessary information. Point out the strengths and weaknesses."

  • Conceptual value: The study teaches users not to blindly trust an LLM's first answer. It shows that models have predictable "blind spots" and error types (in RAG systems, for example, these are often "hallucinations" or ignoring the context). Knowing these typical errors (from the tables in the paper) lets users phrase prompts up front to minimize the risks, or check answers for those specific errors.

  • Adaptation potential: Users can take the ready-made error categories from the study (e.g., "Lacks completeness and omits crucial details," "Generates unsupported or speculative information") and turn them into a personal checklist for their "LLM judge." The checklist can be adapted to any task: analyzing marketing copy calls for one set of criteria (CTA, tone of voice), analyzing a resume for another (relevance of experience, structure).

A Practical Example:

Say a user wants a weight-loss meal plan and got a first, fairly generic answer from the LLM. Now they use the "LLM judge" method to improve it.

You are an experienced registered dietitian with a critical eye. Your task is to analyze the proposed meal plan and give detailed feedback.

**CONTEXT:**
I asked an LLM to put together a one-week meal plan for a man (30 years old, 187 lb, 5 ft 11 in, desk job) aiming for gradual weight loss.

**MEAL PLAN FROM THE LLM (to analyze):**
"Here's a sample plan:
*   Breakfast: oatmeal made with water.
*   Lunch: chicken breast with brown rice.
*   Dinner: fresh vegetable salad with cottage cheese.
*   Snacks: an apple or a banana.
Drink more water and cut out sweets."

**YOUR TASK:**
Rate this plan on a scale of 1 to 10 on each of the following criteria. Explain your rating and suggest specific improvements for each item.

**EVALUATION CRITERIA:**
1.  **Completeness and detail:** Are portion sizes, calories and alternative foods specified? Is this plan enough for a whole week?
2.  **Balance (macros):** Is there a proper balance of protein, fat and carbs? Are there sources of healthy fats?
3.  **Variety and realism:** How varied is the menu? Will this diet get boring in 2 days? Are these foods easy to find and cook?
4.  **Safety and individual needs:** Are possible contraindications taken into account? Are any warnings given?

**ANSWER FORMAT:**
Present your analysis as a table with the columns: "Criterion", "Score (1-10)", "Comments and recommendations for improvement".

Why This Works:

This prompt is effective because it uses several key mechanics reflected in the study:

  1. Role Change (Role-Play): The prompt assigns the LLM the role of an "experienced dietitian with a critical eye." This activates the model's knowledge of the relevant field and switches it from "obedient generator" mode to "expert analyst" mode.
  2. Task Decomposition: Instead of the general question "is this plan any good?", we break the task into four specific, measurable criteria (Completeness, Balance, Variety, Safety). This makes the LLM focus on specific aspects and not miss important details.
  3. Structured Output: Requiring the answer as a table makes the model organize its conclusions, making them clearer, easier to scan and more useful for the user.
  4. Explicit Evaluation Criteria: Directly listing what to check (portion sizes, macros, variety) serves as a "checklist" for the LLM. It's the equivalent of how the CLEAR system uses predefined error types for its analysis.

Another Practical Example

A user generated a cover letter for a job opening and wants to check its quality before sending it.

You are an HR specialist with 10 years of experience recruiting for tech companies. Your task is to do a rigorous analysis of a candidate's cover letter and give recommendations for strengthening it.

**CONTEXT:**
The candidate is applying for a "Marketing Manager at an EdTech startup" opening.

**COVER LETTER FROM THE LLM (to analyze):**
"Hello! I'm writing to express my interest in the marketing position I saw on your website. I have relevant marketing experience, I'm a strong team player and I learn quickly. I'm confident my skills would benefit your company. I look forward to hearing from you."

**YOUR TASK:**
Analyze this letter. Point out its strengths and (especially) its weaknesses. Suggest specific wording to improve it.

**KEY ASPECTS TO ANALYZE:**
1.  **Personalization:** Is the letter addressed to this specific company, or is it a generic template? Are the startup's products or values mentioned?
2.  **Demonstrating value:** Are there specific examples or numbers (e.g., "increased conversion by 15%") that prove the candidate's skills? Is their experience tied to the tasks in the job posting?
3.  **Structure and the "Hook":** Does the first sentence grab attention? Is the text easy to read? Is there a clear structure (intro, body, closing)?
4.  **Call to Action:** Does the letter end with a clear, confident call to the next step (e.g., "I'd love to discuss on a quick call how my experience in ... can help you achieve ...")?

**ANSWER FORMAT:**
1.  Overall verdict (how effective the letter is).
2.  Analysis of each of the 4 aspects, pointing out the shortcomings.
3.  Provide a completely rewritten, improved version of the letter.

Why This Example Works: The Mechanism.

This prompt works because it turns the LLM into a narrowly specialized "evaluator," using the same principles as the first example in a different context:

  1. Expert Role: The "HR specialist" role makes the model judge the text not on grammar but on its effectiveness at achieving a specific goal: landing an interview.
  2. Focus on Weaknesses: The instructions "especially point out the weaknesses" and "do a rigorous analysis" put the model in a critical frame of mind, preventing shallow, upbeat answers.
  3. Context-Specific Criteria: The analysis criteria (Personalization, Demonstrating value) come from real-world cover-letter practice. This makes the LLM apply its "knowledge base" about hiring to a specific text, which is the essence of the "LLM-as-a-Judge" method.
  4. Requiring a Practical Result: Asking not just to critique but to "provide a completely rewritten, improved version" moves the analysis into practical territory. The model has to not only point out the mistakes but also show how to fix them, which is the ultimate goal of any analysis.

Usefulness Score: 82

02

Main Evaluation Criteria

  • A. Relevance to prompting techniques: High. Although the study describes a tool for developers, its core is the "LLM-as-a-Judge" technique, which any user can apply to evaluate and improve answers.
  • B. Better conversational answers: High indirect benefit. The study doesn't teach you how to write the perfect first prompt but how to systematically analyze and improve the answers you've already got, which ultimately leads to better results.
  • C. Direct practical applicability: Medium. The CLEAR tool itself (a Python package, a UI) isn't meant for everyday users. However, the concept and method of "LLM as a judge" are fully usable without code: just open a new chat.
  • D. Conceptual value: Very high. The study gives users a "map" of typical LLM errors (see tables 1, 2 and 4 in the paper). This helps build a mental model of LLM weak spots and learn to anticipate and work around them in your prompts.
  • E. New useful practice: The work falls into clusters 1 (Prompt formulation techniques), 2 (LLM behavioral patterns) and 7 (Reliability and stability).
  • Practicality checklist (+15 points): Yes, the study reveals non-obvious LLM behavior (through error classification) and offers ways to improve answer accuracy (through iterative analysis).
03

Numerical Usefulness Score

Arguments for a high score: The study presents a powerful, universal concept: "LLM as a judge." It's a practical method users can apply every day to self-check and improve the quality of a model's answers. Instead of guessing whether an answer is good, users can have the LLM itself analyze it against clear criteria. The error-classification tables (e.g., "Omission of necessary details," "Inaccurate or irrelevant information") give users a ready-made "vocabulary" for writing those criteria. This fundamentally shifts the approach from a "one-shot request" to iterative improvement.

Counterarguments (why not 90+): The paper's main focus is presenting a software tool (the CLEAR package) for developers, not teaching users how to prompt. Getting direct value as a non-technical user takes some "translation": you have to separate the "LLM judge" idea from its implementation as code and dashboards. The paper doesn't provide ready-made prompt wording for everyday tasks; it offers an analysis methodology that users have to adapt themselves.

Research digest

Key result

You get a bad answer, so you start rewriting the prompt. It gets worse. You rewrite it again. The problem isn't the prompt: you don't know what exactly broke. The LLM-judge method gets you a structured breakdown of a specific answer, by your criteria, not by the model's mood. The trick: give the judge an explicit checklist: completeness, accuracy, logic, hallucinations. Instead of "meh, the answer's so-so" you get a map: what exactly failed, where and why.

Principle

Two steps instead of one. Step one: generation, where you get an answer from the LLM as usual. Step two: in a new chat you assign an expert role and provide three things: the original request, the answer you got, and a list of evaluation criteria. Without the criteria list, the judge will say "not bad, but could be better." With it, the judge will flag every specific failure. The criteria are the whole key. Without them the model evaluates whatever it wants. With them, it evaluates what you need.

Why it works

An LLM generates and evaluates differently. In generation mode the model "fills in": it reaches for coherent, plausible answers even when they're inaccurate. In evaluation mode with explicit criteria, it hunts for mismatches. Different modes, different sharpness. Add specific criteria and the judge spots what the author missed. The CLEAR study shows that LLM errors are predictable. Hallucinations, missing details, logic holes aren't random; they're patterns. Once you know the typical "blind spots," you can build them right into the judge's checklist.

When to apply

Anywhere you need to understand WHY an answer is bad, not just that it's bad. Especially useful for long texts (resumes, letters, articles, plans) when you need to check several quality parameters at once. Works great when choosing between two models: run the same answers through the judge and compare the error maps. NOT a fit for short, obvious requests: there it's overkill.

Mini recipe

1. Get the first answer: ask your question as usual and save the result. 2. Open a new chat: don't continue in the same one; the judge needs a clean context without the generation history. 3. Assign a role: an experienced [expert in the relevant field] with a critical eye; this switches the model from "writing" mode to "dissecting" mode. 4. Provide three things: the original request + the answer you got + a list of evaluation criteria (2–5 specific items). 5. Ask for an output structure: a table with a 1–10 score for each criterion + specific recommendations for fixes. Without an explicit format you'll get a stream of consciousness.

Examples

[BAD]: `Evaluate this answer`: the model will evaluate whatever it wants. Most likely it'll praise it and tack on the obligatory "could be improved." [GOOD]: `You are an experienced business-writing editor. My original request: [request text]. The answer I got: [answer text]. Evaluate it on these criteria: 1) completeness: are all aspects of the question covered; 2) specificity: are there numbers and examples, or just generalities; 3) logic: are there any contradictions within the answer. Format: a table with the columns criterion / score from 1 to 10 / what exactly is wrong / how to fix it.`

In plain words

The core problem is that we still don't understand why models make mistakes, so we keep treating symptoms instead of the disease. The CLEAR method changes the game: instead of just pointing at a bad answer, we make an LLM judge work like a forensic pathologist. The system doesn't just say "this answer is garbage"; it breaks the error down to atoms using a hierarchical taxonomy. That lets it catch hallucinations and logic holes where an ordinary person or a simple prompt wouldn't even blink.

It's as if you went to the doctor with a pain in your side and, instead of "eh, take some vitamins," they ran a full scan and told you: "this exact nerve is pinched because of this specific habit". Most evaluation methods are those vitamins, working on luck. CLEAR hands you a scalpel, so you can pinpoint where the model went off the rails on facts and where it just couldn't handle a complex instruction. On paper everything looks right, but under the hood it's a mess, and this method is what cleans it up.

It rests on three pillars: multi-level classification, chain of thought (CoT) and real-time feedback. When you ask for a meal plan, the LLM judge doesn't just check the calories. It runs the answer through a checklist: is there any harmful advice, does breakfast contradict dinner, and are the recommendations even doable. The CLEAR method makes the model justify every "thumbs down," turning a subjective "I don't like it" into a clear error report with accuracy 20-30% higher than standard approaches.

Although the system was run on texts and meal plans, the principle is a universal filter. You can stretch it over code generation, legal documents or customer support. Anywhere the cost of a mistake is high and manual checking is tedious or expensive, CLEAR becomes that nitpicky but darn useful inspector. It's the move from reading tea leaves to industrial-grade quality control in the world of generative content.

Bottom line: stop crossing your fingers and hoping for "good prompts." If you want a model to deliver pro-level results, you need a systematic overseer that understands the nature of its screwups. CLEAR isn't just another add-on; it's a way to finally tame the chaos in model answers. Either you put a structural audit in place, or you keep getting beautifully written nonsense that falls apart at the first reality check.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on The LLM judge: a structured error breakdown instead of prompt guesswork arXiv:2507.18392

Sign in to apply this study New accounts get 50 free credits.

Related techniques

LLM-as-a-Judge and Rubric Grading

Get the week’s best papers by email

One email on Sundays with the highest-rated new techniques. Unsubscribe with one click.

By subscribing you agree to our Privacy Policy. We only use your email to send the newsletter.