Key Aspects of the Study:
The study examines how well LLMs can act as a "judge" that evaluates and compares two candidate answers (in this case, code snippets). It found that models are prone to "positional bias" (the verdict depends on the order of the options), and that comparing two options directly in one prompt (pairwise comparison) works significantly better than scoring each one separately. It also turned out that models need full context, not just "clean" data, to judge more accurately.
Key Result: For reliable evaluation, ask the LLM to compare two options directly in a single request, but keep the possible effect of their order in mind and provide as much supporting information as you can.
The Core of the Method, Explained:
The core of the method you can take from this study is using an LLM as an "expert evaluator" (LLM-as-a-Judge) with a special prompt structure that makes the model's judgments more reliable and accurate.
For users, the playbook boils down to three main principles:
Use pairwise comparison (Pair-wise Comparison): Instead of asking the LLM to rate one option on some scale (e.g., "Rate this text from 1 to 10") and then the other, it's much more effective to put both options in a single prompt and ask the model to compare them and pick the better one. This approach makes the LLM engage its comparative-analysis machinery, which leads to a more nuanced, well-reasoned decision. It's easier for a model to tell which one is better than to say how good something is in absolute terms.
Remember positional bias (Positional Bias) and test for it: The study proves that LLMs often prefer whichever option comes last in the prompt (recency bias). To avoid falling victim to this cognitive bias in the model, use a simple trick: if you're not sure the answer is objective, swap the options and ask the same question again. If the model still picks the same option (regardless of its new position), its choice is probably objective. If it picks the other option (the one now in last place), its judgment is unreliable.
Practical Applicability Analysis:
Direct applicability: Extremely high. Anyone can start building "Compare A and B" prompts right away. For example: "Here are two headline options for an article. Which one is better and why? Option A: [...]. Option B: [...]". The positional-bias check (swapping places) doesn't require any special skills either.
Conceptual value: Huge. The study gives users an intuitive understanding that an LLM isn't a knowledge base but a reasoning tool that depends heavily on how the task is "framed". Key concepts:
- LLMs handle comparative tasks better than absolute ones.
- LLMs are prone to cognitive biases similar to human ones (such as order effects).
- Context beats brevity when it comes to the quality of judgments.
Adaptation potential: The "LLM-as-a-judge" method with pairwise comparison is universal. It can be adapted to any field:
- Marketing: Picking the best ad.
- Law: Comparing two versions of a contract clause.
- Education: Evaluating two answers to an exam question.
- Personal productivity: Choosing between two plans for the day.
Adapting it is simple: define two options to compare, set clear evaluation criteria and ask the model to make a choice and justify it.
A Practical Example:
You are an experienced HR specialist and copywriter. Your task is to help me pick the best version of a cover letter for a "Product Manager" job application.
**# Context:**
* **Company:** An innovative edtech startup.
* **My experience:** 5 years in marketing, managed small projects for the last 2 years, strong analytical skills, launched two successful ad campaigns.
* **Goal of the letter:** Make a strong impression, show motivation and the relevance of my experience, stand out from other candidates.
**# Task:**
Analyze the two cover letter versions below. Compare them on the following criteria:
1. **Persuasiveness:** How well the text "sells" my experience for this role.
2. **Structure and clarity:** How easy it is to read and grasp the main point.
3. **Tone of Voice:** Whether the writing style fits a startup culture.
Make a final choice and explain in detail why one version is better than the other.
---
**VERSION A**
Dear Hiring Team,
I am writing to express my interest in the Product Manager position posted on your website. With five years of experience in marketing and project management, I am confident that my skills are a good fit for your company. I have successfully launched several campaigns and have strong analytical abilities. I would welcome the opportunity to discuss how my experience could benefit your team.
Sincerely,
[My Name]
---
**VERSION B**
Dear [Company Name] team,
I'm inspired by your mission to transform education, and I want to be part of that story as a Product Manager. My move from marketing into project management was driven by a desire not just to promote products but to build them. Over the past two years I've led project teams and learned firsthand how to turn analytics into a successful product — for example, during the launch of campaign X, which increased engagement by 30%.
I'm confident that my experience in understanding users and managing the project lifecycle will help you reach new heights. I'd love to discuss how I can strengthen your product team.
Best regards,
[My Name]
---Why It Works:
This prompt works because it directly applies the study's findings:
- Pairwise comparison (Pair-wise Comparison): The prompt doesn't ask for each letter to be rated on a scale; it makes the model directly compare "Version A" and "Version B". That triggers deeper analysis and surfaces nuances, since the model looks for relative strengths and weaknesses.
- Rich context (Rich Context): Instead of just handing over two texts, the prompt provides key information (
Context: about the company, my experience, the goal). That's the equivalent of the "full response with comments" from the paper. This context gives the LLM a "yardstick" for judging the relevance of each letter more objectively. - Clear evaluation criteria: Listing the criteria ("Persuasiveness", "Structure", "Tone of Voice") guides the LLM's "judging" process, making it less random and more focused, which makes the answer more reliable and stable.
- 4. Awareness of positional bias: Although it isn't visible in the prompt itself, a user who knows the study knows that for 100% confidence they can swap Version A and Version B and run the prompt again to check whether the model's choice stays the same.
Another Practical Example
You are a travel expert who specializes in family vacations.
**# Context:**
* **Travelers:** A family with two kids (ages 6 and 11).
* **Interests:** The kids love animals and interactive museums. The parents love beautiful scenery and relaxed walks. Nobody likes big crowds or hustle and bustle.
* **Budget:** Mid-range.
* **Task:** Choose the best destination for a one-week vacation in August.
**# Task:**
Below are two vacation options. Compare them on these criteria:
1. **Kids' interests:** How well the destination suits kids aged 6 and 11.
2. **Adults' interests:** Is there something for the parents (nature, peace and quiet).
3. **Logistics and budget:** How easy it is to get around and whether it fits a mid-range budget.
At the end, give a clear recommendation for which option you think is best for this family, with a detailed justification.
---
**DESTINATION 1: Montenegro (the Budva and Kotor coast)**
* **Activities:** Beach time, strolling the old towns of Budva and Kotor, a boat trip around the Bay of Kotor, a visit to Lovćen National Park.
* **Pros:** Beautiful sea, mountains, historic atmosphere.
* **Cons:** Very crowded with tourists in August, packed beaches, can be noisy.
---
**DESTINATION 2: Slovenia (Lakes Bled and Bohinj)**
* **Activities:** Walks around the lakes, a visit to Bled Castle, boating, a hike through Vintgar Gorge, a visit to an interactive museum in Ljubljana (1 hour's drive). There's a small zoo nearby.
* **Pros:** Stunning nature, less hustle, lots of outdoor activities.
* **Cons:** The sea is far away, the weather can be changeable.
---Why This Example Works, Mechanically.
This prompt uses the same proven methodology to get high-quality, reliable advice:
- LLM-as-a-Judge: The model gets the role of a "travel expert", which puts it in an evaluative frame of mind.
- Pairwise comparison: Instead of the abstract question "Where should we go with the kids?", the prompt sets a concrete task of choosing between two clearly described options ("Montenegro" vs "Slovenia"). That makes the model weigh the specific pros and cons of each option against each other instead of generating a generic list of ideas.
- Context and criteria: Spelling out the family makeup, their interests and budget gives a solid basis for comparison. The criteria ("Kids' interests", "Adults' interests", "Logistics") structure the model's analysis, making it look at each aspect for both options in turn, which improves the quality and soundness of the final recommendation.
- Lower risk of bias: Knowing about "positional bias", the user can easily swap "Destination 1" and "Destination 2" to make sure the model's recommendation isn't random but based on analysis of the facts provided.
Usefulness Score: 93
Main Evaluation Criteria
- A. Relevance to prompting techniques: Yes, the study directly compares the effectiveness of different prompting approaches for evaluation tasks (pairwise comparison vs. scale ratings) and analyzes the effect of prompt structure (with or without comments and reasoning).
- B. Better conversational answers: Yes, the principles the study uncovers lead directly to more reliable and consistent LLM answers on tasks that involve evaluating or choosing between options.
- C. Direct practical applicability: Absolutely. Users can apply the findings immediately without any code. The core ideas, "compare two options instead of rating them one at a time" and "watch out for order effects", are trivial to implement.
- D. Conceptual value: Very high. The study clearly demonstrates fundamental behavioral traits of LLMs: positional bias and the superiority of comparative thinking over absolute scoring. It helps users understand that an LLM isn't an objective logic calculator but a system that's sensitive to how information is presented.
- E. New useful practice (clustering): The paper fits several key clusters at once:
- 1. Prompt phrasing techniques: Introduces and proves the effectiveness of the "pairwise comparison" (pair-wise comparison) technique as part of the "LLM-as-a-Judge" pattern.
- 2. LLM behavioral patterns: The key discovery is positional bias. Models tend to prefer the option in second place (recency bias), which is critical knowledge for any user.
- 3. Prompt structure optimization: The finding that full, unprocessed responses with comments and reasoning (raw response) work better than cleaned-up code speaks directly to the importance of providing rich context.
- 7. Reliability and stability: The entire study is about making evaluations more reliable. It shows how to reduce randomness and bias in LLM judgments.
- Practicality checklist (+15 points): Yes, the study provides ready-made constructions (pairwise comparison), explains a problem with information placement (positional bias), shows how to structure complex evaluation requests and reveals non-obvious quirks of LLM behavior.
Numeric Usefulness Score
The study earns a high score because its findings, despite the narrow domain (code evaluation), are universal and extremely useful for anyone who uses LLMs for choosing, comparing or evaluating.
Arguments in favor of the score: * Universal findings: The discovery of "positional bias" (when the model prefers the last option offered) and the proven effectiveness of "pairwise comparison" apply to any task: from picking the best marketing slogan to comparing two travel plans. * Direct action: Users can improve their prompts right away just by changing their structure: instead of "Rate option A. Now rate option B", use "Compare option A and option B and pick the better one". * A conceptual breakthrough for users: The paper shatters the naive view of the LLM as an objective expert and teaches users to treat the model as a powerful but biased tool that needs to be properly "calibrated" with prompts.
Counterarguments (why the score could be lower): * A specialized domain: All the terminology and examples in the paper relate to programming. An untrained user may find it hard to "translate" the findings from the "code evaluation" context into their everyday field on their own (e.g., writing or data analysis). * No ready-made "magic phrases": The study offers a methodology and points out behavioral traps rather than giving specific phrases like "Think step by step" that you can just copy. Applying it takes a little thought.