Key Aspects of the Research:

The study proves that LLMs often make judgments based not on the substance of the items being compared but on their position in the prompt (favoring the first or the last option). This "position bias" is unpredictable and can change from one request to the next. The fix it proposes is simple and effective: ask for the evaluation several times, changing the order of the items being compared, to get a more objective result.

Key Result: Simply reordering the candidates in the prompt and asking again is an effective way to expose and neutralize the model's bias, significantly improving the reliability of its judgments.

The Core of the Method, Explained:

For practical use, the method boils down to one simple principle: "Swap and check".

The study shows that LLMs are not perfectly objective logic machines. They're prone to cognitive biases similar to human ones, in particular position bias. That means if you ask a model "Which is better, Option A or Option B?", its answer may depend on the fact that A comes first. This is called the "primacy effect" (primacy bias). In other cases the model may, on the contrary, better remember and prefer the last item: the "recency effect" (recency bias).

The most important finding for users is that this bias is unstable. On one request the model may prefer the first option, and on another, very similar one, the last. So you can't just decide to "always put the important stuff first".

The practical technique that follows from the study looks like this: 1. Write a comparison request for two (or more) items. 2. Get the model's answer. 3. Don't trust it blindly. Make an exact copy of your prompt but swap the items being compared. 4. Get the answer again. 5. Compare the results: * If the answer didn't change (the model picked the same item even though it's now in a different position), you can be much more confident the answer is objective. * If the answer changed (the model picked whichever item landed in the "winning" position), that's a clear sign of position bias. You can't trust that judgment; either rephrase the prompt or accept that, for the model, these options are very close.

This method turns a one-off request into a simple reliability test that any user can run.

Practical Applicability Analysis:

• Direct applicability: Maximum. Any user who asks an LLM to compare anything (two products, two ideas, two resumes, two candidate answers) can use this method right away. It's just copying the prompt and swapping the parts being compared. No technical knowledge required.

• Conceptual value: Huge. The study gives users a key insight: an LLM is not an objective oracle. Its answers depend on subtle factors like word order. This builds a healthy dose of skepticism and teaches users to "test" and "verify" the model's answers rather than take them on faith. The concept of "position bias" becomes part of the user's mental model of how an LLM "thinks".

• Adaptation potential: The method needs no adaptation, since its basic version ("swap and check") is already as simple and ready-to-use as it gets. It scales easily from comparing two items to picking the best from a short list (through several pairwise comparisons or by reordering the whole list).

Practical Example:

You are an experienced marketer and copywriter. Your job is to help me choose the best slogan for a new line of organic health foods.

**Context:**
The products are all-natural, sugar-free and aimed at a young audience (25-40) who care about their health and value quality ingredients.

**Task:**
Analyze the two slogan options below. Rate them on the following criteria:
1.  Memorability.
2.  Emotional impact.
3.  Fit with the brand's values (natural, healthy).

Pick the single strongest option and explain in detail why you made that choice.

---
**Options to compare:**


Pure energy from nature.



Your tasty step toward health.

---

Applying the method: After getting the answer to this prompt, the user creates a second request with the options swapped:

... (the rest of the prompt text stays the same) ...

---
**Options to compare:**


Your tasty step toward health.



Pure energy from nature.

---

Why It Works:

This prompt works through isolating and swapping the items being compared.

1. Isolation: Using the XML tags <option_A> and <option_B> helps the model clearly separate the two objects being compared, minimizing confusion.
2. Swapping (the mechanic from the study): The core mechanism is running a second prompt with the slogans swapped. It's a direct test for position bias.
   • If in both cases the model picks, say, "Pure energy from nature", it considers that slogan objectively better by the given criteria, regardless of its position.
   • If in the first prompt it picked "Pure energy from nature" (option A) and in the second "Your tasty step toward health" (which is now option A), that screams that the model is subject to the primacy effect and its judgment can't be trusted here.

So the user doesn't just get an answer; they run a mini-experiment to check how reliable it is.

Another Practical Example

You are a career coach with 15 years of experience. You help people make well-considered decisions about changing jobs.

**My situation:**
I've been working as a frontend developer for 3 years. I've received two job offers. I can't decide which one is better for my long-term career growth.

**Task:**
Compare the two job offers described below. Analyze their strengths and weaknesses in terms of career growth, work-life balance and earning potential.

Give me a well-reasoned recommendation on which offer looks more promising, and explain why.

---
**Offers to compare:**


**Company:** A large, well-known tech corporation.
**Role:** Middle Frontend Developer.
**Work:** A mature, stable product with millions of users.
**Pros:** Stability, a strong brand on your resume, clear processes, good benefits.
**Cons:** Slow growth, lots of bureaucracy, less influence on the product.



**Company:** A fast-growing startup (Series B).
**Role:** Senior Frontend Developer (first on the team).
**Work:** Building the product from scratch, shaping the frontend architecture.
**Pros:** Fast track to team lead, big responsibility and influence, stock options.
**Cons:** Risk of the company failing, possible overtime, chaotic processes.

---

Why This Example Works: The Mechanism

The mechanism here is exactly the same, but the stakes are higher, since this is a major life decision.

1. Structuring: The information for each offer is clearly structured with subheadings and the tags <offer_1> and <offer_2>. This helps the model process the context properly.
2. Bias check: The key mechanic is rerunning the prompt with <offer_1> and <offer_2> swapped. A decision about changing jobs shouldn't depend on which offer you happened to mention first.
   • If the model recommends the startup both times, citing growth potential, that's a strong signal in favor of that choice.
   • But if the first time it recommends the corporation (Offer 1) for stability, and the second time the startup (which is now in first place), that means the model has no clear "opinion" and is subject to position bias. In that case its recommendation is useless, and the user needs to decide based on their own risk assessment rather than the LLM's "advice".

This approach lets you use the LLM not as a "decider" but as a "mirror" that helps you structure your thinking, but whose reliability you have to verify.

Usefulness Score: 90