Key Aspects of the Study:

The researchers asked: can a Large Language Model (LLM) judge the quality of text (using code commit messages as the test case) as well as an experienced human can? They found that an LLM not only can, but outperforms all existing automatic metrics, coming close to human-level evaluation quality.

Key Result: An LLM achieves its best results when the prompt combines two techniques: Chain-of-Thought (asking it to reason step by step) and Few-Shot (providing several examples of good and bad answers).

The Core of the Method, Explained:

The essence of the method is to turn the LLM from a simple "chat partner" into an "expert evaluator" that can analyze text by your rules. Instead of asking the LLM "is this good text?", you build a prompt that teaches the model exactly what you consider good text and forces it to justify its rating.

The technique rests on three pillars:

1. Task decomposition: Instead of an overall "good/bad" rating, you break it down into specific, clear criteria. In the study these are "What changed?" (What) and "Why did it change?" (Why). For evaluating a business email, they might be "Politeness," "Clarity of purpose," "Call to action."

2. Forcing it to reason (Chain-of-Thought, CoT): You directly order the model in the prompt: "First give your reasoning for each criterion, and only then assign the final score." This makes the LLM analyze the text methodically instead of guessing the answer, which dramatically improves the quality and objectivity of the evaluation.

3. Calibration through examples (Few-Shot Learning): You give the model 2-3 examples of what you consider "excellent" and "bad" text by your criteria, with a short explanation. It's like showing a new employee samples of good and poor work. The model "calibrates" to your expectations and starts rating new text against the benchmarks you provided.

Combining these three elements gets you not just a score from the LLM but a detailed, well-argued breakdown comparable in quality to an analysis from a real expert.

Practical Applicability:

• Direct applicability: Any user can immediately use this approach to create a "personal critic" or "editor." You can evaluate anything: work emails, social media posts, résumés, marketing copy, student essays. Just define the criteria, add the phrase "Think step by step" to the prompt and include a couple of examples. It's a ready-made template for hundreds of practical tasks.

• Conceptual value: The study offers three key insights:

  1. The LLM as a second-order tool: Its main strength lies not only in generating content but in analyzing and evaluating it.
  2. Process matters more than the result: By making the LLM show its reasoning process (CoT), we get a better final result.
  3. Context is everything: By providing examples (few-shot), we set the right context and quality standard, making the evaluation relevant to our task instead of abstract.

• Adaptation potential: The method is completely universal. Instead of a "commit message," you can plug in any type of text. The adaptation mechanism is simple:

  1. Define what you're evaluating (for example, "a customer support reply").
  2. Write 2-4 key evaluation criteria (for example, "empathy," "completeness of the answer," "speed of resolution").
  3. Find or write 1-2 examples of an "ideal" and a "bad" response.
  4. Put it all together in one prompt using the template from the example below.

A Practical Example:

You are an experienced HR specialist and career consultant. Your task is to evaluate a candidate's cover letter for a "Marketing Manager" position.

Rate the letter on a 10-point scale based on three key criteria:
1.  **Relevance:** How well the letter is tailored to this specific position and company.
2.  **Structure and clarity:** Is the letter easy to read, does it have a clear logic.
3.  **Motivation:** How convincingly the candidate demonstrates their interest and their value to the company.

**INSTRUCTIONS:**
First, carry out a step-by-step analysis (Chain-of-Thought). In the "Reasoning" section, break down the letter in detail on each of the three criteria. Only after that, in the "Result" section, output the scores in numeric format.

---
**Calibration examples (Few-Shot):**


"Hello. I saw your job posting on your website. I have 5 years of experience in marketing. My résumé is attached. Ready for an interview."
*   **Analysis of the bad example:** Relevance - 1/10 (generic, doesn't mention the company), Structure - 4/10 (too short), Motivation - 1/10 (not expressed).



"Dear [Company name] team, I've been following your projects with great interest, especially the launch of [Product name]. My 5 years of experience in B2B marketing, including successfully running campaigns with ROI >200%, will allow me to tackle the challenges listed in the marketing manager job description effectively. I'm confident my content strategy skills will help strengthen your market presence. More about my case studies in my résumé."
*   **Analysis of the good example:** Relevance - 9/10 (mentions the company and the product), Structure - 10/10 (clear and to the point), Motivation - 9/10 (shows value to the company).

---

**LETTER TO EVALUATE:**
"Good afternoon! I'm interested in your marketing manager position. I've worked in this field for a long time and believe my experience will be useful to you. I'm a strong team player and a fast learner. I would like to apply my knowledge at your company. Thank you."

Why It Works:

This prompt is effective because it doesn't just ask the LLM for a score; it builds a complete working environment for it: * Role ("HR specialist"): Gives the model the right context and a "professional lens" for the analysis. * Decomposition ("Relevance, Structure, Motivation"): Breaks a complex task into simple, clear, measurable parts. The model doesn't have to guess what a "good letter" is; it has clear criteria. * Chain-of-Thought ("First, carry out a step-by-step analysis"): This instruction makes the model spell out the logic behind its conclusions, which lowers the odds of random or superficial judgments and makes the result verifiable. * Few-Shot ("Calibration examples"): The bad and good letter examples work as benchmarks. They "show" the model what quality level corresponds to a "1/10" and what to a "9/10," calibrating its internal scale to your requirements.

Another Practical Example

You are an experienced social media manager who specializes in engaging Instagram content. Your task is to evaluate an idea for a post on a travel blog.

Rate the idea on a 10-point scale using three criteria:
1.  **Originality:** How fresh and non-clichéd the idea is.
2.  **Engagement potential:** How likely the post is to rack up likes, comments and saves.
3.  **Visual appeal:** How easy it is to imagine the photos or videos that could be shot for this post.

**WORKFLOW:**
1.  **Reasoning (Chain-of-Thought):** Write a short analysis of the idea on each of the three criteria. Point out strengths and weaknesses.
2.  **Recommendations:** Give 1-2 tips on how to improve this idea.
3.  **Result:** Output the final scores for each criterion.

---
**Examples to understand the task (Few-Shot):**


Idea: "A post about the Eiffel Tower."
*   **Analysis:** Originality - 2/10 (cliché), Engagement - 3/10 (too general), Visuals - 5/10 (standard photos).



Idea: "A post: 3 little-known spots in Paris with the best view of the Eiffel Tower and no tourist crowds. A side-by-side comparison of photos from each angle."
*   **Analysis:** Originality - 9/10 (useful, niche content), Engagement - 10/10 (encourages saves and comments), Visuals - 10/10 (a clear and interesting photo series).

---

**IDEA TO EVALUATE:**
"Make a post about where to eat well in Rome."

Why This Example Works, Explained.

The mechanism behind this prompt is identical to the previous one and rests on the same principles proven in the study: * Getting specific through role and criteria: The "social media manager" role and the "Originality, Engagement, Visuals" criteria turn the abstract request "rate this idea" into a concrete, professional task. The model knows what angle to look from and what to pay attention to. * Structured thinking (CoT): Requiring analysis first and the score second makes the model switch on its analytical abilities. It can't just say "the idea is good"; it has to justify that point by point, which deepens the analysis. * Setting the quality bar (Few-Shot): Comparing a cliché idea ("a post about the Eiffel Tower") with a creative one ("3 little-known photo spots") gives the model a clear sense of exactly what you value. It learns to tell superficial content from useful, engaging content and applies that standard to the new idea. As a result, the evaluation is no longer generic but focused on creating effective social media content.

Usefulness Score: 95