Key Aspects of the Study:

The researchers set out to learn whether LLMs can generate and, more importantly, assess the quality of a specific type of text ("user stories"). They found that LLMs evaluate almost at a human level, but only on one condition: the prompt must contain very clear, detailed instructions with criteria and examples, a so-called "codebook." They also found that LLMs generate less diverse and less creative texts than people, and are especially bad at explaining reasons and rationale ("why this is needed").

Key Result: Giving an LLM a detailed "codebook" with evaluation criteria and examples is the decisive factor in getting high-quality, reliable assessments comparable to human ones.

The Core Idea of the Method:

The method the study proves effective is about moving from simple, vague requests to structured prompts with explicit evaluation criteria. Instead of asking the LLM to "Evaluate this text," the user should act like a client handing over a clear spec for the evaluation.

This approach, called a "codebook-augmented prompt" in the paper, means in practice that an everyday user includes three key components in their request:

1. Define the criteria: Spell out exactly which parameters the text should be judged on. For example: "Evaluate on these criteria: 1. Clarity, 2. Persuasiveness of the arguments, 3. Originality of the idea."
2. Rating scale: Set a clear scale. For example: "Rate each criterion on a scale from 1 (poor) to 3 (excellent)."
3. Calibration examples (the most important part): Give the model short examples of what you consider a good or bad result for each criterion. For example: "For 'Clarity,' a 3 means 'The text reads in one breath,' and a 1 means 'You have to reread a sentence twice to get it.'"

This method turns the LLM from a subjective "opinion" into an objective evaluation tool that works by your rules. The study also highlights an important trait: LLMs are great at evaluating structure and form (LC, IC) but struggle to evaluate depth of meaning and rationale (RC). That means when writing the prompt, you need to put special emphasis on explaining the "why," or not expect the model to deeply understand hidden motives.

Practical Applicability Analysis:

• Direct applicability: Users can immediately apply this approach to any task that involves critique, evaluation or generating text to a template. For example, you can give the LLM your own "codebook" for evaluating business emails, blog posts, résumés or even startup ideas. The method is implemented by simply adding structured text to a regular prompt.

• Conceptual value: The study gives the user two key concepts:

  1. The LLM as an "evaluator by instruction": To get a quality assessment from the model, you first have to teach it your standards by providing a clear set of rules and examples. The LLM doesn't guess your expectations; it follows instructions.
  2. Knowing the "blind spots": Models see the "what" (structure, grammar) very well but poorly understand the "why" (rationale, purpose). Knowing this, the user can either avoid demanding the impossible from the LLM or deliberately reinforce the prompt with an emphasis on explaining purpose and logic.

• Adaptation potential: The adaptation mechanism couldn't be simpler. For any task, mentally break the idea of a "good result" into its components (criteria). Then for each criterion, describe what "good" and "bad" look like, and package it in the prompt as a list of rules and examples before the main task. This framework works for any domain.

Practical Example:

# ROLE
You are an experienced editor at the popular food blog "Taste of Life." Your audience is cooking beginners looking for simple, easy-to-follow recipes.

# TASK
Rate the draft recipe submitted by an author on a 3-point scale according to my "Editor's Codebook." Your goal is to give the author constructive feedback to improve the text.

# EDITOR'S CODEBOOK

**1. Beginner-Friendliness (BF):**
*   **3 (Excellent):** All steps are described simply, with no complicated jargon. Anyone cooking for the first time will understand what to do. Example: "Finely chop the onion" instead of "Sweat the shallots."
*   **2 (Acceptable):** Mostly clear, but there are 1-2 terms that could confuse a beginner (for example, "blanch").
*   **1 (Poor):** The recipe is written for professionals, with lots of complex terms and techniques and no explanation.

**2. Practical Value (PV):**
*   **3 (Excellent):** Exact measurements and cooking times are given, plus useful tips (for example, "if you don't have parchment paper, grease the baking sheet with butter").
*   **2 (Acceptable):** The basic data is there, but useful details and alternatives are missing.
*   **1 (Poor):** Measurements are "eyeball it," no times given, no tips.

**3. Engaging Style (ES):**
*   **3 (Excellent):** The text is lively, makes you hungry, and includes a short backstory or an interesting fact about the dish. Example: "This recipe was handed down to me by my grandmother, and its aroma instantly takes me back to childhood."
*   **2 (Acceptable):** The style is dry but functional. Just a list of steps.
*   **1 (Poor):** The text is boring and written in bureaucratic language.

# TEXT TO EVALUATE
"Take flour, eggs, sugar. Mix the ingredients until smooth. Bake in the oven. Serve."

# RESPONSE FORMAT
Give a rating for each criterion (BF, PV, ES) and a short but clear justification for each one explaining why it received that score.

Why It Works:

This prompt is effective because it leaves the LLM no room for guesswork or subjectivity. It works thanks to the following mechanics described in the study:

1. Clear criteria (BF, PV, ES): The model doesn't have to invent what a "good recipe" means on its own. It's given three specific axes of analysis: clarity, practicality and style.
2. A defined scale (1 to 3): This standardizes the evaluation and makes it measurable.
3. Concrete examples (Example: ...): This is the most important part. Examples "calibrate" the model by showing it in practice what the user means by "excellent" or "poor." The model learns from these examples and applies the pattern to a new text.
4. A set role and context: Specifying the role ("food blog editor") and audience ("beginners") helps the model better understand the overall goal and tone of the evaluation.

Instead of an abstract "make this recipe better," we hand the model a precise tool (a rubric) it can use to carry out a detailed, useful analysis.

Another Practical Example

# ROLE
You are an HR specialist who helps job seekers improve their cover letters. Your job is to give honest but motivating feedback.

# TASK
Rate a cover letter for a "Marketing Specialist" position on a 3-point scale, using my "Cover Letter Codebook."

# COVER LETTER CODEBOOK

**1. Personalization and Targeting (PT):**
*   **3 (Excellent):** The letter is addressed to a specific company, mentions the position, and shows the candidate has researched the employer. Example: "I was especially inspired by your recent 'X' campaign, and I'm confident my experience in 'Y' can help build on it."
*   **2 (Acceptable):** The company and position are mentioned, but without deep personalization.
*   **1 (Poor):** A template letter that could be sent to any company.

**2. Relevant Experience Shown (RE):**
*   **3 (Excellent):** The candidate doesn't just list duties but gives 1-2 specific achievements with numbers that tie directly to the job requirements. Example: "At my last job, I grew social media reach by 40% in 6 months."
*   **2 (Acceptable):** Experience is described, but without specific numbers or achievements.
*   **1 (Poor):** Experience is irrelevant or described too vaguely ("did marketing").

**3. Call to Action (CA):**
*   **3 (Excellent):** The letter ends with a polite, confident call to the next step. Example: "I'd be glad to discuss in an interview how my skills could benefit your team."
*   **2 (Acceptable):** There's a hint of wanting to continue the conversation, but it's vague.
*   **1 (Poor):** The letter just stops without a logical ending.

# TEXT TO EVALUATE
"Hello. Saw your marketing job posting. I have a lot of marketing experience, I'm responsible and a good communicator. Ready to work. Thanks."

# RESPONSE FORMAT
Output a table with three columns: "Criterion," "Score (1-3)," "Comment for improvement."

Why This Example Works.

This example runs on the same fundamental principle as the previous one, the one proven in the study: turning a subjective task into a structured analysis.

1. Making an abstraction concrete: "A good cover letter" is an abstract idea. The prompt breaks it into three measurable components: personalization (PT), relevance of experience (RE) and closing (CA). This gives the model a clear structure for its analysis.
2. Calibration through examples: The examples for each score on each criterion act as a "training set" for the LLM. The model sees the pattern "template letter = 1 point" and "letter with numbers and facts = 3 points." Given a new text, it looks for these patterns and bases its verdict on them.
3. Reducing the LLM's cognitive load: Without a "codebook," the model would have to "guess" what matters in a cover letter based on its general knowledge. That could lead to generic, useless advice. The "codebook" focuses the model's attention on the aspects that matter to the user and makes it give specific, useful feedback on each one.

Usefulness Score: 88