TL;DR
Feedback-to-Rubrics is a technique that extracts an expert's hidden criteria from accumulated edits and turns them into an explicit list of requirements (a rubric) you can use to evaluate new texts. The mechanics are simple: the LLM reads a stack of your edits, figures out what's behind them, formulates criteria, and then refines them iteratively by checking: "if I apply these criteria to a text, do I get similar comments?"
The problem: when you ask an LLM to improve a text, it uses generic criteria: "clear", "persuasive", "logical". But your editor, client or boss has their own requirements. For years they've lived only in their comments: "too formal", "add numbers", "cut that word". They're not written down anywhere. The LLM doesn't know them, so it edits the text in a way that isn't what they specifically want.
The solution: give the LLM the accumulated edits → it extracts a rubric → tests it on new texts → refines it wherever its prediction diverges from the real comments. The result is an explicit list of criteria that accurately reflects the preferences of a specific person or organization. You can reuse it again and again.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
(Two stages. The first is a separate request; the second is a loop of 2-3 requests)
STAGE 1 — Extract the rubric
STEP 1: Collect the expert's real edits/comments on your past texts
→ Paste into the prompt: [text fragment] + [comment on it]
STEP 2: The LLM analyzes the patterns → outputs a rubric: a list of criteria
(each criterion: when it applies + the gist of the requirement + an example)
STAGE 2 — Iteratively refine the rubric
STEP 3: Take a new text → ask the LLM to predict comments based on the rubric
STEP 4: Compare the predicted comments with the real ones → find the mismatches
STEP 5: Ask the LLM to update the rubric based on the mismatches
(repeat STEPS 3-5 for a few rounds)
RESULT: A refined rubric → use it to evaluate and edit new texts
Example in Action
Task: You're the social media director at an agency writing Instagram posts for the coffee shop brand "Good July". The founder, Michael Carter, regularly sends edits. Over six months you've piled up 40+ comments like: "sounds like a corporate press release", "this feels cheap — take the discount out of the headline", "good, leave it as is". You want to understand his logic and apply it without him.
Prompt (Stage 1 — extracting the rubric):
You are an analyst of editorial preferences.
Below are sample text fragments and one specific editor's comments on them.
Your task is to study the patterns and build a rubric:
an explicit list of the criteria this editor goes by.
Format for each criterion in the rubric:
- Criterion name
- When it applies (for what type of fragment/situation)
- The gist of the requirement (what's needed / what's not allowed)
- Boundary with neighboring criteria (how it differs from similar rules)
- Example: original fragment → how it should be
---
SAMPLE EDITS:
Fragment 1: "15% off all cappuccinos in April!"
Editor's comment: "Take the percentage out of the headline — it looks like a flyer
someone shoves at you outside the subway, not a brand with character."
Fragment 2: "We are pleased to announce the opening of our new location in the West Loop."
Editor's comment: "Who talks to real people like that? Rewrite it like a human."
Fragment 3: "Mornings start here. Quiet, warm, unhurried."
Editor's comment: "Perfect. Exactly like that."
Fragment 4: "Our barista Leo makes lattes from his own signature recipe."
Editor's comment: "Good, but add something sensory — a smell, a texture, a moment."
[paste 10-20 more of your real pairs here]
---
Build a rubric of 5-9 criteria.
Make it so that a new text can be checked against it
to produce edits similar to the editor's comments.
Prompt (Stage 2 — applying the rubric):
You are an editor working strictly by the rubric below.
{rubric from Stage 1}
---
Here is the new text for the post:
{text}
Check the text against every criterion in the rubric:
1. Say which criterion is violated (or met)
2. Quote the specific fragment
3. Suggest an edit
Then show the full corrected version of the text.
Result:
In Stage 1, the model shows a structured list of 5-9 criteria, each with when it applies, how the requirement is worded, and examples. That's "Michael's rubric". In Stage 2, you get a line-by-line review of the new text with concrete edits for each rubric item, plus a corrected version. After 2-3 rounds of refinement, the rubric gets sharper and starts predicting the editor's real comments.
Why It Works
An LLM is good at seeing patterns in language but knows little about a specific person's implicit preferences. When you ask it to "improve the text", it leans on general notions of quality. But every editor, client or boss has their own unwritten playbook. It lives only in their edits.
The key mechanism: predict → compare → refine. The LLM doesn't just read the edits; it tests hypotheses: "if this criterion were correct, what comment would it produce?" If the prediction diverges from the real one, the criterion is off and needs refining. That turns a stack of chaotic edits into structured knowledge.
Control knobs: - Number of sample edits — the more "fragment → comment" pairs, the more accurate the rubric. Minimum: 10-15 pairs - Number of refinement rounds — 2-3 rounds give a noticeable accuracy boost; beyond that, diminishing returns - Granularity of criteria — you can ask the LLM to make criteria more or less detailed ("split this broad criterion into two narrow ones") - Positive vs. negative criteria — the system works better if you include "leave it as is" comments too; they mark what NOT to change
Prompt Template
Stage 1 — Extracting the rubric
You are an analyst of editorial preferences.
Study the examples below: each one is a text fragment
and a comment on it from {who_comments}.
Build a rubric: an explicit list of the criteria
this {who_comments} goes by.
Format for each criterion:
- Name
- When it applies
- The gist of the requirement
- Boundary with similar criteria
- Example: [original] → [how it should be]
---
EXAMPLES:
{fragment_1}
Comment: {comment_1}
{fragment_2}
Comment: {comment_2}
[add 10-20 pairs]
---
Build a rubric of {number_of_criteria} criteria.
It should make it possible to predict similar comments
for new texts of the same type.
Stage 2 — Evaluating a new text against the rubric
You are an editor working strictly by the rubric.
{rubric_from_stage_1}
---
Text to check:
{new_text}
For each criterion in the rubric:
1. Violated or met?
2. Quote from the text
3. A concrete edit (if needed)
At the end, the corrected version of the text.
Stage 3 — Refining the rubric (optional)
Compare two sets of comments on the text {text}:
Real comments from {who_comments}:
{real_comments}
Comments based on the current rubric:
{predicted_comments}
Find the mismatches. Update the rubric so
it predicts the real comments better.
What should be added, clarified, removed?
Current rubric:
{rubric}
Placeholders:
- {who_comments} — editor / client / marketing director
- {fragment_N} / {comment_N} — real pairs from your own work
- {number_of_criteria} — 5-9 for most tasks
- {new_text} — the text you need to check
- {rubric} — the output of Stage 1
🚀 Quick start — paste into the chat:
Here's a Feedback-to-Rubrics template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask who's commenting, what type of texts and how many sample edits you have, because without concrete examples there's no rubric to build. It'll help you gather the pairs and kick off the process.
Limitations
⚠️ You need accumulated feedback: The method only works if you already have at least 10-15 real comments on specific fragments. For a brand-new project with no history, there's nothing to analyze.
⚠️ The rubric reflects the past: Criteria are extracted from old edits. If the editor's preferences changed six months ago, the rubric will be out of date. Refresh it periodically.
⚠️ Accuracy drops on subjective criteria: If the editor comments on gut feeling ("the vibe is off", "something's not right"), the LLM struggles to pin down a clear criterion. It works better with concrete, recurring patterns.
⚠️ Iterative refinement takes effort: The full "extract → test → compare → refine" chain takes 30-60 minutes the first time. But you can reuse the rubric.
How They Tested It
The Sakana AI team took a practical approach: they picked nine different tasks, from reviews of research proposals and student essays to medical annotations and expert texts in biology, chemistry and cybersecurity. Two datasets had real edits from live experts; the rest were synthetic but with known rubrics (which made it possible to measure accuracy).
The baseline they compared against (RAG): instead of a rubric, just pull 3 similar comments from the past and give them to the LLM in context. Intuitively it seems like that should work, but the rubric method beat it. That's an important result: generalized criteria work better than feeding concrete examples directly. A rubric captures the essence, not the surface.
A curious detail: iterative refinement works through pairwise linking: each comment is tied to the specific criteria that produced it. That matters: if you dump all the mismatches in at once, the model can't tell what exactly is broken. When a mismatch is tied to a criterion, it's clear what to change. That's what explains why the "comment-wise" version beats the plain aggregate version.
Adaptations and Extrapolations
Adaptation 1: A rubric from other people's reviews
🔧 Technique: take public reviews → extract the expert community's criteria
If you don't have personal edits, use public ones: Amazon reviews of competitors' products, reviews of books in a particular genre, comments in professional Discord servers and newsletters. The method works on any accumulated comments, not just "yours".
Here are 15 reader reviews of business books in the "personal productivity" genre.
[reviews]
Extract a rubric: what criteria do readers
of this genre use when judging a book?
What do they praise, what do they criticize, and under what conditions?
Adaptation 2: Vector comparison without iterations
🔧 Technique: one step instead of a loop for a quick check
If you don't have time for iterations, you can use the rubric "as is" after the first extraction. Even a rough rubric from 10+ examples is far more accurate than "improve the text" with no criteria. Save the full loop for tasks that need high accuracy.
Extrapolation: transferring style through a rubric
The method solves a related problem: transferring an editorial style. If a new copywriter on the team has to write "the way Sarah used to", you don't need to explain the rules in words. Collect the edits Sarah gave in the past → extract a rubric → hand the rubric to the new copywriter (or an LLM). An explicit rubric transfers style more precisely than the instruction "write like this, look at her texts".
Resources
Paper: Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
Authors: Kotaro Yoshida, So Kuroki, Yuki Imajuku, Taishi Nakamura, Ryunosuke Iwai, Haruki Goda, Takuya Akiba
Affiliations: Sakana AI, Institute of Science Tokyo
Related work: - HealthBench (Arora et al., 2025) — a benchmark of medical annotations - ExpertLongBench (Ruan et al., 2026) — expert texts across 6 domains - RAG baseline: Lewis et al. (2020) — Retrieval-Augmented Generation