Moderation-oriented Guideline Repurposing: adapting human instructions for LLM annotators

A 3-phase iterative loop (several LLM requests):

PHASE 1 — Annotation:
Input: text + detailed guidelines
LLM annotator → labels the samples
Output: labeled data

PHASE 2 — Evaluation:
The system compares against the gold standard → computes the F1-score
If F1 < threshold → go to Phase 3
If F1 ≥ threshold → done

PHASE 3 — Moderation:
The system identifies: false positives (FP) + misses (FN) + category errors
LLM moderator analyzes → generates a report on the causes
LLM moderator (or a human) → updates the guidelines
Back to Phase 1 with the new guidelines

Human-in-the-loop (optional): In Phase 3 an expert can review the LLM moderator's suggestions and adjust the guidelines by hand — this adds +8% accuracy (F1: 0.50 → 0.58).

Task: You're a marketer at an online store that sells on Amazon. You need to classify 2,000 product reviews into categories: "Product quality", "Shipping", "Packaging", "Customer service", "Price". You have a playbook (5 pages), but the LLM keeps mixing things up: it tags "arrived fast" sometimes as "shipping" and sometimes as "customer service", and "totally worth the money" sometimes as "price" and sometimes as "quality".

Prompt (Phase 1 — first iteration):

You are an annotator of online store reviews.

GUIDELINES:
1. Product quality: mentions of features, defects, match with the description
2. Shipping: speed, condition on arrival, the courier's work
3. Packaging: packaging quality, damage in transit
4. Customer service: interactions with support, returns, replacements
5. Price: value for money, discounts, overpaying

TASK: Label the reviews from the list below. For each review, list ALL matching categories, separated by commas.

REVIEWS:
1. "Arrived fast, but the box was crushed. The item itself was intact, excellent quality!"
2. "Totally worth the money! Delivered the next day."
3. "Support replied after 2 days and didn't help me return the item."
4. [2 more samples]

ANSWER FORMAT:
1. [categories]
2. [categories]
...

Result: The LLM labels the first 5 reviews. You compare that with how you'd label them yourself (or with a colleague's ready-made gold labels). You spot errors — for example, the LLM missed "Packaging" in review #1 and tagged review #2 only as "Price", even though it also covers "Shipping".

(Phase 2): You list the discrepancies: FP (extra categories), FN (missed ones), mix-ups.

(Phase 3 — moderation):

You are a labeling expert. Here are the guidelines for classifying reviews [paste from above] and a list of the LLM annotator's errors:

ERRORS:
- Review "Arrived fast, but the box was crushed..." → the LLM missed "Packaging"
- Review "Totally worth the money! Delivered..." → the LLM tagged only "Price", missed "Shipping"
- Review "Support replied after 2 days..." → the LLM tagged "Customer service", correct

TASK:
1. Analyze the causes of the errors
2. Suggest how to refine the guidelines so the LLM doesn't get confused

FORMAT:
Cause: [...]
Fix: [...]
New version of the guideline: [...]

The LLM moderator might suggest, for example:

• "Add to the 'Packaging' guideline: 'mentions of the condition of the box/bag count as packaging, even if the item is intact'"
• "Clarify 'Shipping': 'phrases like 'arrived fast', 'delivered the next day' = shipping'"

You update the guidelines → go back to Phase 1 → the LLM labels again. Repeat until the accuracy is good enough (3-5 iterations in the study).

LLM weakness: The model doesn't "understand" context the way a trained human does. An instruction like "classify into categories" without detail yields an abstract understanding — the LLM leans on frequency patterns from its training data rather than the specifics of your task. For example, "the box was crushed" could mean "Shipping" (a problem in transit) or "Packaging" (packaging quality) — the context is ambiguous, and without explicit rules the LLM picks at random.

LLM strength: The model is excellent at following explicit instructions and can analyze error patterns. Instead of just "label the reviews", give it "phrases about the box's condition = packaging, about delivery speed = shipping, even in the same sentence" — and accuracy goes up. Plus, an LLM can generate meta-instructions: analyze why it got something wrong and suggest guideline improvements.

How the method uses this: Instead of guessing by hand "how best to explain it to the LLM", the method automates the iteration. First the LLM labels using basic rules → weak spots surface (where it gets confused) → a second LLM (the moderator) spells out what exactly is unclear and how to improve the instruction → the updated instruction gets tested again. It's a feedback loop that human labeling teams run through training sessions and meetings — but for an LLM it has to be materialized as text.

Control knobs:

• Sample size (5 reviews) → increase to 10-20 if you want to cover more edge cases, but expect higher token use and cost
• F1-score threshold (0.8) → raise it to 0.9 if accuracy is critical; lower it to 0.7 if basic labeling is enough
• Moderator role → replace the LLM moderator with yourself: analyze the errors and write the improvements yourself (slower, but more accurate)
• Guideline detail → start with short rules (3 points), then add edge cases as you iterate (don't overload it all at once)

Phase 1 — Annotation:

You are an annotator for the task: {task_description}.

GUIDELINES:
{detailed_labeling_rules}

Edge case examples:
{tricky_case_examples}

TASK: Label the following samples according to the guidelines.

SAMPLES:
{texts_to_label}

ANSWER FORMAT:
{output_structure}

Phase 3 — Moderation:

You are an expert in improving labeling instructions.

CURRENT GUIDELINES:
{guidelines_from_phase_1}

LLM ANNOTATOR ERRORS:
False positives (FP): {FP_list}
Misses (FN): {FN_list}
Category mix-ups: {mis-categorized_list}

TASK:
1. Analyze the cause of each error
2. Propose specific guideline improvements
3. Rewrite the problematic guideline sections to account for the errors

FORMAT:
Cause of error: [...]
Fix: [...]
Updated guideline: [...]

Placeholders:

• {task_description} — what you're labeling (reviews, resumes, news, contracts)
• {detailed_labeling_rules} — the rules for each category/label, inclusion criteria
• {tricky_case_examples} — 2-3 examples of ambiguous situations with explanations
• {texts_to_label} — 5-10 samples for the first iteration
• {output_structure} — what the result should look like (a list, table, JSON)
• {FP/FN/mis-categorized_list} — concrete examples of errors, showing what the LLM did and what it should have done

🚀 Quick start — paste into the chat:

Here's a method for improving data labeling by iteratively updating the instructions.
Adapt it to my task: [your labeling task].
Ask me questions to fill in all the fields.

[paste the Phase 1 and Phase 3 templates above]

The LLM will ask: (1) what exactly you're labeling, (2) which categories/labels you use, (3) whether you have examples of tricky cases, (4) how many samples for the first check, (5) what output format you need. It takes the structure from the template and builds a working prompt for your task.

⚠️ Scope Mismatch: LLMs are bad at picking up domain-specific definitions. In medicine, "tumor" counts as a disease under the MeSH terminology, even though semantically it's a symptom. The LLM goes by everyday language and skips such terms unless the terminology is spelled out explicitly. You have to state the exact boundaries by hand (for example, "in our classification, 'tumor' = disease").

⚠️ Abbreviations and ambiguity: Short forms like "DM" (diabetes or dystrophy), "CT", "IVF" were the source of ~51% of errors in the study. The LLM picks the most frequent meaning from its training data and ignores the context. Even with detailed guidelines, accuracy on ambiguous abbreviations stays low.

⚠️ Category edge cases: Mix-ups between neighboring categories ("Disease Class" vs "Specific Disease" vs "Modifier") are a persistent problem. For example, "APC gene" should be labeled "Modifier", but the LLM often labels it "Specific Disease". The method reduces errors through iteration but doesn't eliminate them entirely (an improvement from 0.36 to 0.58 F1, not to 0.9+).

⚠️ Tokens and cost: Every iteration = a new LLM request with the full guidelines. If the guidelines are long (10+ pages) and there are many samples (50+), the cost climbs. The study used 5 samples per iteration to keep costs down. For full-scale labeling of thousands of texts, it's better to combine the method with other approaches (for example, fine-tuning once the guidelines have stabilized).

⚠️ You need a gold standard: Phase 2 requires gold samples (gold annotations) — pre-labeled examples to compare against. If you don't have any, you'll need to create at least 10-20 by hand for the first iteration. Without a gold standard, the method doesn't work.

The team used the NCBI Disease Corpus — 793 annotated medical abstracts with 6,892 disease mentions labeled by humans according to detailed guidelines. The task: find disease mentions in PubMed texts and assign them to one of 4 categories: Specific Disease (a particular disease like "colorectal cancer"), Disease Class (a family of diseases like "hereditary disorders"), Modifier (a modifier like "APC gene"), Composite Mention (several diseases in one phrase, like "Crouzon and Pfeiffer syndromes"). It's a hard task even for humans — the minimum inter-annotator agreement was 0.8, which became the threshold for automation.

They tested 3 approaches: (1) Baseline — a simple "label the diseases" prompt, (2) Guideline Prompt — the prompt + the full guidelines with no moderation, (3) Moderation-oriented — iterative guideline improvement via an LLM moderator. Plus a Human-in-the-loop variant, where a person reviewed the LLM moderator's suggestions and made manual corrections.

They measured 4 metrics: Strict Match (exact boundaries and category), Strict Match without category (exact boundaries, category doesn't matter), Soft Match (boundaries may be off by 1-2 words) and Soft Match without category. Results: Baseline scored F1 = 0.36, simply adding the guidelines → 0.46, automatic moderation → 0.50, human-in-the-loop → 0.58. Interestingly, Soft Match reached 0.78-0.80 — which shows the LLM understands what to label but is bad at pinning down exact boundaries (where a term starts and ends).

Why did the results come out this way? The researchers found 2 key problems: (1) Scope Mismatch — the LLM doesn't know specialized terminology (e.g., MeSH), and (2) Category Mismatch — confusion between categories, especially with modifiers. The most curious part: 51% of errors came from abbreviations like "DM", "CT", "APC" — the LLM couldn't choose between multiple meanings even with detailed examples. Another 18% came from general terms like "tumor", which the medical classification counts as a disease but the LLM treats as a symptom.

Practical takeaway: Even detailed guidelines don't replace explicitly specified terminology and resolved ambiguity. If your task involves specialized terms or words with multiple meanings, add not just rules to your guidelines but a glossary (what each term means in the context of the task).

Context: The researchers didn't provide a single prompt; they described a 3-phase workflow. Below is the structure of each phase from the paper (simplified, but with no loss of detail):

Phase 1: Annotation

You are a text annotator for disease mentions in biomedical abstracts.

ANNOTATION GUIDELINES:
[Full NCBI Disease Corpus annotation guidelines - 5+ pages of detailed rules]

Categories:
- Specific Disease: mentions that link to one specific definition
- Disease Class: mentions referring to a family of diseases
- Modifier: textual string that modifies a noun phrase (e.g., "APC gene")
- Composite Mention: two or more separate disease mentions in one string

TASK: Annotate the following PubMed abstracts according to the guidelines.

ABSTRACTS:
[5 sample abstracts]

OUTPUT FORMAT:
For each abstract, list disease mentions with:
- Text span
- Start/end character offsets
- Category (Specific Disease / Disease Class / Modifier / Composite Mention)

Phase 2: Evaluation

[System compares LLM output with gold annotations]
- Compute Precision, Recall, F1-score (Strict Match)
- If F1 < 0.8 → proceed to Phase 3
- Extract discrepancies:
  * False Positives (FP): LLM marked, but not in gold
  * False Negatives (FN): in gold, but LLM missed
  * Mis-categorized: correct span, wrong category

Phase 3: Moderation

You are an expert in improving annotation guidelines.

CURRENT GUIDELINES:
[Guidelines from Phase 1]

ERRORS IDENTIFIED:
False Positives: [list with examples]
False Negatives: [list with examples]
Mis-categorized: [list with examples]

TASK:
1. Analyze root causes of each error type
2. Propose specific guideline improvements
3. Generate updated guideline sections with:
   - Clarified definitions
   - Additional edge case examples
   - Disambiguation rules for ambiguous terms

OUTPUT:
Error Analysis Report:
- Cause: [...]
- Proposed Solution: [...]
- Updated Guideline: [...]

The researchers used GPT-4o for both roles (annotator + moderator), a sample size of 5 abstracts, and an F1 threshold of 0.8 (the minimum human agreement in the corpus). The loop repeated until the threshold was reached or for at most 5 iterations.

💡 Adaptation for social media content moderation

The same logic works for classifying content for toxicity, spam and platform rule violations. A classic example: moderating comments in a Facebook group or a Discord server.

You are a comment moderator for a Facebook group (500k members).

MODERATION RULES:
1. Spam: links to outside sites, unapproved ads, repetitive messages
2. Insults: direct aggression toward users, profanity aimed at people (NOT emotional exclamations)
3. Off-topic: discussing subjects unrelated to the group's theme
4. Politics: mentions of political parties, elections, the government (neutral context is OK)

TASK: Label the comments. If there are several violations, list them all.

COMMENTS:
[5 samples]

After the first iteration: compare with the gold standard → identify the errors → ask the LLM to refine the rules.
For example, "emotional profanity ('f*ck, what a sick track!') ≠ an insult" or "mentioning a city in the context of an event ≠ politics".

🔧 Technique: Checklist questions instead of categories

Instead of direct classification ("label the categories"), give the LLM a checklist of questions. This reduces ambiguity.

For each review, answer the questions YES/NO:
1. Are product features mentioned (color, size, material, defects)? → Quality
2. Is the delivery speed/method or the condition on arrival mentioned? → Shipping
3. Is the box/bag/packing material mentioned? → Packaging
4. Are interactions with support/returns/exchanges mentioned? → Customer service
5. Is value for money/discounts/overpaying mentioned? → Price

If YES → assign the category. One review can have several categories.

This lowers the LLM's cognitive load — instead of "pick from 5 categories", it answers specific binary questions.

💡 Extrapolation: applying it to legal documents

The method works for classifying contract types, spotting risks in contracts and labeling regulations.

You are a legal analyst. You label contracts for problematic terms.

GUIDELINE:
A problematic term is a clause that creates risks for the client:
1. Unlimited contractor liability (phrases like "in any event", "bears full responsibility")
2. Unilateral termination without compensation (phrases like "may terminate unilaterally without...")
3. Vague deadlines (phrases like "within a reasonable time", "as soon as practicable")

TASK: Find the problematic clauses in the contract [text].

[Moderation loop]: Compare with a lawyer's gold labels → identify misses → refine the wording.

After 2-3 iterations the LLM will start catching subtleties like "the phrase 'as soon as possible' = a vague deadline" or "unilateral termination ≠ a problem if there's 30 days' notice".

Paper: Kim, K.W., Islamaj, R., Kim, J.D., Boudin, F., & Aizawa, A. (2024). "Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study". Proceedings of EMNLP 2024.

Dataset: NCBI Disease Corpus — https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/

Authors: Kon Woo Kim, Rezarta Islamaj, Jin-Dong Kim, Florian Boudin, Akiko Aizawa (SOKENDAI / National Institute of Informatics / NLM / CNRS)