TL;DR
If you ask an LLM to label texts using a list of labels with definitions (a codebook, i.e. annotation instructions), overall accuracy guarantees almost nothing. The average score can look decent while rare labels get annotated badly. Answers drift from run to run. The model often leans on label names, not on your definitions. And "how to write a codebook" recipes from other papers can backfire on your data.
Here's the classic pain. The report says "60% accuracy," so you count frequencies per category and draw conclusions. But a rare label is annotated noticeably worse than the overall average. On top of that, the model hands out labels more generously than experts do and inflates how often things happen. A second run on the same texts gives different numbers. The reason is simple: the final score averages across all labels, and frequent categories score high. Rare labels barely register in that average. Meanwhile, the model reads the label name and fills in the meaning from general knowledge, not from your definition.
The authors propose a minimal set of checks. Measure quality for each label separately. Run the annotation several times. Swap label names for neutral ones ("Label 1") or shuffle names and definitions to see whether the model actually reads the definitions. Test the codebook on text from a different source. If the prompt was refined by an LLM itself, reread the label definitions: their meaning may have shifted.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Build a control sample labeled by humans → a "text — gold labels" table
STEP 2: Label it with the model and measure quality FOR EACH label → find the worst label and its gap from the average
STEP 3: "Does the model read the definitions?" diagnostics (three separate requests):
• Label-Only — names only, no definitions
• Generic — names replaced with "Label 1, Label 2…"
• Swapped — names and definitions shuffled
STEP 4: Repeat the labeling 3–10 times → check per-label variance, average the results
STEP 5: Test on text of a different type/source → compare labels
STEP 6: If an LLM edited the prompt → manually compare label definitions "before/after"
Each step is a separate request or a separate series of requests. Steps 3–5 rerun the same prompt with a single change.
Example in Action
Task: You run customer support for an Amazon-style marketplace. You need to label 5,000 customer reviews with these labels: "Delivery," "Product Quality," "Refund," "Not as Described," "Suspected Counterfeit." The last one is rare: in a sample of 200 reviews, only 6 got it. Your VP wants the share of counterfeit reviews by quarter. First, you check whether the model can be trusted.
Prompt (main one, run 5 times on the same sample of 200 reviews):
You are labeling customer reviews from an online marketplace. For each review, choose
all matching labels from the list. Rely ONLY on the definitions below;
do not infer anything that is not in the text.
Labels and definitions:
1. Delivery — the review is about shipping times, packaging in transit, the courier, or the pickup point.
2. Product Quality — the review is about materials, breakage, defects, wear.
3. Refund — the customer describes trying to get their money back or return the item, or problems with a return.
4. Not as Described — the item differs from the photos, size, color, or specs in the listing.
5. Suspected Counterfeit — the customer explicitly says they believe the item is not genuine
(fake brand, differences from the original, missing markings).
Plain poor quality is NOT counterfeit.
Output: a JSON list of labels for each review. If none apply, an empty list.
Reviews:
{reviews}
Prompt (Generic diagnostic — same prompt, but label names carry no meaning):
Same reviews, same definitions, but the label names are replaced:
Label A — the review is about shipping times, packaging in transit, the courier, or the pickup point.
Label B — the review is about materials, breakage, defects, wear.
...
Label E — the customer explicitly says they believe the item is not genuine...
Result: You get five annotation tables for the same sample. Compare them with the human labels, one label at a time. Most likely "Delivery" scores high, while "Suspected Counterfeit" scores noticeably lower and varies more between runs. According to the authors, rare labels suffer the most. If quality nosedives in the Generic variant, the model is leaning on label names. In that case, spell out in the definitions exactly how "counterfeit" differs from "just poor quality," and recheck. If you never compared frequencies across runs, you'd never learn that the counterfeit share in your VP's report "wobbles."
Why It Works
The weakness. The final metric (accuracy, overall F1) averages labels weighted by the number of examples. Frequent categories pull the average up, while rare ones barely move the total. On top of that, the model doesn't "treat definitions as law": it generates an answer from its overall impression of the text and from the label names, which already mean something to it. Add generation randomness: at temperature above zero, runs differ.
The strength. The model is good at reacting to formal differences: swap the names for "Label 1" and the result changes. Such swaps are cheap and unambiguous. Comparing your annotation with human labels per label is easy too.
How the method exploits this. Swaps and repeats turn a vague "seems to work" into measurable signals. A quality drop under Generic or Swapped means the model relies on names. Large run-to-run variance means you can't take that label from a single run. A big gap between the worst label and the average means you can't compute frequencies for that label.
Levers: - Number of runs: 3 already smooths out noise noticeably (per the authors), 10 is more reliable; for expensive models, 3 is enough. - Codebook format (short / full / with markers like "definition:", "example:"): don't take it on faith, test it on your own texts. The authors showed the effect depends on the data. - Label order: test both forward and reverse order. - Rare labels: keep enough examples of them specifically in the control sample.
Prompt Template
The main labeling prompt plus three diagnostic variants (only the label block changes):
You are labeling {text_type}. For each text, choose {one_or_more}
labels from the list. Rely ONLY on the definitions below. Do not infer anything
that is not in the text. If nothing applies, return an empty list.
[VARIANT 0: original]
{Name_1} — {definition_1}
{Name_2} — {definition_2}
...
[VARIANT Label-Only: no definitions]
{Name_1}
{Name_2}
...
[VARIANT Generic: neutral names]
Label 1 — {definition_1}
Label 2 — {definition_2}
...
[VARIANT Swapped: names and definitions shuffled]
{Name_1} — {definition_2}
{Name_2} — {definition_1}
...
Output format: JSON, a list of labels for each text.
Texts:
{texts}
Fill in: the text type (reviews, support tickets, posts), labels with definitions, and a sample of 100–200 human-labeled texts for comparison. Run the variants one by one, each several times.
🚀 Quick start — paste into the chat:
Here is a template for auditing codebook-based labeling. Adapt it to my task: {your task: what I'm labeling and which labels}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what texts, labels and definitions you have, and whether any labels are rare. It needs exactly the definitions and label rarity: they determine which diagnostics will work and what counts as the worst label. It will assemble four prompt variants tailored to your task.
Limitations
⚠️ You need gold-standard labels. Without human labels for a control sample, there's nothing to compare against. People on your team should label at least 100–200 texts, and rare labels must be represented in them.
⚠️ Older and smaller models. The main experiments ran on a compact open model (Mistral-7B) with low randomness. There is a comparison across five models, but those are small and mid-size models too. The paper doesn't show how this plays out on today's strongest models.
⚠️ There's no universal codebook recipe. Advice like "write shorter," "add structural markers," or "put frequent labels first" helped on some data and hurt on other data. You have to test these rules on your own material.
⚠️ No winning model. The best model on one task turned out to be the worst on another. Picking "the strongest" without testing on your own data is unreliable.
⚠️ Automatic prompt optimization. The authors warn that automatically refining a codebook against a metric can quietly shift label meanings and amplify failures. Details of this part are cut off in the available text. Only the stated conclusion is here, without numbers.
⚠️ The paper doesn't provide a ready-made prompt. The diagnostics are described, but the template above was built from them.
How They Studied It
The researchers used four datasets. Two already existed: US protests (CCC, 4 labels) and minimum wage effects (12 labels). They built two new ones on the topic of "value creation": one from scientific texts (8 labels), the other from news (60 documents, to test transfer to a different source). The main model was the compact Mistral-7B; they also compared four more, including GPT-4o-mini and DeepSeek-V3. Each experiment was repeated 10 times, with confidence intervals and significance tests.
The key design choice: quality was measured separately for each label, including the worst label's F1. The gap between micro-F1 and the worst label was about 26–30 points. An aggregate report doesn't show that. On scientific texts, the share of documents where every label was correct was about 2%, even though micro-F1 was around 45%. Precision was lower than recall for 17 of 24 labels: the model assigns labels more generously than the experts.
Next, the authors took three tips from earlier work: a shorter codebook, structural markers, and label order. They applied each one separately. The results contradicted each other: markers helped a lot on the longest texts and hurt the rest, and simplifying the codebook hurt on two of three datasets. The takeaway: different papers reach opposite recommendations because they tested them on different data.
The label-name diagnostics delivered a surprise too. The "names only, no definitions" variant worked surprisingly well, while replacing names with "Label 1" or shuffling them made results drop. So the model relies on the names, not on your definitions. Repeated runs showed variance, especially on rare labels, and averaging three runs smoothed it out noticeably.
Adaptations and Extrapolations
🔧 Technique: add voting across runs → a stable label. Run the annotation 3 times and keep a label if at least 2 of 3 runs assigned it. For rare labels, flag texts where the runs disagreed and send them for human review. Averaging three runs smoothed variance, according to the paper. Voting as a rule is my idea, not from the paper.
🔧 Technique: enrich the control sample with rare labels → an honest score. If the label you care about shows up only 5 times in a random sample of 200 texts, hand-add another 20–30 examples with that label and measure its quality separately. The paper shows rare labels suffer the most. The way of topping up examples is my extrapolation.
Resources
- High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis: Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank.
- Universities and institutes: MaiNLP (LMU Munich & MCML), Fraunhofer ISI, IT University of Copenhagen.
- Data: huggingface.co/datasets/ElenaSenger/value-creation-science, huggingface.co/datasets/ElenaSenger/value-creation-news
- Code: github.com/elenasenger/high-scores-hidden-failures
- Referenced works: Halterman & Keith (2025), Generic Label and Swapped Label diagnostics; Ruckdeschel (2025), on reinterpretation; Abraham et al. (2025), Automatic Prompt Optimization; Zhao et al. (2025), SCALE.