Apply to my task
arXiv:2610.09377 ★ 81 Oct 7, 2026

Check the Whole Tree: Why a Node-by-Node LLM Judge Misses Broken Taxonomies

Key result

The paradox: an LLM judge checks every node of a category tree and scores 0.97–1.00, yet almost every leaf's name is just its ancestor's name plus one new word. The method lets you find such hidden defects in generated category trees (for example, for customer feedback) with a plain script, no LLM judge. The trick: the defect lives not in a node but in the tree as a whole. Instead of asking "is this node good," compute properties of the whole tree: the share of leaves repeating an ancestor's name, vocabulary diversity, branching per level and leakage of records between branches. In the study, on one level each of 158 parents had exactly one child, and not a single node-level judge noticed.

01

TL;DR

The study shows that checking "one node at a time" can't see the main defects of a generated category tree. The authors replaced it with whole-tree checks: what share of leaf names repeat their ancestors' names, how varied the vocabulary is, how each level branches, and how many records land in several branches at once. These metrics are computed by a plain script, no LLM judge needed.

A typical pain point: an agent (Codex on GPT-5.x) builds a category hierarchy for customer feedback. An LLM judge checks each node separately: "the name fits the description," "the parent is correct." Everything passes at 0.97–1.00, and the tree looks professional. But almost every leaf's name is just its ancestor's name glued to one new word. On one level, each of the 158 parents has exactly one child, meaning the tree is made of "wrappers." The judge can't see this because it looks at one node at a time, and the defect exists only at the level of the whole tree.

The method: after generation, don't trust local checks; run two sets of whole-tree checks instead. The first is structural "template detectors": ancestor repetition, vocabulary diversity, a branching ceiling, absence of shared nodes. The second is "separability by team": how often a single record lands in several top-level branches.

Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.

02

Method Blueprint

STEP 1: Generate the tree (agent or LLM)              → taxonomy.json / list of paths
STEP 2: Structural checks (no LLM judge, from the tree file):
        • ancestor name repeated in the leaf name (share of leaves)
        • information gain of a name = 1 − share of words already used by ancestors
        • vocabulary diversity by level (unique words / all words)
        • branching: how many children each parent has on each level
          (flag: ≥10 parents and none with more than 3 children)
        • shared nodes under several parents (how many)
STEP 3: Separability checks (requires all records to be labeled against the tree):
        • leakage = share of records that land in >1 top-level branch
        • record purity = share of its paths in the main branch
        • modularity by leaf co-occurrence (Q>0 means branches hold together)
STEP 4: Compare with a reference or a repeat run → decide whether the tree is usable
03

Example in Practice

Task: A product manager at Instacart asks Claude Code to build a category tree for 3,000 app reviews: order picking, shoppers, payment, promotions, support. The result has to be split across teams. It looks great, but you can't check 800 leaves by hand. You need to find hidden defects before the tree goes into analytics.

Prompt:

Below is a category tree for reviews of a grocery delivery app. It is the path to each leaf joined with " > " (L1 > L2 > L3 > Subtopic).

Do not rate each node "good / bad." Check the tree AS A WHOLE and compute:

1. The share of leaves whose name fully contains the name of any ancestor. Give the 5 worst examples.
2. For each leaf: information gain = 1 − (number of the leaf's meaningful words already used by ancestors) / (number of the leaf's meaningful words). Give the tree-wide average.
3. Vocabulary diversity for each level: number of unique words in the level's names / total number of words in the level's names.
4. For each pair of adjacent levels: how many children each parent has (min / median / max). "Ceiling" flag: ≥10 parents and none with more than 3 children.
5. How many nodes sit under more than one parent.

Return a table, then 3–5 lines of conclusions: which systemic defects are visible (templated names, single-child chains, the same concept duplicated in different branches). If you can't compute something exactly, say so and propose a Python script.

Tree:
{tree_paths}

Result: The model returns a table with five metrics and examples of the worst leaves. In its conclusions it names systemic patterns: glued-together names, a branching ceiling, no shared nodes. On a large tree it will most likely propose a Python script. You can run it as is.

04

Why It Works

The weakness: an LLM judge answers local questions: "does the name fit the description." Each answer is correct, but they get aggregated into a single pass rate. Properties of the whole tree don't belong to any single node: a naming template, a rigid branching ceiling, records leaking between branches. So the judge doesn't detect them.

The strength: whole-tree metrics are computed with simple string, set and graph operations: a substring, a ratio, a child count. A script can produce them without an LLM, and they give a yes/no answer rather than an eyeball estimate. Here the LLM is just a helper that writes the script and interprets the result.

Knobs to turn: - The ceiling flag threshold (≥10 parents, ≤3 children): raise or lower it for small trees. The authors got one false alarm on a reference tree. - Which levels to check: for routing to teams look at L1–L2, for topic analytics look at the leaves. - The reference: take a vetted tree or a second run of the same generation. The metrics show the difference, not an absolute. - Tree size: a few hundred nodes are fine in chat; for thousands, ask for a script.

05

Prompt Template

Role: category hierarchy auditor. You do NOT rate nodes one by one.
You look for properties that are visible only across the whole tree.

<Input>
Tree: {tree_as_paths}
Number of records (if labeled): {num_records}
Reference tree for comparison (if any): {reference}
</Input>

<Checks>
1. containment = share of leaves where the full name of any ancestor (≥2 words) appears as a substring of the leaf name
2. gain(n) = 1 − |words(n) ∩ words(ancestors of n)| / |words(n)|; output the average
3. TTR(level) = unique words / all words in the level's names
4. fan-out for each pair of levels: min / median / max children;
   cap-suspect if parents ≥ {parent_threshold} and none has children > {child_threshold}
5. reuse = number of nodes with more than one parent
6. (if records are labeled with paths)
   leakage = share of records touching >1 top-level branch
   purity = average share of a record's paths in its main branch
</Checks>

<Output>
Metrics table → 5 worst containment examples → conclusion: which defects are systemic.
If you can't compute something exactly, say so and give a Python script.
Don't write "the tree looks good" without numbers.
</Output>

Fill in: {tree_as_paths} is lines like "Delivery > Shopper > Late arrival"; {parent_threshold} and {child_threshold} default to 10 and 3.

🚀 Quick start: paste this into the chat:

Here is a template for checking a category hierarchy as a whole. Adapt it to my task: {my task: what kind of tree, what it's for}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what format your tree is in, whether your records are labeled, and whether you have a reference to compare against. It needs this to figure out which checks it can compute and which it can't: leakage can't be computed without labels. It will take the pattern from the template and build a prompt or script for your data.

06

Limitations

⚠️ One agent harness: all the generated trees came from the same agent (Codex on GPT-5.x). Other agents and prompts may produce different defects.

⚠️ Private and limited data: two feedback datasets from two commercial products, six trees. The pattern repeated on both datasets and in every run, but the sample is small.

⚠️ The references aren't a clean control: the customer's production trees were built with checks similar to the new metrics, so their good profile is partly baked in. The most reliable comparison is between two runs of the same agent.

⚠️ Separability metrics depend on the labeling: records are classified against each tree separately, and the result depends on the quality of that classification, not just on the tree.

⚠️ The metrics catch defects; they don't prove usefulness: "no repeated ancestors" doesn't mean the tree is good. If you optimize generation for these metrics, they'll stop reflecting quality (Goodhart's law). Some numbers, like the maximum number of children, only give context, not a score. The "ceiling" threshold produces false positives.

⚠️ The paper text cuts off: the modularity formula and the separability table aren't fully visible. The separability effect relies on conclusions from the abstract and introduction.

07

How They Studied It

Researchers from Enterpret took two customer feedback datasets from real products (1,940 and 5,000 records). For each dataset they assembled three trees: the customer's production tree and two independent runs of the same agent on an identical task. The task described a five-level schema and a "no duplicates" rule. Everything went through ordinary LLM checks: does the name fit the description, is the parent correct, are siblings distinguishable. All six trees passed at 0.92–1.00. A deeper check of product-term coverage even preferred the generated trees over the references.

Then the authors computed whole-tree metrics, and the picture changed. In the generated trees, 97.7% to 100% of leaf names repeat an ancestor, versus 13.9% and 2.9% in the references. In one tree, all 158 parents on a level have exactly one child each. The generated trees have no shared nodes under multiple parents at all; the references have 48 and 73. When real records were routed, between a third and three quarters of records in the generated trees landed in several top-level branches, versus about a fifth for the references. Two runs of the same harness, which ordinary checks rate as equally correct, differ by 27 percentage points in cross-branch leakage. One of them splits related topics worse than a random partition.

The surprise: repeat runs of the same agent on the same task produce different trees, and local checks don't notice. The practical takeaway: node-level grading creates a false sense of reliability, so you need to measure the tree as a whole.

08

Adaptations and Extrapolations

Below are my own ideas inspired by the paper; the authors didn't test them.

🔧 Technique: add prohibitions to the generating agent's task → fewer templated names

In the system prompt or CLAUDE.md for the agent that builds the taxonomy:

Naming and structure rules:
- A leaf name must not fully contain an ancestor's name.
- Don't create a level where every parent has exactly one child.
- If one concept belongs to several branches, make one shared node with several parents, not copies.

🔧 Technique: a self-check at the end of the task → the defect is caught before handoff

After building the tree, compute: the share of leaves that repeat an ancestor's name;
the number of parents with a single child on each level;
the number of nodes with several parents.
If the repeat share is >30% or there's a level where everyone has one child, rebuild that level and show the metrics before and after.
09

Resources

  • Paper: From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. Authors: Prabhath Chellingi, Raviraja G, Viraj Bagal (Enterpret).
  • Related work cited in the paper: TnT-LLM (evaluating taxonomies with LLM judges), InsightNet (taxonomy from reviews), Newman modularity.
  • Harness used: Codex, GPT-5.x; LLM judges: gpt-5.4-mini and gpt-5.5.

Research digest

Key result

The paradox: an LLM judge checks every node of a category tree and scores 0.97–1.00, yet almost every leaf's name is just its ancestor's name plus one new word. The method lets you find such hidden defects in generated category trees (for example, for customer feedback) with a plain script, no LLM judge. The trick: the defect lives not in a node but in the tree as a whole. Instead of asking "is this node good," compute properties of the whole tree: the share of leaves repeating an ancestor's name, vocabulary diversity, branching per level and leakage of records between branches. In the study, on one level each of 158 parents had exactly one child, and not a single node-level judge noticed.

Principle

The rule is simple: after generation, don't trust "one node at a time" checks. Run two sets of checks on the whole tree. The first set is template detectors. They work from a single tree file: • share of leaves where an ancestor's name appears in the leaf's name; • information gain in a name: 1 minus the share of words ancestors already used; • vocabulary diversity by level: unique words / all words; • branching: how many children each parent has. The "ceiling" flag is ≥10 parents and none with more than 3 children; • how many nodes sit under several parents. The second set is separability by team. It needs records labeled against the tree: • leakage: share of records that land in more than one top-level branch; • record purity: share of its paths in the main branch. It's like inspecting a house as a whole instead of checking each brick separately: every brick is intact, but the house is standing without a single load-bearing wall. All the metrics are substrings, ratios and child counts. No "eyeball estimates."

Why it works

An LLM judge answers local questions: "does the name fit the description?", "is the parent correct?" Each answer is correct. Then everything is merged into a single pass rate. A naming template, a branching ceiling and cross-branch leakage don't belong to any single node, so a local judge physically can't see them. A script won't miss these problems. It computes a substring, a ratio, a child count and gives a clear yes/no. The LLM is a helper: let it write the script and explain the result. On trees from an agent (Codex on GPT-5.x), the pattern repeated on two feedback datasets and in every run. One caveat. If you then tune generation toward these metrics, they'll stop reflecting quality (Goodhart's law). The metrics catch defects; they don't prove the tree is useful.

When to apply

Generating hierarchies and taxonomies with agents or LLMs → specifically category trees for reviews, support tickets or content topics, especially when the tree has hundreds of leaves you can't check by hand, and the result will feed analytics or route records to teams. Which levels to check: for routing to teams, L1–L2; for topic analytics, the leaves. Adjust the "ceiling" threshold (≥10 parents, ≤3 children) to the size of the tree. The authors had one false alarm on a reference tree. NOT a fit as proof that a tree is good: "no repeated ancestors" doesn't yet mean "useful tree." Leakage can't be computed without labeled records. The study is small: one agent, two private feedback datasets, six trees.

Mini recipe

1. Export the tree as paths: one line per leaf, like "Delivery > Shopper > Late arrival." 2. Ban node grading: say it outright: "don't rate each node; check the tree as a whole." 3. Compute ancestor repetition: share of leaves where an ancestor's name sits inside the leaf's name. Ask for the 5 worst examples. 4. Check information gain: how many of a leaf's words are new and how many ancestors already used. You want the tree-wide average. 5. Look at branching: min / median / max children for each pair of levels. If the "ceiling" fires (≥10 parents and none with more than 3 children), look for single-child chains. 6. Find shared nodes: how many nodes sit under several parents. Zero on a large tree is a red flag. 7. Large tree? Ask for a script: a few hundred nodes are fine in chat; for thousands, have the model write Python. If it can't compute exactly, it should say so. 8. Have labeled records? Add leakage (share of records in more than one top-level branch) and record purity. 9. Compare with a reference: take a vetted tree or a second run of the same generation. The metrics show the difference, not an absolute.

Examples

[BAD]: `Check the review category tree. For each node, say whether the name fits the description and rate it from 1 to 10.` [GOOD]: `Below is a category tree for reviews of a grocery delivery app. The path to each leaf is joined with " > ". Don't rate nodes one by one. Check the tree AS A WHOLE and compute: 1) the share of leaves whose name fully contains the name of any ancestor, plus the 5 worst examples; 2) average information gain = 1 − (leaf words already used by ancestors) / (all leaf words); 3) vocabulary diversity by level = unique words / all words; 4) for each pair of adjacent levels, the min / median / max children per parent, with a "ceiling" flag: ≥10 parents and none with more than 3 children; 5) how many nodes sit under several parents. Return a table and 3–5 lines of conclusions about systemic defects. If you can't compute something exactly, say so and give a Python script. Tree: {tree_paths}` Result: instead of "everything looks professional," the model returns a table with numbers, the worst leaves and systemic patterns: glued-together names, a branching ceiling, no shared nodes.

In plain words

Models have learned to build elaborate category trees, but until now we had no decent way to grade the result. Point a popular LLM judge at each individual node, and the model cheerfully reports: all logical, no complaints. But that's a complete illusion of quality. Language models suffer from tunnel vision: they easily judge the local meaning of a single branch but are blind to systemic flaws in the whole structure.

It's like checking whether a house is sound by examining each brick separately under a magnifying glass. The bricks are perfect, no argument there, except the builders forgot the foundation and threw together a lopsided house of cards. The result: a tree with hundreds of categories looks plausible but is actually an unusable mess, with the same idea smeared thinly across ten different corners.

The authors ditched expensive AI judges and replaced them with simple scripts that evaluate the graph as a whole. What actually works: ancestor overlap analysis (so leaves don't just parrot their roots' names), a check for lexical diversity, and strict control of data leakage between branches. No prompting magic: dry math with no neural networks instantly exposes skews and duplicates.

The approach was tested on 3,000 reviews of a delivery service, but the principle is completely universal. The method is essential anywhere agents build hierarchies: from online store catalogs with a million products to support knowledge bases. Feed a model raw customer pain points, and a crooked taxonomy is guaranteed to bury all your analytics and send the team down the wrong path.

Stop spending budget on LLM grading of structure: check the result at the macro level with code, not someone else's subjective opinion. Structural metrics instead of neural judges will save you a ton of time and protect you from screwups in production. Otherwise your smart agent will build a stillborn hierarchy that your analysts drown in on day one.

AI lab assistant

Apply this study to your task

Describe what you're working on. AI takes the method and the measured results from this paper and adapts them to your case, or writes a ready prompt built on them.

Based on Check the Whole Tree: Why a Node-by-Node LLM Judge Misses Broken Taxonomies arXiv:2610.09377

Sign in to apply this study New accounts get 50 free credits.

Related techniques