Confusion-Aware Classification: Explicit Tie-Breaker Rules for Look-Alike Categories

STEP 1 (once, setup): Find the confusion
→ Run classification on examples with known correct answers
→ Collect the list of category pairs the model confuses most often

STEP 2 (once, setup): Generate the rules
→ For each confused pair, show the model examples of its errors
→ Ask it to spell out: "which feature distinguishes A from B"
→ Collapse the many observations into one clear rule

STEP 3 (on every new classification): Apply
→ In the prompt with the text and the category list, add:
   - both labels of every confused pair (even if retrieval found only one)
   - the ready-made tie-breaker rules for those pairs
→ The model picks the final category

Steps 1 and 2 are offline prep, done once per taxonomy. Step 3 runs on every new classification.

Task: The support team of an online marketplace tags customer requests by category: hundreds of tickets a day, and categories like "Product return", "Product exchange", "Quality complaint" and "Defect complaint" constantly get mixed up.

Prompt (step 1, find the confusion):

Here is the list of request categories: {category_list}.

Here are 20 requests where the previous classification was wrong 
(in parentheses: the correct category, then the one the model chose):
{list_of_error_examples}

Analyze: which pairs of categories get confused most often? 
For each pair, state the direction of the confusion 
(category A mistaken for B, or vice versa, or both ways).

Prompt (step 2, generate a rule for a pair you found):

Here are examples of requests that belong sometimes to the category 
"{category_A}" and sometimes to the category "{category_B}":
{examples}

Write one clear rule: which specific words 
or phrases in the text tell you it's "{category_A}" 
and not "{category_B}", and vice versa.

Prompt (step 3, final classification with rules):

Classify the customer request into one of these categories: {category_list}.

Take into account the rules for telling similar categories apart:
- "{category_A}" vs "{category_B}": {rule}
- "{category_C}" vs "{category_D}": {rule}

Customer request: "{request_text}"

Answer with the category name only, no explanation.

Result: In step 1 the model will produce a list of confused pairs with the direction of the error. In step 2, one tie-breaker rule per pair, in the form "if the text has feature X → category A". In step 3, the final category, with noticeably higher accuracy on look-alike categories than without rules in the prompt.

The model struggles to distinguish categories that are semantically close but specific to a particular taxonomy. For example, how "Return" differs from "Exchange" under your store's specific policies is something the model can't know from general knowledge. Simply growing the candidate list doesn't help: when there are more categories, there are also more options to get wrong.

On the other hand, the model is good at following an explicit rule if it's stated concretely and given right in the prompt. A rule like "if the customer wants a different item instead, it's an Exchange; if they want their money, it's a Return" removes the ambiguity the model can't resolve on its own.

The method uses that strength: instead of counting on the model to "figure out" how to tell similar categories apart, it gets a direct instruction for each specific problem pair.

Control levers: - Number of errors analyzed in step 1 → the more error examples, the more reliable the list of confused pairs; for a quick test 15-20 examples is enough - How the rule is worded → you can ask the model to express the rule as specific keywords/features rather than a general description, which makes it easier for the model to apply the rule literally - Number of pairs you generate rules for → if you have many categories, start with the 5-10 most frequent confusions; don't try to cover everything at once

STEP 1 — find the confusion:
Here is the list of categories: {category_list}.
Here are examples of requests where the classification was wrong 
(correct category → category the model chose):
{error_examples}

Find the pairs of categories that get confused most often. 
For each pair, state the direction of the confusion.

---

STEP 2 — write a rule (repeat for each pair):
Here are examples of texts that belong sometimes to "{category_A}" 
and sometimes to "{category_B}":
{examples}

Give one clear rule: which feature in the text distinguishes these two categories.

---

STEP 3 — classification with rules:
Classify the text into one of these categories: {category_list}.

Rules for telling similar categories apart:
- "{category_A}" vs "{category_B}": {rule}

Text: "{text}"

Answer with the category name only.

Plug in your own category list, your own error examples and the text to classify. The rules from step 2 accumulate in a separate list and get reused in every step 3 prompt.

🚀 Quick start: paste this into the chat:

Here's a template for the "telling similar categories apart" method. Adapt it to my task: 
{your classification task}. Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what your categories are and whether you have examples of misclassifications, because without concrete categories and error examples it's impossible to write tie-breaker rules. It will take the template's structure and tailor it to your taxonomy.

⚠️ The effect is weak for short category lists: if you have 5-7 clearly distinct categories with no overlap in meaning, the method won't give a noticeable gain. There's almost no confusion, so tie-breaker rules are overkill.

⚠️ You need error examples: the method works off of seeing where the model actually makes mistakes. Without at least a dozen labeled examples with correct answers, you can't find the problem pairs and will have to guess by hand.

⚠️ The rules are specific to your taxonomy: a rule written for one set of categories doesn't carry over to another set. If the categories change, steps 1-2 have to be repeated.

The team tested the method on three large datasets: court documents (100 categories), scientific papers (134 categories, two levels of hierarchy) and online store products (351 categories, three levels). For each dataset they compared the standard approach (take the top-K similar categories and ask the model to choose) against adding "confusable partners" and generated rules.

The error analysis revealed something curious: when similar categories were added to the candidate list, the chance that the correct category was in the list at all rose almost to the maximum, but the model started making more mistakes in the choice itself among similar options. That's exactly why expanding the list alone isn't enough: you also need tie-breaker rules, which eliminated most of these new errors.

Separately, they tested knowledge transfer: rules generated by a large model delivered the same quality gain for models many times smaller and cheaper. In other words, the heavy lifting of writing the rules can be done once with a powerful model, and the classifying can then be done by a simpler, faster one. They also compared against fine-tuning: on clean data the fine-tuned models were more accurate, but with noisy labels or few examples the rules-based method proved more robust and lost far less accuracy.