STEP 1: Independent labeling (each agent separately)

Prompt with few-shot examples → Categorical CoT format:
  Category_1: [reasoning] (confidence X/10) → Yes/No (confidence Y/10)
  Category_2: [reasoning] (confidence X/10) → Yes/No (confidence Y/10)
  ...
  Final answer: [list of categories]

STEP 2: Consensus check - All agents agree? → Those labels = the final answer - Disagreement? → Move to debate

STEP 3: Structured debate

Each agent receives: - Its own initial answer - The answers of all other agents

For each category the agent decides:

Category_X:
  Decision: KEEP / CHANGE
  Justification: [critique of the others' arguments + own reasoning]
  New confidence: Z/10

STEP 4: Final decision - Consensus reached? → Done - No consensus? → Majority vote OR a judge LLM reads the whole discussion and decides

You can run it in a single prompt (ask the model to simulate the agents) or through separate requests (different models or different temperatures).

Task: You're going through 50 startup founders' Demo Day pitches to investors. For each pitch you need to mark which topics the founder covered: Product/Technology, Team, Unit Economics, Market Size, Competitors, Traction. A single pitch can touch several topics or none at all (pure emotion and vision).

With a plain prompt like "identify the topics in this pitch," the model latches onto 1-2 standout moments and skips the rest. You need detailed labeling.

Prompt:

You are a startup-ecosystem analyst. Task: label a founder's pitch across 6 topics.

For EACH topic:
1. Give your reasoning: is this topic mentioned? Which phrases point to it?
2. State your confidence in the reasoning (1-10)
3. Final answer: YES or NO
4. State your confidence in the final answer (1-10)

**Topics to check:**
- Product/Technology
- Team
- Unit Economics
- Market Size
- Competitors
- Traction (growth metrics)

**Response format:**

Product/Technology:
Reasoning: [your analysis] (Confidence: X/10)
Final answer: YES/NO (Confidence: Y/10)

Team:
Reasoning: [your analysis] (Confidence: X/10)
Final answer: YES/NO (Confidence: Y/10)

[continue for all categories]

---

**Pitch transcript:**
{paste the pitch text}

---

Now create 2 more versions of this analysis with a slightly different focus. 

Then compare all 3 versions. For categories with disagreement, explain why and pick a final decision with a justification.

Result:

The model will produce three independent analyses, each with detailed reasoning and confidence for every topic. Then it shows a comparison: where all three agree (high reliability), where opinions split (with arguments for and against on each disputed topic), and a final decision explaining why that option won. The end product: a clean list of topics with confidence marks and no skipped categories.

In multi-label tasks, an LLM acts like a distracted expert: it sees the loud signals and misses the subtle ones. With general CoT the model generates text linearly: as soon as it's written about product and market, it has already "forgotten" to check team and competitors. Especially when there are many categories (6-8+) and the mentions are implicit.

LLMs are excellent at following structured instructions. Give a clear format like "Category A: reasoning → conclusion. Category B: reasoning → conclusion" and the model mechanically works through every item; skipping one is impossible. Categorical CoT turns free-form reasoning into a checklist.

CFD leverages this through forced completeness: the model must weigh in on every category. Per-category confidence shows where the model has doubts: an agent at 4/10 on "Team" sees the arguments of an agent at 9/10 and reconsiders. Debate surfaces missed signals: one agent caught a phrase about the team lineup, the others missed it, and through the exchange of arguments everyone sees that signal.

Prompt control levers:

• Number of agents (3 in the example) → drop to 2 to save tokens on simple tasks, raise to 4-5 if accuracy is critical
• Confidence scale (1-10) → swap for words ("high/medium/low") if you don't need numbers
• Consensus condition → you can use a "2 out of 3 agree" threshold instead of full consensus
• Debate instructions → add "focus on the disputed categories" to shrink the output

Option 1: Simulating multiple agents in a single prompt

You are a team of 3 independent experts. Task: label the data across the categories {category_list}.

**STEP 1: Independent analysis (each expert separately)**

Format for EACH category:
{Category}: 
  Reasoning: [analysis] (Confidence in reasoning: X/10)
  Final answer: YES/NO (Confidence in answer: Y/10)

Expert 1: {their analysis of all categories}
Expert 2: {their analysis of all categories}
Expert 3: {their analysis of all categories}

**STEP 2: Debate**

For categories where the experts DISAGREE, each expert gives:
- Decision: KEEP (I stand by my view) / CHANGE (I switch to my colleagues' view)
- Why: [critique of the others' arguments OR admission of my own mistake]
- New confidence: Z/10

**STEP 3: Final answer**

Consensus OR majority vote.
Final list of categories: [...]

---

**Few-shot examples:**
{2-3_examples_in_the_target_format}

---

**Data to label:**
{your_data}

Placeholders: - {category_list}: your labeling categories (for example: "Shipping, Quality, Price, Support, Interface") - {2-3_examples_in_the_target_format}: take 2-3 real examples, label them by hand, and show the full output format - {your_data}: the text to label (a review, post, transcript)

Option 2: Separate requests (for different models or temperatures)

Requests 1-3 (to each "agent" separately):

You are a data-labeling expert. Task: determine which categories are present in the data.

Categories: {category_list}

For EACH category, give:
1. Reasoning: are there signs of this category? (Confidence: X/10)
2. Final answer: YES / NO (Confidence: Y/10)

Few-shot examples:
{examples}

Data:
{your_data}

Run it 3 times at temperature 0.7-0.9 OR in different models (ChatGPT + Claude)

Request 4 (debate and consensus):

Here are 3 independent analyses of the same data:

Expert 1: {result_of_request_1}
Expert 2: {result_of_request_2}
Expert 3: {result_of_request_3}

For categories where the experts DISAGREE:
- Compare their arguments
- Decide who's right and why
- Give a final answer for the disputed categories with a justification

Final list of categories: [...]

🚀 Quick start: paste into the chat:

Here's a CFD template for multi-label labeling. Adapt it to my task: {describe your labeling task}. 
Ask me questions about the categories, data format, and examples.

[paste Option 1 above]

The LLM will ask which categories you need, what data you're labeling, and whether you have examples. It'll take the template's structure (Categorical CoT + debate + confidence), adapt it to your task, ask for 1-2 examples, and generate a working prompt.

The problem with the standard approach: an LLM generates text sequentially (token by token). In multi-label tasks that means the model reasons about the first 2-3 categories in detail, then "gets tired": it writes shorter and shallower and forgets to check the rest. It's not a bug; it's a property of autoregressive generation.

Categorical Chain-of-Thought breaks that sequence with external structure. Instead of "think freely and give me a list," we hand it a rigid template: "Category_1: ... Category_2: ... Category_N: ...". The model is forced to go through each one, or the format breaks. It's like a pilot's checklist: you can't forget to check the flaps if the list says "Flaps: ☐".

Fine-grained confidence solves the problem of hidden uncertainty. An overall confidence score (say, "7/10 confident in my answer") is useless: confident about what? All labels equally? Or 10/10 on one and 3/10 on another, averaging 7? Confidence for each category shows where the model is unsure. During the debate, a low-confidence agent is more receptive to arguments, which mimics real human consensus dynamics.

Structured debate isn't just "look at the others and update." It's a targeted review: an explicit KEEP/CHANGE decision plus a justification for each category. If an agent chooses CHANGE, it has to explain which colleague's argument convinced it. That surfaces hidden signals in the data: one agent noticed a non-obvious phrase, the others missed it, and through the debate everyone gains access to that insight.

Multi-agent diversity works as an ensemble of viewpoints. Even the same model at different temperatures focuses on different parts of the context. Agent 1 caught the explicit mention of "our team of three PhDs," Agent 2 focused on metrics and missed the team, Agent 3 picked up an indirect signal, "Mike and I started this in our dorm room" (also about the team). The debate combines these findings.

The consensus mechanism is tunable: - Strict (everyone agrees) → max accuracy, but more tokens - Soft (2 of 3) → faster, but some labels may be contested - With a judge → an arbiter (another LLM or the same one) reads the whole discussion, sees both sides' arguments, and decides

Levers for adapting it:

1. Number of categories: CFD is especially strong with 5-10+ categories. With fewer than 5 the gain is less noticeable (plain CoT will cope). Beyond 15 you risk bloating the context.

2. Number of agents: 2 agents = a quick debate, but it can deadlock (1:1 with no majority). 3 agents = the sweet spot (there's a majority). 4-5 agents = for critical tasks, but expensive.

3. Confidence scale: You can swap numbers for words ("sure / not sure / doubtful"): it works, but it's worse for automated processing. Numbers give you gradations.

4. Debate exit conditions: You can cap the rounds (max 2 rounds of debate) or use an agreement threshold (if 80%+ of categories match, that's enough).

5. The judge's role: The judge can be stronger than the agents (a bigger model) or the same model. You can give the judge access to the source data OR only to the debate transcript (blind judging).

⚠️ Token costs: The method takes 3-5x more tokens than a single request: initial generation (3 agents) + a debate round + possibly a judge. At large volumes (hundreds of examples) it gets noticeably pricier. Justified for critical tasks; for bulk labeling, weigh the costs.

⚠️ Needs a clear category taxonomy: CFD works when there's a predefined list of categories. If the categories are fuzzy ("text creativity," "emotional depth"), the method loses its edge. Best results come on objective, unambiguous labels ("is a location mentioned," "are there metric figures").

⚠️ For labeling, not generation: CFD is NOT a way to improve creative generation. It's a tool for classification and annotation. If the task is "write a text" or "come up with an idea," agent debates won't help (in fact, they may average things out and make them worse).

⚠️ Workflow overhead for one-off tasks: If you're labeling 5-10 examples by hand, CFD is overkill. The method pays off on recurring tasks with dozens to hundreds of examples, where consistency is critical.

⚠️ Consensus ≠ truth: All the agents can be wrong together. If the data truly has no signal for a category but the category wording is ambiguous, the agents may agree on the wrong answer. Few-shot examples are critical: they set the "calibration" for what counts as Yes and what counts as No.

The researchers tested CFD on two tasks:

1. Mental health: Labeling life events in Reddit posts: 8 event categories (career, relationships, health, education...). The CLPsych 2025 dataset.

2. Online safety: Labeling sharenting risks (risks of posting information about children) in Facebook posts: 5 types of risky behavior (child's location, health, age...).

Cat-CoT format from the paper:

For each life event category, provide:
1. Reasoning: [your analysis]
   Reasoning confidence: X/10
2. Final answer: YES / NO
   Answer confidence: Y/10

Career & Employment:
Reasoning: The post mentions "got promoted to senior..."
Reasoning confidence: 9/10
Final answer: YES
Answer confidence: 9/10

Relationships:
Reasoning: No mention of romantic or family relationships
Reasoning confidence: 7/10
Final answer: NO
Answer confidence: 8/10

[continue for all categories]

Experiment results:

The researchers used Qwen2.5-32B and Mistral-Small-24B as agents and Llama3.3-70B as the judge.

Fine-grained vs. coarse-grained confidence:

In the experiments, fine-grained confidence (separate for each category) consistently beat coarse-grained (one score for the whole answer) by 0.5-1.5% Macro F1. Especially noticeable in the multi-LLM setup.

Two ways to estimate confidence:

1. Self-verbalized: The model itself writes "Confidence: 7/10" in the text
2. Sampling-based: You generate 5 answer variants at different temperatures and measure their agreement with an NLI (Natural Language Inference) model

Sampling-based is slightly more accurate (+0.2-0.5%), but more expensive (5 generations just to estimate confidence).

Adaptation 1: Labeling e-commerce reviews

Task: An online retailer gets thousands of product reviews. You need to automatically tag the sources of dissatisfaction (there's no keeping up by hand): Shipping, Packaging, Product Quality, Matches Description, Price, Seller Support.

The problem with a standard prompt: "Identify the issues in this review" → the model sees a loud shipping complaint, writes about it, and forgets to check packaging (where there was also a signal: "the box was dented").

CFD prompt:

You are a team of 3 review analysts. Label the review across 6 aspects.

Aspects: Shipping, Packaging, Product Quality, Matches Description, Price, Seller Support

For EACH aspect:
- Reasoning: is a problem mentioned? Which phrases?
  (Confidence in reasoning: X/10)
- Final answer: PROBLEM / NO PROBLEM
  (Confidence: Y/10)

[2-3 example reviews with labels]

Review:
{review text}

Next: if the analysts disagree, debate the disputed aspects.
Outcome: the final list of problems.

How to use it: Run batches through the API, get consistent labels → filter out the critical reviews → prioritize responses.

Adaptation 2: Mapping competitive advantages in marketing copy

Task: You're analyzing competitors' landing pages in a SaaS niche. For each site you need to mark which USPs (unique selling propositions) they claim: Price, Speed, Integrations, 24/7 Support, Data Security, Ease of Use, Localization (multi-language support).

CFD prompt:

Task: determine which USPs are claimed on a competitor's landing page.

USPs to check:
- Price (cheaper / free tier)
- Speed
- Integrations with other services
- 24/7 Support
- Data Security
- Ease of Use
- Localization (multi-language support, regional specifics)

Cat-CoT format: for each USP → reasoning (confidence) → answer YES/NO (confidence).

Create 3 independent analyses. Compare the results. Debate the disputed USPs. Final list.

Landing page text:
{text copied from the competitor's site}

How to use it: You build a positioning map: you see where competitors are crowding in (everyone talks security) and where the gaps are (nobody emphasizes localization).

Adaptation 3: Checking content against a publication's editorial policy

Task: An online publication runs 20-30 articles a day. Editorial policy requires checking mandatory elements before publishing: Are quotes sourced, Are there unverified facts, Is the tone neutral, Is there unlabeled advertising, Are names and titles correct.

CFD prompt:

You are an editorial board of 3 editors. Check the article against the editorial policy.

Criteria (YES/NO for each):
- Quote sourcing: is a source given for every direct quote?
- Facts: does every verifiable claim have a source?
- Neutrality: is the text free of the author's value judgments?
- Advertising: if brands are mentioned, are they labeled as sponsored?
- Names: are all names and titles spelled correctly?

[Cat-CoT format with confidence]

Article:
{article text}

The editors check independently. Then they debate the disputed points. Outcome: READY TO PUBLISH / NEEDS EDITS (specify which).

How to use it: An automatic first-pass check before the final copyedit. It saves senior editors time on obvious mistakes.