CoSQ (Chain-of-Self-Questioning): the model checks its own knowledge first, then answers or declines

STEP 1: Break the question into required "information units" (facts needed to answer) → list of facts
STEP 2: Rate confidence on each fact (0.0–1.0) → numeric scores
STEP 3: Compute the average confidence → compare to a threshold (e.g., 0.90)
   IF average ≥ threshold → answer, using only facts with sufficient confidence
   IF average < threshold → decline to answer ("not enough certainty")

All three steps fit into a single prompt — no separate requests needed.

Task: You run a legal-advice newsletter on Substack, and you're asking Claude to answer a subscriber's question: "Is it true that if I pay a traffic ticket within 20 days, the fine drops by 50% — even if I've appealed the citation and the appeal hasn't been resolved yet?"

Prompt:

Answer the question below, but run a self-check before you do:

Question: {is it true that a traffic ticket gets a 50% discount if paid within 20 days, even while the citation is under appeal and the appeal hasn't been resolved}

STEP 1. List every fact you'd need to know to answer this question accurately.
STEP 2. For each fact, rate your confidence from 0 to 1 (how sure are you that this is correct and not confused with a similar rule).
STEP 3. Compute the average confidence across all facts.
STEP 4. If the average confidence is 0.90 or higher — give a precise answer using only the high-confidence facts.
If the average confidence is below 0.90 — do not answer substantively. Write: "Not confident enough to give a precise answer. I'd recommend checking your state's vehicle code or consulting an attorney," and specify exactly which facts raised doubt.

Result: The model will produce a list of 2-4 facts (e.g., "the 20-day rule applies," "the discount applies regardless of appeal status," "the discount is specifically 50%") and rate its confidence on each. If even one fact is shaky, the model admits it outright instead of handing over a falsely confident answer. That's especially valuable for tricky legal or medical details, where "almost true" is more dangerous than a plain "I don't know."

Plain CoT solves only one problem — how to reason — but not the problem of whether to answer at all. A model can build a flawless logical chain and still get the fact wrong at the end, because reasoning and factual knowledge are two different things.

An LLM's real strength is that it's decent at "sensing" when it's confusing similar facts or working off vague associations — if you explicitly ask it to rate that per item, rather than giving one blanket "I'm 80% sure." Splitting things into separate information units forces the model to check itself in more detail — instead of one overall confidence score, it gives you several granular ones, and the weak spot shows up immediately.

The method doesn't measure the model's actual probability of being right — it's not a calibrated percentage. It's simply a lever that shifts the balance between "answer more often, but be wrong more often" and "answer less often, but be more reliable."

Levers you can pull: - Confidence threshold (0.5–0.9) → a higher threshold means fewer answers, but the ones you get are more accurate. For high-stakes topics (medical, legal advice), set it to 0.85-0.90. For a casual group chat, 0.5-0.6 is fine. - What counts as "critical" — you can ask the model to sort facts into "critical" and "supporting" and gate only on the critical ones (the Critical-CoSQ variant from the paper) — this makes the model decline less often for no good reason. - How the refusal is worded — instead of a plain "I don't know," ask the model to specify exactly which fact raised doubt — that's a hint about what to go verify.

Answer the question, but run a self-check first:

Question: {question}

STEP 1. List the information units (specific facts) needed to answer this accurately.
STEP 2. Rate your confidence on each fact from 0 to 1.
STEP 3. Compute the average confidence across all facts.
STEP 4. If the average confidence is ≥ {threshold, e.g. 0.9} — give an answer using only the high-confidence facts.
If the average confidence is below {threshold} — do not answer substantively. State that you're not confident and specify which facts raised doubt.

Fill in: {question} — your actual query, {threshold} — anywhere from 0.5 (lenient mode, answers almost always) to 0.9 (strict mode, declines more often but lies less often).

🚀 Quick Start — paste this into the chat:

Here's a self-check-before-answering method template (CoSQ). Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask you which confidence threshold to use and whether to split facts into critical/supporting — because that determines how often the model ends up declining to answer.

⚠️ The cost of refusal: the method sometimes declines to answer questions that plain CoT would have answered correctly. In the study, roughly two-thirds of all refusals were on questions where the baseline method gave the right answer. In other words, you're trading coverage for reliability.

⚠️ Not a calibrated probability: the confidence number (0.7, 0.9) isn't the model's actual probability of being correct — it's just a tuning knob. Don't treat it as a scientific accuracy metric.

⚠️ Limited test domain: the method was tested on factual multiple-choice questions and short Q&A. For creative, analytical, or subjective tasks (business advice, evaluating a piece of writing), the "gate on facts" logic may not even apply — there are no clean "information units" to work with.

⚠️ Extra tokens/requests: the method requires more reasoning steps per question — meaning it's slower and pricier than a direct answer. For simple questions, it's overkill.

The researchers took 817 questions from TruthfulQA — a benchmark specifically designed with "trap questions" that both humans and models tend to get wrong due to stereotypical thinking. They ran the questions through eleven different models (ranging from Llama to GPT-5.5 and Claude) under seventeen different conditions: plain direct answering, CoT, and three CoSQ variants at five different confidence thresholds.

A key design detail: the order of answer choices (A/B/C/D) was deliberately rebalanced so the model couldn't guess based on position rather than content — a common trick that skews benchmark results.

The surprising part wasn't that CoSQ worked (that was expected) — it was that plain CoT didn't help at all compared to answering directly with no reasoning at all: the error rate stayed nearly the same (13.1% vs. 12.8%). In other words, "think before you answer" by itself doesn't prevent factual errors — you need a separate mechanism to decide "should I answer at all." They additionally tested on 300 questions from Natural Questions, and the effect held up on a different type of question, suggesting the finding isn't specific to just one benchmark.