Metacognitive Harness: Steering Retries with the Model's Own Confidence Signals

Task: You're preparing an investor pitch for an EdTech startup, an online tutoring marketplace. In 3 minutes you need to explain why you won't get crushed by competition from Wyzant and Varsity Tutors.

Prompt:

I'm going to give you a task. Before answering, complete three steps strictly in order.

---

STEP 1, ONLY THIS:
On a 0.0–1.0 scale, rate how confident you are that you can give an accurate and in-depth answer to the question below. Don't reason, don't start answering: just a number and one sentence on why.

Question: "How can a small EdTech startup (a tutoring marketplace, $15K in monthly revenue) convince an investor in a 3-minute pitch that it can withstand competition from Wyzant and Varsity Tutors?"

STEP 1 format:
FOK: [number]
Reason: [one sentence]

---

STEP 2, after you've written the FOK:
Answer the question in detail. Give concrete arguments and a pitch structure.

---

STEP 3, right after the answer:
On a 0.0–1.0 scale, rate how confident you are that your answer is accurate, useful and applicable to this specific situation. State the main reason for doubt (if any).

STEP 3 format:
JOL: [number]
Doubt: [one sentence]

Result:

The model will first give a FOK rating, say 0.7, with a note like "I don't have data on the specifics of this market". Then a detailed answer with arguments. Then a JOL, say 0.6, with a clarification like "I don't know Wyzant's current metrics or the startup's actual niche".

If the JOL is below 0.75, you paste in the second prompt (template below). The model will try again knowing only the result and the reason for doubt, without repeating its old reasoning. The final attempt is usually more accurate and more specific.

The LLM weakness: a model generates text on autopilot, one token after another, never looking back. It doesn't stop halfway and say "wait, something's off". Without an explicit request for self-assessment, that signal stays inside and has no effect on behavior.

The LLM strength: if you explicitly ask "rate your confidence", the model gives a signal that correlates with actual accuracy. This was confirmed on nine different models: the higher the FOK/JOL, the more correct answers. The model doesn't lie about its uncertainty; it just doesn't use that knowledge on its own.

How the method uses this: it breaks the passivity. Instead of confidence just sitting somewhere inside, we pull it out as an explicit number and use it as a command: "low confidence = try again". The key trick: on a retry, the model sees only the result of the previous attempt and the reason for doubt, not the reasoning itself. This avoids the "anchoring to the old mistake" effect: the model starts fresh, knowing exactly what bothered it, but without copying the failed line of thinking.

Control levers: - JOL threshold (e.g. 0.75): below it → retry. Lower the threshold for simple tasks, raise it for critical decisions - Number of attempts: in the study the average was 2.4. Cap it at 2–3 to save money - Retry context: pass only the JOL + the reason, not the full reasoning. This is essential - Aggregation: with several attempts, strip the JOL ratings from the context of the final pick, so the model judges by quality of reasoning, not by self-confidence

I'm going to give you a task. Complete exactly three steps.

---

STEP 1, only this:
On a 0.0–1.0 scale, rate your confidence that you'll give an accurate answer to the question below.
Don't reason, don't start answering: just a number and one sentence.

Question: {your_question}

FOK: [number]
Reason: [one sentence]

---

STEP 2:
Answer the question in detail.

---

STEP 3, right after the answer:
On a 0.0–1.0 scale, rate your confidence that your answer is accurate and applicable.
State the main reason for doubt.

JOL: [number]
Doubt: [one sentence]

If JOL < {threshold}, the retry prompt:

Your previous answer: {previous_answer}
Your confidence was: {JOL}
Reason for doubt: {doubt}

Try answering again, keeping these doubts in mind.
Don't copy your previous reasoning; start from a clean slate.

After the answer, give again:
JOL: [number]
Doubt: [one sentence]

If there are several attempts, the final pick:

I have {N} candidate answers to the question: {question}

{Attempt 1: reasoning + answer}
{Attempt 2: reasoning + answer}

Compare the candidates by quality of reasoning and accuracy of the answer.
Pick the best one and explain why.

Placeholders: - {your_question}: the task itself - {threshold}: your trust bar (I recommend 0.75–0.8 for hard tasks) - {previous_answer}, {JOL}, {doubt}: taken from the STEP 3 response - {N}, {Attempt 1...}: for the final aggregation

🚀 Quick start: paste this into your chat:

Here is a Metacognitive Harness template. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which question to pose, which JOL threshold to use, and what counts as a "good enough" answer, because without that it won't know when to stop and what to revisit.

⚠️ Not all models are equally self-aware: In the study, only Claude Sonnet-4.6 passed the full diagnostic: its FOK/JOL genuinely correlate with accuracy and are well calibrated. Gemini-Flash could tell right answers from wrong ones, but its numbers were unreliable as threshold values. Gemma-4 gave almost no useful signal. Before using this seriously, test it: ask the model to rate itself on 10 questions with known answers and see whether its confidence matches reality.

⚠️ Not for simple tasks: The method increases the number of requests (2.4x on average). For obvious questions that's wasted effort. Use it on tasks where correctness really matters: analysis, strategy, complex conclusions.

⚠️ Doing it by hand gets tedious at volume: If you need to process 50 similar tasks in a row, it's slow without automation. The method works best for one-off, high-stakes requests in a chat.

⚠️ FOK without reasoning goes against the model's nature: The STEP 1 prompt must explicitly forbid any calculation before the rating. If the model starts solving before the FOK, the signal loses its meaning: it'll be rating its confidence while already knowing the answer.

A team from UC San Diego took nine different LLMs and gave them one test: rate your confidence before and after answering, and let's see whether it matches actual accuracy. The result was surprising: most of them have the signal. Models with high FOK/JOL really did answer more accurately. But here's the striking part: that signal had no effect on their behavior. Confident and unconfident questions got the same amount of "thinking effort". The gap between knowing and acting was obvious.

So the researchers built a control loop: they fixed a single base Claude Sonnet-4.6 (no weight changes) and added only a control layer: FOK before, JOL after, and an SVM classifier to decide "trust or retry". They tested it on three benchmarks: expert STEM questions (HLE-Verified), hard coding problems (LiveCodeBench v6), and multimodal tasks (R-Bench-V), 1,859 examples in total. Importantly, the diagnostic set (100 questions for training the SVM) was kept strictly separate from the test set to avoid data leakage.

The bottom line: +8.6 percentage points of accuracy at an average cost of 2.4 attempts instead of one. The biggest gain was on hard coding problems (+19.6 on the Hard split). A side-by-side comparison showed this isn't just "more attempts": plain sampling of 4 attempts delivered less at a higher cost. The main contribution is the ability to stop early when confident and try again when not.

🔧 Technique: FOK without a number → qualitative self-assessment

If the 0–1 numeric scale seems unreliable for a particular model, swap it for categories:

Copy

Before answering, rate yourself: "Confident / Fairly confident / Not confident / Don't know"
and explain why in one sentence.

Models sometimes give a more honest rating through categories than through numbers, since there's less pull toward a "respectable" number.

🔧 Extrapolation: Metacognitive Harness as a filter for important decisions

Use FOK/JOL not for retries but as a flag: if the JOL is below the threshold, that's a signal to check the answer some other way (search, an expert, a different model). Not an automatic retry, but a recommendation: "this answer needs verification".

Copy

After the answer, give:
JOL: [0.0–1.0]
Doubt: [what you're unsure about]
Verification recommendation: [what to check, where to confirm it]

Paper: LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling

Authors: Qi Cao, Peijia Qin, Yufan Wang, Shuhao Zhang, Pengtao Xie — University of California, San Diego

Theoretical foundation: Nelson–Narens metacognition framework (cognitive psychology)

Benchmarks: HLE-Verified, LiveCodeBench v6, R-Bench-V