Multiple-rollout cruxes: how to find the real forks in a model's reasoning by comparing its repeated answers

STEP 1: N independent runs of the same request (separate chats) → N answers with reasoning
STEP 2: Compare every pair of answers → a list of yes/no questions
        where the two sides disagree + each side's answer
        (compare even the pairs that reached the same conclusion)
STEP 3: Merge rephrasings of the same question → frequency of each crux

Steps 2–3 are separate LLM requests. Step 1 is several fresh chats or runs.

Task: The owner of a small machine shop in Ohio is deciding whether to finance a new CNC machine now or wait for the Fed to cut rates. He asks ChatGPT: "Will the Fed cut rates at its next meeting?" The answer sounds convincing, but he has no idea what it hinges on. If he simply asks "name the key factors," he gets a tidy list that won't match the points where the model actually changes its mind.

Prompt (step 1, repeat in 5 new chats):

You are an analyst. Estimate the probability that the Federal Reserve will cut
interest rates at its next meeting. Use the latest available data
(inflation, inflation expectations, the labor market, Fed statements, the lending market).
Reason step by step, then give a probability in percent and a conclusion.

Prompt (step 2, for each pair of answers):

Below are two independent analyses of the same question. Find the factual questions
on which they take OPPOSITE sides — even if their final probability
is similar.

For each question give:
- a yes/no phrasing;
- Analysis A's answer;
- Analysis B's answer.

Do not include questions on which both sides agree.

[Analysis A]
[Analysis B]

Prompt (step 3, once for all pairs):

Here is a list of yes/no questions from all the comparisons. Merge the ones that are
the same question worded differently: two questions are the same
if the same piece of evidence would confirm or refute both. Return
a list of unique questions and, for each, how many pairs it appeared in.

Result: First you get 5 answers with different probabilities and rationales. Then, for each pair, short lists of yes/no questions where the answers split — for example, on inflation expectations or on how the Fed will react to the jobs data. At the end you get a combined list of unique cruxes with their frequency. The most frequent ones are exactly what you should check yourself against primary data before deciding on the loan.

The LLM's weakness. When a model "names the important factors," it generates a plausible list from general knowledge of the topic. It doesn't know how its own future answers will behave and can't peek into its own distribution. That's why the factor list and the real cruxes barely overlap.

The strength. A model is good at comparing two concrete texts and spotting where they contradict each other. That's a reading task, not an introspection task.

How the method exploits this. It doesn't ask the model to guess where it will waver. It takes the real disagreements the model has already produced and only asks it to describe them. That also removes the need to brainstorm every possible factor up front: you're left with only the ones that actually surfaced.

Control levers: - Number of runs → new cruxes keep appearing roughly linearly with the number of runs. Five to seven runs already give you most of the cruxes nine would. - Compare pairs with the same conclusion → keep doing it. Two runs can reach the same verdict and still disagree on assumptions. - Merge strictness → "the same evidence confirms or refutes both" (strict) or "resolving one settles the dispute in the other" (lenient). Lenient means fewer duplicates, but coarser results. - Sources of variation → if you want different search results, let each run search on its own. If you want to isolate the model's own variation, give every run the same bundle of materials.

Prompt for step 2 (the core of the method):

<Role>
You are an editor looking for cruxes between two independent analyses
of the same question.
</Role>

<Task>
The question both analyses answered: {original_question}

Find the factual questions on which Analysis A and Analysis B take
OPPOSITE sides. Count disagreements even if their final conclusion
is the same.
</Task>

<OutputFormat>
For each crux:
- Question (yes/no phrasing): ...
- Answer A: ...
- Answer B: ...
If there are no cruxes, write "no disagreements".
Do not include anything the analyses agree on.
</OutputFormat>

<AnalysisA>{analysis_A}</AnalysisA>
<AnalysisB>{analysis_B}</AnalysisB>

Prompt for step 3 (merging):

Below is a list of yes/no questions from different comparisons: {crux_list}

Two questions count as one if the same piece of evidence
would confirm or refute both (the order of the answers doesn't matter).
Group the rephrasings and return a list of unique cruxes with the number
of comparisons each one appeared in.

Placeholders: - {original_question} — the text of the request you ran in step 1. - {analysis_A}, {analysis_B} — two independent answers with reasoning. - {crux_list} — the concatenated results of all comparisons.

🚀 Quick start — paste into a chat:

Here is a template for finding cruxes between a model's independent answers.
Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what decision you're making, which data should be shared across all runs, how many runs you can realistically do, and whether you want cruxes to be strict or broad. It needs this to phrase the original question and set the merge strictness, which together decide what counts as "the same crux."

⚠️ Asking up front doesn't work: "Name the key factors behind the decision" doesn't show where the model's answers actually diverge. The match with real cruxes is worse than random. Don't treat such a list as a risk map.

⚠️ One run isn't enough: Picking apart the reasoning of a single answer is noticeably better than asking up front, but it still misses a significant share of cruxes.

⚠️ Cost: The number of pairs grows fast (9 runs means 36 pairs). Doing it by hand in a chat is bearable for 4–6 runs; beyond that you need automation. For merging, the authors use a separate reranker model on a GPU — readers won't have that, so you'll have to do the merging with the LLM itself.

⚠️ Forecasts only: It was tested on binary forecasts (sports games, Metaculus events). Whether it transfers to open-ended tasks (writing, code) hasn't been tested.

⚠️ It doesn't make the answer better by itself: The method shows where the model is uncertain, but it doesn't make the forecast more accurate. You still have to check the cruxes against primary data.

⚠️ The paper text is incomplete: The fragment provided cuts off the single-rollout results for Metaculus and the remaining sections. Conclusions about them are taken from the abstract.

The idea is simple: if a question really is a crux, independent runs should give it different answers. The team took two sets of forecasting tasks: 15 events from the Metaculus platform and 59 games from the March Madness basketball tournament. On Metaculus each question got 30 runs, with variation coming both from the model and from web search results. For basketball they froze the data: first they collected three reports per team, and every run read the same reports. That left only the model's own variation.

Then they compared three approaches: asking for cruxes up front, extracting them from a single run, and finding them in pairs of many runs. The tricky part was that the same crux gets worded in different ways. To merge duplicates they used a reranker (a small model that scores how similar two texts are in a single pass) instead of an LLM judge: roughly 86 times cheaper. To make sure the result wasn't high by chance, they compared it against a permutation baseline: the same cruxes, but with the "up front" and "from runs" labels shuffled randomly.

The surprise: cruxes predicted up front matched the real ones less often than a random shuffle did. In basketball the match was about 13% versus a baseline of about 39%. 55 of 59 games came in below their own baseline. The practical takeaway: a model's list of "key factors" is not a map of its doubts.

Another surprise: the number of unique cruxes grows roughly linearly, even though the number of pairs grows quadratically. Cruxes repeat, so just five of nine runs already yielded about 61% of all unique cruxes.

🔧 Technique: require "an answer with both sides" → the crux is easier to check

The authors store each crux as a yes/no question with two answers. That's handy for verification: you immediately see what you need to find. Add to the output format: "What exactly would confirm answer A, and what would confirm answer B."

🔧 Technique: count frequency → set checking priority

After step 3, just sort the cruxes by the number of comparisons they appeared in. This is my add-on on top of the method, not from the paper: the more often a question surfaces, the more weight it gets when you check.

Extrapolation: a coding agent. As motivation, the paper mentions an agent that silently skips input data that breaks a pipeline. The idea applies here too: run the agent on the same task several times in clean sessions, then give another model pairs of diffs and this prompt:

Copy

Compare two solutions to the same task. Find the implicit assumptions on which
they disagree (what was skipped, what was treated as unimportant, how
malformed input data is handled). For each, give a yes/no question and the position
of each solution.

This carries the idea over to code; the authors did not test it.

• Paper: Can LLMs Anticipate Their Cruxes? (preprint, under review)
• Authors: David Reber, James Reber, Ari Holtzman, Victor Veitch
• University: University of Chicago (Department of Computer Science, Data Science Institute, Department of Statistics)
• References: the concept of a crux (Sabien, 2017); CruxBench (Dai et al., 2026); Qwen3-Reranker-8B (Zhang et al., 2025); the Metaculus platform