AuditForecast: splits a forecast into a precise base rate and separate adjustments, instead of blending everything into one big guess

STEP 1 (Base rate): 
Define a reference model / base statistic for the question → p0 (a number from 0 to 1)

STEP 2 (Find factors outside the base):
Find factors that the model from Step 1 does NOT account for
→ check "is this really outside the base?" (scope check)
→ check "does this factor duplicate one already counted?" (independence check)
→ list of distinct factors with supporting evidence

STEP 3 (Score and multiply):
Assign each factor a multiplier:
  >1 = pushes toward "YES"
  <1 = pushes toward "NO"  
  =1 = neutral
→ Final odds = (base odds) × (multiplier1) × (multiplier2) × ...
→ Convert the odds back into a probability

All three steps can be run in a single prompt — the LLM is capable of reasoning through this structure step by step, no external tools required.

Task: Estimate the probability that the Fed will cut interest rates at its next meeting — the classic question that floods every finance Twitter/X feed before each meeting.

Prompt:

Estimate the probability that the Fed will cut interest rates at its next meeting.
Follow this structure:

STEP 1. BASE RATE:
Define a reasonable base model — for example, the analyst consensus forecast,
or the historical frequency of rate cuts under similar inflation conditions.
Give the base probability p0 and explain what it's based on.

STEP 2. FACTORS OUTSIDE THE BASE:
List factors this base estimate does NOT yet account for —
for example, recent Fed official statements, fresh inflation data,
the dollar index movement over the past week.
For each factor, check:
— Is this really not already covered by the base estimate?
— Does this factor duplicate one already listed?

STEP 3. MULTIPLIERS AND FINAL RESULT:
Give each factor a multiplier: 
>1 if it increases the probability of a cut, <1 if it decreases it, 
=1 if neutral. Justify each number.
Convert the base probability into odds (p/(1-p)), multiply by all multipliers, 
convert back into a probability. Show the full calculation.

Result: The model produces a structured report in three parts: first the base probability with justification (e.g., "analyst consensus puts it at 30%"), then a list of 2-4 factors with explicit numeric multipliers and an explanation for each, and finally an open calculation (odds × multipliers = final odds → final probability). You get more than just a number — you see the entire path to it, and you can push back on any single step.

When an LLM is simply asked to "give a probability," it blends all the arguments into one vague impression — much like a person who weighs a pile of pros and cons in their head and names a number that just "feels right." The problem is that such a number is unstable and opaque: ask again and you'll get a different one, and no one can explain why.

What LLMs are actually good at is narrow, concrete sub-tasks: naming a base statistic, scoring one factor in isolation, comparing it against factors already named. Each of these mini-tasks isn't a "vague impression" — it's a concrete decision with clear inputs.

The method takes this strength and builds a process around it: instead of one big fuzzy estimate, you get a chain of small, precise decisions (which base? which factor? which multiplier?) that then get mechanically multiplied together. A mistake in one step is easy to spot and fix; a mistake baked into an overall "impression" is impossible to even detect.

Control levers: - Rigor of the base model — for simple questions, "historical frequency of similar cases" is enough; for complex ones, ask the LLM to explicitly spell out a formula or model. - Number of factors — cap it at 2-3 for a quick estimate; allow more for a high-stakes decision. - Multiplier scale — set your own: for example, "use only 0.5/1/1.5/2/3" instead of arbitrary numbers — this makes it much easier to compare factors against each other. - Independence check — if you notice two factors are essentially describing the same thing (say, "rising inflation" and "the Fed's rhetoric about inflation"), explicitly ask the model to merge them — otherwise the probability can get skewed in one direction.

Estimate the probability of the following event: {question about a future event}.
Follow this structure, don't blend the steps together:

STEP 1. BASE RATE:
Find a suitable base model or reference class of similar cases 
for this question. Give the base probability p0 and explain what it's based on.
If {additional data} is available, use it in the calculation.

STEP 2. FACTORS OUTSIDE THE BASE:
List factors the base estimate hasn't accounted for yet.
For each factor, check:
— Is this really not already part of the Step 1 base estimate?
— Does this factor duplicate one already listed?

STEP 3. MULTIPLIERS AND FINAL RESULT:
Give each factor a multiplier to the odds:
>1 pushes toward "yes," <1 pushes toward "no," =1 neutral. Justify the numbers.
Convert p0 into odds, multiply by all the multipliers, convert back into a probability.
Show the full calculation explicitly.

Fill in: {question about a future event} — a specific binary event with a date attached (not "will the company be successful" but "will the company turn a profit by year end"); {additional data} — any numbers you have on hand (exchange rates, statistics, reports).

🚀 Quick start — paste this into the chat:

Here's a template for AuditForecast structured forecasting. Adapt it to 
my task: {describe your question about a future event}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which base model makes sense for your specific question and whether you have any fresh data — because Step 1 can't be done accurately without that. After clarifying, it will walk you through all three steps using your actual data.

⚠️ You need an explicit numeric or reference anchor. The method works worse for questions with no natural base model — for example, subjective judgments ("will this movie be good"). In such cases, Step 1 degenerates into the same kind of guessing the method is supposed to eliminate.

⚠️ The risk of double-counting rests on the quality of the check. If the LLM misses that a Step 2 factor is already "hidden" inside the Step 1 base model, the multiplier will double-count the same argument and inflate confidence. Explicitly ask the model to re-verify this for every factor.

⚠️ The accuracy of the base rate depends on how current the model's knowledge is. In the study, the model pulled up-to-date data via real-time web search. In an ordinary chat without internet access, the base probability may rest on stale data — for fast-moving topics (exchange rates, breaking news), feed it fresh information yourself.

AuditForecast: A Structured Scaffold for Agentic Forecasting — Yuanpu Cao, Yongkang Du, Yurui Chang, Lu Lin, Jinghui Chen, Pennsylvania State University. Tested on live forecasting platforms Polymarket and Kalshi (via ProphetArena), plus the FutureX benchmark, using GPT-5.4, GPT-5.4-mini, and Gemini-3-flash.