TL;DR
DIVA (Decision-Informed Value-of-information Acquisition) splits the work between the model and the math. The model only answers short numeric questions: “what’s the base rate of a bad outcome?”, “how often does this check fire for bad cases versus good ones?”. Then Bayes’ rule and simple arithmetic work out which check could flip the decision and whether it pays for itself. When none of them pays off, the process stops.
If you just ask an LLM “what else should I check, and is it time to decide?”, it spends money on checks that won’t change anything: it grabs the most “interesting” one, not the one most useful for the decision. For example, it pays for a report where every possible result would leave the verdict exactly where it was. It keeps asking even when it already has enough data: in those cases it requests more 67% of the time with standard prompting and 84% of the time with “pick the most informative question.” Models choose checks by how much uncertainty they remove, not by whether they could push the decision from “yes” to “no.” Trying to make the model estimate in its head how the probability would shift after each outcome doesn’t help: the estimates come out inconsistent and are often worse than skipping the checks altogether.
The method in four steps: (1) the model outputs numbers, (2) Bayes recomputes the probabilities for every hypothetical outcome of every check, (3) for each check, compute the payoff for the decision minus its cost, (4) as long as some check has a positive net, request the best one; if none does, decide. The DIVA-U variant also averages over several versions of “how strong is this check,” because the model’s estimates are noisy.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: The model answers numeric questions → base rate of the bad outcome + P(result | bad), P(result | good) for each check
STEP 2: Bayes computes the probability of the bad outcome after EVERY possible result of every check → a “result → new probability” table
STEP 3: For each check: value for the decision − cost → net benefit
STEP 4: Best net benefit > 0 → run that check, add the result, repeat steps 2–4 (no more model calls needed)
Best ≤ 0 → stop, decide based on the current probability
Step 1 is a single model call. Steps 2–4 are arithmetic, and they have to happen outside the model’s “head”: in Python, a spreadsheet, or code.
Example
Task: An Amazon seller is about to place a $50,000 order with a new supplier. If all goes well, they make $12,000. If the supplier lets them down (defects, missed deadlines), they lose $40,000. The base rate of new suppliers in this niche who flake is 20%. Available checks: - a company background report from Dun & Bradstreet for $200; - a trial order for $1,500; - calling two of the supplier’s clients: free, but it eats an hour of your time (call it $100).
The question: which of these is even worth doing?
Prompt:
You are helping make a decision under uncertainty.
DECISION: place the order (ACT) or pass (SKIP).
Payoffs: if the outcome is good — gain g = $12000. If bad — loss l = $40000. Passing = 0.
Bad outcome: the supplier let us down (defects, missed deadlines).
CHECKS (result, cost):
1. Dun & Bradstreet background report: “red flags” / “no flags”, $200
2. Trial order: “passed” / “failed”, $1500
3. Calling the supplier’s clients: “good reviews” / “bad reviews”, $100
STEP 1. Don’t reason and don’t calculate. Answer with numbers only, in JSON:
- pi: probability of the bad outcome before any checks (as a fraction);
- for each check and each result: P(result | bad outcome) and P(result | good outcome).
Estimate each number on its own; don’t tune them to fit each other.
STEP 2. Then compute in Python:
- decision threshold b† = g / (g + l);
- V*(b) = max((1−b)·g − b·l, 0);
- for each check: P(result) = b·P(res|bad) + (1−b)·P(res|good);
the new probability b_v via Bayes’ rule for each result;
VOI = Σ P(result)·V*(b_v) − V*(b);
net benefit = VOI − cost.
Output a table: check | new probability for each result | does it change the decision | VOI | cost | net benefit.
STEP 3. If the best check has net benefit > 0 — name it and stop. I’ll send you the result, and you’ll repeat steps 2–3 with it included.
If net benefit ≤ 0 for every check — say “no checks needed” and give the decision based on the current probability.
Result: The model outputs JSON with the numbers plus a script with the table: for each check you can see whether any outcome flips the decision and whether that justifies the cost. Checks where both outcomes leave the decision unchanged get VOI = 0. The model names one check or tells you that you can decide right away. Once the result comes in, the loop repeats with the updated probability.
Why It Works
The LLM’s weakness. Models are bad at keeping probabilities consistent “in their head.” When asked to estimate how a probability would change after a hypothetical check result, their estimates don’t match Bayes’ rule. They don’t even match themselves: the outcome-weighted average drifts from the original probability by 6 percentage points on average. And the value of a check is a difference of maxima, so errors in that difference get amplified. Picking the best check from this kind of “imagination,” the model does barely better than chance.
The strength. The model knows how things usually go in the real world: how often suppliers like this flake, how a red flag in a background report relates to trouble down the line. You can pull that knowledge out with short numeric questions.
How the method plays the strength against the weakness. DIVA separates knowledge from computation: the model supplies the numbers, and the formula guarantees consistency. The core idea: a check is worth paying for only if at least one of its outcomes pushes the probability across the decision threshold. If it doesn’t, its value is zero, no matter how informative it is. The stopping rule follows directly: stop when no check pays for itself.
Control levers: - Payoffs g and l → set the decision threshold. The costlier a mistake, the lower the threshold and the more often checks pay off. - Check costs → in dollars, hours, days. A free check isn’t always worth running: if it can’t change the decision, it just eats your time. - Shrinkage of estimates → pull the “P(res|bad)” and “P(res|good)” pair closer together for each check. Otherwise model noise makes useless checks look strong. - Uncertainty about check strength (σ in DIVA-U) → needed when your task doesn’t tell you how reliable the checks are and you take that from the model’s head. If reliability is known (from statistics, a contract, past data), σ = 0.
Prompt Template
The paper itself contains no prompts. The template below is reconstructed from the method description: numeric questions, Bayes’ rule, the value of a check, and the stopping rule.
You are helping make a decision: {decision}.
PAYOFFS:
- ACT with a good outcome: gain g = {gain}
- ACT with a bad outcome: loss l = {loss}
- SKIP: 0
A bad outcome means: {what_counts_as_bad}.
ALREADY KNOWN: {known_facts}
POSSIBLE CHECKS (possible results, cost):
1. {check_1}: “{outcome_a}” / “{outcome_b}”, cost {cost_1}
2. {check_2}: “{outcome_a}” / “{outcome_b}”, cost {cost_2}
...
STEP 1 — NUMBERS. No reasoning. Return JSON only:
- "pi": probability of the bad outcome given the known facts (before any checks);
- for each check and each result: "P(result | bad)", "P(result | good)".
{if_check_reliability_is_given_in_the_task: These numbers are given above — just copy them over.}
Estimate each number separately, without fitting it to the others.
STEP 2 — CALCULATION (in code, not “in your head”). Write and run Python:
- threshold b† = g / (g + l);
- V*(b) = max((1 − b)·g − b·l, 0);
- b = current probability of the bad outcome (from pi and any check results already known);
- for each remaining check e:
P(v) = b·P(v|bad) + (1 − b)·P(v|good);
b_v = b·P(v|bad) / P(v);
VOI(e) = Σ_v P(v)·V*(b_v) − V*(b);
net_benefit(e) = VOI(e) − cost(e).
- output a table: check | b_v for each result | does b_v cross the threshold b† | VOI | cost | net benefit.
STEP 3 — STOPPING RULE.
- If the best check has net benefit > 0: name it and wait for the result. After the result comes in, update b and repeat steps 2–3.
- If every check has net benefit ≤ 0: say “no checks needed” and give the decision: ACT if b < b†, otherwise SKIP.
{optional, DIVA-U version — when check reliability isn’t known and comes from the model:
Before finalizing VOI, compute it for three versions of check strength: “weaker”, “as estimated”, “stronger” (pull the logits of the check’s two probabilities together or push them apart by about one standard deviation, σ = 1). Average the VOI with weights 1/6, 2/3, 1/6, and only then subtract the cost.}
What to plug in: {decision} — a binary yes/no choice. {gain} and {loss} — in money, hours or any other units, as long as the check costs use the same ones. {what_counts_as_bad} — the event you’re hedging against (a default, defects, a blown deadline, a misdiagnosis). {check_N} — real paid actions, each with a finite set of outcomes.
🚀 Quick start — paste this into the chat:
Here’s the DIVA template for “check more or decide now” problems. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the payoffs (gain and loss), the check costs, and the possible outcomes of each check. It needs these to compute the decision threshold and the value of each check. It will fill in the rest from the template.
Limitations
⚠️ Needs computation outside the model: the method works because Bayes’ rule is computed in code, not “in the LLM’s head.” In a chat without a code interpreter or a spreadsheet, the method loses its core. The authors didn’t test a manual in-chat version.
⚠️ Binary decisions with known payoffs only: the method is built for “act / skip” and two outcomes. For open-ended tasks (“what’s the best way to do this?”) and fuzzy goals, you first have to boil everything down to two outcomes and a dollar gain and loss.
⚠️ The model’s numbers are noisy: if check reliability isn’t specified anywhere and the model pulls it out of thin air, it can overrate useless checks. The remaining error comes mostly from the LLM’s estimates: with true probabilities the method does even better. That’s exactly what shrinkage and DIVA-U are for.
⚠️ DIVA-U can over-ask: on SEC stock-volatility data it sometimes “over-buys” checks and loses to the “check nothing” option. On these tasks, the DIVA version with “exact” estimates does even worse than “always check.”
⚠️ Tested on open models up to 72B: closed flagships (GPT, Claude, Gemini) weren’t tested. Even though the gain held across all 20 models, that doesn’t mean flagships can’t compute probabilities better on their own.
How They Tested It
The researchers built 12 decision-making environments. Eight of them are “knowledge-demanding”: credit and clinical tables (Taiwan, German credit, heart disease diagnosis, and others) plus S&P 500 filings for 2025–2026 (returns and volatility). Check reliability isn’t given there, so the model has to supply it from its own knowledge. The other four are synthetic (finance, credit, medicine, law); reliability is specified exactly, and the optimum is computed with dynamic programming. The team ran 20 open LLMs (from 0.6B to 72B) on these environments and compared DIVA against four prompting methods (Direct, CoT, SC, ReAct), “ask or act” approaches (Ask-P, VoI-Ask), information-gain selection (BED-LLM), PAL, and two reference points — “never check” and “always check.”
The headline finding: on real data, almost all prompting methods do worse than “check nothing”: their regret is several times higher. Models pick checks by informativeness, not by their ability to flip the decision. And they don’t know when to stop: with enough data already in hand, they ask for more 67% of the time (standard prompting) and 84% of the time (information gain), versus 11% for DIVA.
The ablation showed where the gains come from. Swapping decision value for “blind” information gain on the same numeric model raises regret from 0.83 to 5.03. Adding cost to information gain recovers almost all of it. So it’s not about fancy Bayes; it comes down to two ideas: compute whether an outcome changes the decision, and subtract the cost. Replacing the model’s estimates with true probabilities cuts losses from 0.83 to 0.31, which means the weakest link now is the LLM’s numbers.
Adaptations and Extrapolations
🔧 Technique: drop the math and keep only the question “could this result flip the decision?” → a quick check with no code
For simple tasks you can skip computing VOI and just ask the model: “What’s the worst and the best result this check could give? In each case, would my probability cross the threshold {threshold}? If not, the check is useless — don’t suggest it.” This is cruder than the original (it ignores outcome probabilities) and the authors didn’t test it, but it weeds out checks that clearly can’t change the decision. All you need is the threshold b† = g / (g + l).
Extrapolation: a stopping rule in an agent’s instructions (not tested by the authors)
If a tool-using agent decides on its own how many times to call paid or slow actions (search, a database query, an API), you can put the paper’s principle into the system prompt:
Before every paid tool call, answer in one line: “If the result is A, I’ll choose …; if it’s B, I’ll choose …”. If the choice stays the same for every outcome — don’t make the call. If you have enough data to decide — stop and act.This is a verbal version of the VOI idea, with no probability math. The authors only tested the full method with numbers and code, so check the effect on your own tasks.
Resources
- Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs — Seunghan Lee et al., LG AI Research.
- Code: github.com/seunghan96/diva
- Theoretical foundation: value-of-information theory (Howard, 1966; Raiffa & Schlaifer, 1961).
- Closest related work: BED-LLM (Choudhury et al., 2026), VoI-Ask (Dong et al., 2026), DeLLMa (Liu et al., 2025), Ask-P / FinRiskAtlas (Zhong et al., 2026).