KWBench: Models Know the Theory but Don't Recognize the Problem — and How to Close the Gap with a Prompt

This is a research finding, not a ready-made technique. A two-step approach follows from it:

STEP 1: Diagnosis → the model names the problem type(s) from the 6 structures + what that says about the situation
STEP 2: Analysis → a solution through the lens of the structures it found

Both steps go in one prompt.

The study formalized six patterns that keep recurring in professional situations. This is your diagnostic tool:

Task: You're the founder of a B2B service. A major e-commerce platform has sent you an acquisition offer: $18 million, an answer required within 72 hours, 60 days of exclusivity (you can't talk to other buyers). You want to know — is this a good deal or a trap?

Prompt:

I received an acquisition offer for my company. Here are the terms:
— Price: $18 million
— Deadline to respond: 72 hours
— Exclusivity: 60 days after acceptance
— Structure: 70% cash upfront, 30% earnout over 2 years
— Buyer: a large platform, our potential distributor

Before giving a recommendation, complete two steps:

STEP 1. Check which of these structures are present in the situation:
- Signaling game — the structure of the terms reveals hidden information about the buyer
- Principal-agent — whose interests do the advisors or intermediaries in the deal serve?
- Mechanism failure — which deal rules work against me structurally?
- Coalition dynamics — are there coordinated parties on the buyer's side?
- Strategic interdependence — how does my acceptance or refusal change the game for others?
- Strategic omission — what's missing from the offer that should be there?

For each structure found — one sentence: what exactly it tells us about the real situation.

STEP 2. Through the lens of the structures found: what do the deadline and exclusivity actually signal about the buyer's position? What should I do in the next 72 hours?

Result: In step 1 the model will identify a signaling game (hard deadline + exclusivity = the buyer wants to cut off price discovery — meaning they know the price is below the real value) and strategic omission (no justification for the urgency, no mention of synergies, no clarity on what happens to the team). In step 2 — a concrete plan: how to use the 72 hours to send parallel signals to the market, and how to read exclusivity as a pressure tool rather than a standard term.

Without step 1, the model will most likely crunch the multiples and say "the price looks in line with the market."

An LLM is a precise executor and a weak diagnostician. When the task is spelled out ("value it with a DCF"), models do it well. When the task isn't spelled out — when it has to be inferred from raw data — the model tends to grab the strongest pattern from its training. "Acquisition offer → financial analysis" is a strong pattern. "Offer with a deadline → signaling game → what does this reveal about the buyer" is a rare pattern that requires an intermediate step the model doesn't take on its own.

Knowing ≠ applying in context. Models carry game theory inside them, but they generate text by the most likely next step — and most training examples go "got an offer → run the analysis," not "got an offer → first identify the type of game." An explicit diagnostic step in the prompt switches the generation mode.

You replace an automatic choice of frame with a deliberate one. When the prompt says "check whether there's a signaling game," the model is already tuned to that mode of analysis. The list of six structures works like a pre-flight checklist: the pilot already knows the procedures, but the checklist keeps them from skipping the obvious.

Control levers: - The list of structures → add your own patterns for your industry or drop irrelevant ones - "One sentence per structure" → sharpens the diagnosis and keeps the model from rambling - Splitting into two steps → remove step 1 and you're back to the old behavior - A third "devil's advocate" step → ask what was missed in the diagnosis — the model double-checks itself

Here is the situation: {situation_description — paste the raw data: an email, terms, 
a proposal, a conflict, a contract}

Before giving a recommendation, complete two steps:

STEP 1. Identify which of these structures are present:

- Signaling game: the other side's terms or behavior reveal their hidden information
- Principal-agent: whose interests does the advisor or counterparty actually serve?
- Mechanism failure: the system of rules creates the wrong incentives, rather than a broken process
- Coalition dynamics: there are coordinated parties I'm treating as separate
- Strategic interdependence: my decision changes the optimal strategy of others
- Strategic omission: what's missing from the materials is as significant as what's there

For each structure found — one sentence on what it says about the situation.

STEP 2. Through the lens of the structures found: {desired_output}

What to fill in: - {situation_description} — raw data without interpretation: an offer, an email, contract terms, a team situation, a partner's proposal - {desired_output} — a concrete output: "what to do in the next 24 hours," "what to watch for before signing," "how to build a negotiating position," "which question to ask first"

🚀 Quick start — paste into the chat:

Here is a template for two-step analysis of professional scenarios. 
Adapt it to my situation: [describe your situation].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask for the scenario details and the output format you need — because without them it can't fill in step 2 properly. It will keep the diagnostic structure and tailor the recommendation format to your task.

⚠️ Diagnosis raises the odds, it doesn't guarantee anything. Even with an explicit recognition step, the model can still misdiagnose. The method improves the result — it doesn't make it infallible.

⚠️ High stakes need verification. In M&A, legal disputes, medical decisions — check the model's diagnosis with a domain specialist. The method helps you frame the right question for the expert; it doesn't replace the expert.

⚠️ Not for straightforward tasks. If the task is simple ("write an email," "run the numbers"), the two-step approach is overkill. The method works where a false frame is sitting on the surface.

⚠️ No single model dominates at recognition. The overlap of correct answers between the top 2 models is only ~32%. If you need high confidence, ask several different LLMs and compare their diagnoses.

A researcher from Clio AI collected 223 real professional scenarios from six fields: M&A, contract negotiations, clinical pharmacy, organizational politics, fraud analysis, and incentive design. 185 tasks came from real incidents, 38 were reworked tasks from existing benchmarks (WildBench, ProfBench and others). Each scenario was formalized as one or more of six game-theoretic structures, and each was annotated: the key insight, the predicted mistakes, and what exactly the model should notice.

The clever part of the design: all materials went through "hardening" — deliberate removal of hints. If a source document contained a phrase like "The terms are surprisingly lenient," it was rewritten neutrally. If an analyst had already calculated a multiple, it was removed and only the raw numbers were left. The signal is always there but never highlighted — you have to spot it yourself.

Scoring used a three-tier rubric with a hard mandatory gate: fail the mandatory criterion and you get zero for the whole task, even if everything else is brilliant. The criteria aren't built around "the right answer" but around "avoiding the dangerous wrong one" — specific traps that even good professionals fall into. They tested 16 models from 10 organizations, and every model always had a code interpreter — so a failure couldn't be blamed on arithmetic.

The most telling finding: models that scored zero on recognition still passed about half of the optional criteria. Structured, persuasive, detailed text — about the wrong problem. That confirmed the main thesis: the failure isn't in execution, it's in recognition.

🔧 Add a "devil's advocate" step

After the two basic steps, you can add a third:

STEP 3. Play the role of a seasoned advisor who looks at my analysis and says: 
"You missed something important." What exactly, and why does it change the recommendation?

This forces the model to recheck its own diagnosis — and often uncovers a layer that was missed.

🔧 Add industry patterns from your field

The six base structures are universal, but you can add domain-specific ones to step 1:

For founders: "anti-dilution trap" — next-round terms that dilute your stake more than the term sheet suggests
For HR: "survivor bias in the data" — people who left don't answer surveys, so the data skews toward those who stayed
For sales: "lock-in via the first discount" — a low entry price as a switching-cost strategy

🔧 Routing between models

The study showed that the top 8 models together solve 50.7% of tasks, while the best single model solves only 27.9%. The overlap of correct answers is minimal — different models recognize different structures.

Practical takeaway: give the same ambiguous situation to two or three different LLMs. If the diagnoses diverge, that's a signal the situation really is ambiguous and calls for caution or an expert. If they match, your confidence goes up.

KWBench: Measuring Unprompted Problem Recognition in Knowledge Work
Ankit Maloo — Clio AI ([email protected])

Dataset: https://huggingface.co/datasets/clio-ai/kwbench
Leaderboard: https://kwbench.github.io
Eval Harness: https://github.com/ankitmaloo/fasteval

Theoretical foundations: Spence, 1973 (signaling games) · Jensen & Meckling, 1976 (principal-agent) · Hurwicz, 1960 (mechanism design) · Nash, 1951 (strategic interdependence) · Shapley, 1953 (coalitional dynamics) · Akerlof, 1970 (information asymmetry)