PASSING: getting the model to explicitly check your expertise level before answering

PASSING is a method that stops the AI from guessing your knowledge level based on how you phrase a question, and instead has it explicitly ask 1-2 short clarifying questions before answering. The model checks you against a structured checklist of expertise dimensions instead of relying on a gut feeling.

Turns out LLMs are pretty bad at figuring out a user's knowledge level from a single question — and it's not just a minor quirk. Ask the model the same query five times and it gives a different verdict every time: sometimes you're a beginner, sometimes an expert. Worse, the model is easily fooled by terminology: if you copy-pasted a complex question straight from an article online, the model will assume you're an expert even if you have zero clue about the topic. This leads to a familiar pain point: the answer is either buried in jargon or dumbed down into obvious platitudes — because the model miscalibrated.

PASSING fixes this with a two-step clarification loop: first the model decides which aspect of your expertise to probe ("what to ask"), then it phrases the question so you won't feel embarrassed answering honestly ("how to ask"). After 1-2 such rounds, the model fills out an expertise profile across five dimensions and only then delivers a final, tailored answer.

Method Blueprint

STEP 1: Model builds context of the request (question + draft answer) → internal context, no output
STEP 2 (loop, 1-2 rounds, in dialogue with the user):
   2a. "What to ask" → pick which expertise aspect to probe
   2b. "How to ask" → phrase the question for that aspect
   2c. Question to the user → user responds
   2d. Update the expertise profile based on the answer
STEP 3: Assess final level (novice → beginner → intermediate → advanced → expert)
STEP 4: Final answer, tailored to that level

Step 1 and the assessment in step 3 happen "inside the model's head"; step 2 requires an actual back-and-forth with the user.

Example in Action

Scenario: You run an investing newsletter on Substack. A subscriber writes: "Explain what portfolio diversification and asset rebalancing mean." It's unclear whether this is a student who just opened a brokerage account, or a trader throwing around terms to sound smart while actually being shaky on the basics.

Prompt:

A subscriber sent me this question: "Explain what portfolio 
diversification and asset rebalancing mean."

Before answering, figure out their expertise level — don't guess 
from how the question is phrased, clarify it with 1-2 short, 
friendly questions.

Check against these aspects (pick the 1-2 most important for this question):
- KNOWLEDGE: do they just know facts, or already understand when to apply them, 
  or act on intuition?
- PERCEPTION: do they rely on general rules, or on context and their own experience?
- DECISIONS: do they reason step by step, or jump straight to conclusions?
- INVESTMENT: is this topic abstract to them, or is their own money on the line?

Ask no more than 2 clarifying questions, in plain language, without condescension. 
Once you have the answers, determine the level (novice/beginner/intermediate/
advanced/expert) and write a newsletter post suited to exactly that level.

Result: The model first asks something like "Do you already have a portfolio, or are you just planning to build one?" and "What matters more to you — not losing money, or beating the market?" Once you (or the subscriber) answer, it produces a post tailored either as a beginner-friendly primer or as a more technical breakdown with specific ratios for an advanced reader.

Why It Works

LLMs judge your expertise from surface-level signals — fancy words in the question, sentence length, sentence structure. That's trivially easy to game: just copy a smart-sounding question from Google. And even without gaming it, the model's estimates jump around from run to run, because there's no direct signal — just a guess.

On the other hand, the model is genuinely good at asking clarifying questions and adapting its tone when directly asked to. PASSING leverages that strength: instead of guessing from the text of the question, it gets a direct signal from the user's answers to 1-2 targeted questions.

Control levers: - Number of clarification rounds (1-2 rounds) → drop to 1 for quick chats, bump to 3 for complex topics where mistakes are costly - List of aspects (knowledge/perception/decisions/investment) → trim to the 1-2 most relevant for your specific topic, don't ask about all of them at once - The "without condescension" instruction → remove or strengthen it depending on whether you want blunter or gentler questions - Final levels (novice...expert) → swap for your own scale if you need to gauge something else, like emotional state or urgency, instead of expertise

Prompt Template

I'm going to ask a question about "{topic}". Before answering, figure out 
my expertise level specifically on this question — don't guess from 
how I phrase it, clarify it with 1-2 short questions.

Assess me on these aspects (pick the 1-2 most important for this topic, 
don't ask about all of them at once):
- KNOWLEDGE: do I just know the facts, or do I also understand when 
  to apply them, or do I have an intuitive feel for it?
- PERCEPTION: do I rely on general rules, or on context and experience?
- DECISIONS: do I reason step by step, or jump straight to intuitive conclusions?
- INVESTMENT: is this topic abstract to me, or is it personally important?

Rules:
1. First decide WHAT to ask — which aspect is most important to clarify 
   for this question: "{question}".
2. Phrase the question simply, with no pressure — so I don't feel 
   embarrassed answering honestly even if I don't know much.
3. Ask no more than {number} clarifying questions.
4. After my answers, determine the level (novice / beginner / 
   intermediate / advanced / expert) and give an answer suited to 
   that level.

Question: {question}

Fill in the topic of conversation in {topic}, the actual question in {question}, and the desired number of clarification rounds in {number} (usually 1-2).

🚀 Quick start — paste into the chat:

Here's a template for checking my knowledge level before you answer. 
Adapt it to my situation: {describe your situation}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what your topic is and what level of detail matters — since that determines which of the 4 aspects in the framework are worth checking first.

Limitations

⚠️ An extra step for simple questions: if you need a quick factual answer ("what's 15% of 200,000"), checking expertise is friction, not value. This method is built for explanatory, teaching-style queries.

⚠️ An abstract framework doesn't guarantee good questions: the five aspects of the Dreyfus model are universal, but for a narrow topic (say, a specific legal case) the model might land on a not-so-relevant clarifying question. Check the quality of the questions rather than trusting the framework blindly.

⚠️ Requires at least one extra exchange: if you need an answer right here, right now, with no back-and-forth, this method isn't for you — it's built for multi-turn conversation.

How It Was Studied

The researchers first checked how consistently LLMs (GPT-4o and others) judge a user's expertise from the same question — asking the same query five times in a row. Result: even the best methods (Chain-of-Thought, Self-consistency, ExpertPrompting) showed low agreement across runs — the model kept "changing its mind" from one run to the next. Especially telling: ExpertPrompting, a method specifically designed to assess expertise, often failed to produce a judgment tied to the specific question at all — it confused a user's general savviness with their level in a specific narrow topic.

Next, the team trained probing strategies offline through "self-play": simulated users with different personality traits and knowledge levels answered clarifying questions, while the system analyzed which questions yielded the most useful information and gradually refined its strategies. In tests on MMLU-Pro and ARC-MCAS, asking an average of just 1.2 clarifying questions, PASSING nearly tripled expertise-detection accuracy compared to the best baseline method. An interesting detail: part of the gain comes specifically from clarifying questions about *how* someone reasons (how-to-ask), not just *what* they know (what-to-ask) — removing either component noticeably hurt accuracy.