TL;DR
CDP is a technique that replaces a vague persona ("an average student," "a beginner") with a concrete list of what the person can and can't do, broken into separate items. The model gets not a general level but a detailed skill map, and plays the role far more accurately.
The problem: when you ask an LLM to "answer like a typical student," the model almost always answers too correctly and too uniformly, practically like a straight-A student, no matter what level of person you asked it to portray. The reason is simple: the model is trained to be a helpful assistant that tries to give the right answer, and it has nothing to "hook onto" in order to make plausible mistakes. A generic "weak student" label doesn't tell the model what exactly this student can't do.
CDP solves it like this: first the skill is broken into concrete sub-skills (for fractions, say, "can subtract simple fractions," "can borrow from a whole number," and so on), then for the virtual person you decide which items they've mastered and which they haven't, and turn that into a text description of the profile. The model gets this profile before the task, and answers according to specific knowledge gaps rather than a general stereotype.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Break the target skill into K concrete sub-skills → a list of K items
STEP 2: Decide which items the "virtual person" has mastered (yes/no) → a binary profile
STEP 3: Turn each item into text: what the person does confidently, what they can't do at all → a ready profile description
STEP 4 (for a set of people): Set how often each profile occurs — evenly or per real audience stats → a pool of virtual people
STEP 5: Give the LLM the profile + the task, ask it to answer as a person with this profile → an answer that matches the knowledge gaps
STEP 6 (for the test as a whole): Collect answers across all profiles, see where questions turned out too easy/too hard
Steps 1–3 and 5 are done in one chat in a single request. Steps 4 and 6 are only needed if you want to run many personas for statistics; then a script is better, but in a chat you can hand-build 3–5 profiles.
Example
Task: A product manager is testing a new feature in a mobile carrier's app, a plan comparison screen, and wants to know before release where users with different levels of digital literacy will get confused.
Prompt:
Here is a feature in the app: a plan comparison screen. The user can pick up to 3 plans
and see their specs side by side: price, gigabytes, minutes, roaming terms.
Simulate how a user with the following skill profile would go through this feature:
- Can do confidently: find the right section in the main menu, read the price and data allowance
- Can't do at all: use filters, understand terms like "unlimited with fine print", "grace period"
- Can do partially: compare two items visually, but gets confused by a third option
Answer in the first person. Walk through the feature step by step: what you do, where you get stuck,
what you say out loud at that moment.
Result: the model produces a step-by-step first-person story in which the user finds the screen easily but freezes on the terms and gets confused when picking a third plan. Repeat the prompt with a different profile (say, "tech-savvy user") and you'll see the contrast, and immediately notice which UI elements really trip up people with specific gaps, rather than abstract "beginners."
Why it works
When asked to play a "weak" or "average" person, an LLM by default drifts back to its base role: a helpful assistant trying to answer correctly. A vague label like "average student" gives the model no concrete reason to make a mistake exactly where a real person would.
But the model is excellent at following explicit, detailed instructions about what a character knows and doesn't know. Say not "weak student" but "can subtract simple fractions but can't borrow from a whole number," and the model gets a concrete behavioral rule and sticks to it much more strictly.
CDP uses this strength: instead of one fuzzy label, it provides a list of concrete facts about the character. The model stops "playing the role in general" and starts obeying specific constraints, which is where the more realistic spread of answers comes from.
Control levers: - Number of sub-skills (K) → more items means a sharper profile but a longer prompt. For simple tasks 3-4 items are enough. - Three mastery levels (confident / can't at all / partial) instead of a plain yes/no → more flexible, lifelike character behavior. - Real audience stats (if you have data on your users) → add them to the prompt as a reference point: "70% of users get confused by filters." This sharply boosts simulation accuracy in models that already hold the profile reasonably well.
Prompt template
Here is {context: test/feature/scenario}: {short description}.
Simulate how a person with the following profile would {react to / go through this}:
- Can do confidently: {skill 1, skill 2}
- Can't do at all: {skill 3, skill 4}
- Can do partially: {skill 5}
Answer in the first person, showing this person's actions and thoughts step by step.
Fill in: the context (what you're testing), and 3-5 concrete skills split across the three mastery categories.
🚀 Quick start — paste into the chat:
Here is the Cognitive Diagnostic Profiling template. Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which specific skills/knowledge matter in your situation and how to split them into separate items, because the method only works when the profile is made of concrete statements, not general ones.
Limitations
⚠️ Requires an expert breakdown: If the skill is split crudely or incorrectly (e.g., items that are too general), the profile won't deliver a real gain. The method's quality depends on how concrete the wording is.
⚠️ Weaker without "thinking mode": Models with reasoning turned on (thinking mode) hold the profile much more accurately than those without it; on non-reasoning models the effect is weaker.
⚠️ Statistical calibration needs a tool for bulk runs: You can check one or two profiles by hand in a chat. But if you need to run dozens of profiles for a statistically reliable result (as in testing educational materials), you'll need a script or the API.
How it was studied
The researchers took real data from 536 schoolchildren who took a 15-item fraction subtraction test with a known breakdown into 5 specific skills. They compared how 8 different LLMs (from Google, DeepSeek, Qwen, Mistral) answered the same items in three modes: with no profile at all, with a uniformly random detailed profile, and with a profile weighted by the real statistics of skill distribution among the students.
Without a profile, the models behaved like straight-A students: almost all answered everything correctly regardless of question difficulty, and the match with the real students' distribution was weak. Give them a detailed profile and the match jumped: for one of the models, for example, the accuracy of predicting item difficulty shot up from a correlation of 0.24 to 0.86–0.90 with real data.
The most interesting part: models with "thinking" turned on did much better than regular versions of the same models. And adding real statistics on skill distribution among students helped most where the basic detailed profile was already working decently; in other words, a hint about the real world amplifies a good signal but doesn't rescue a weak one.