Demographic Injection: A Line About Diversity Makes AI Invent a Patient's Race and Income That the Question Never Mentioned

Add a single phrase like "consider diversity, equity and inclusion (DEI) in your answer" to a medical question, and the model starts inventing the patient's race, income or social status, even though the question said nothing about them. It weaves the made-up details into its reasoning as if they'd been part of the problem from the start.

Without the phrase, the model fabricates patient demographics in 0.7% of answers. With the DEI phrase, in 33% of answers, a 47-fold jump, and it happens in every single one of the 47 models tested, from GPT-5 to small open models. The cause isn't the extra text tacked onto the question (that was tested separately); it's specifically the equity content, which drags demographic associations out of the training data. Most of what gets added is harmless (general statistics like "this is more common among..."), but in 0.25–2.4% of cases the model attributes an invented trait to this particular patient and changes its chosen diagnosis, and in 99.8% of those cases it changes to the wrong one.

The reason: in the texts the model was trained on, talk about equity almost always comes paired with mentions of race, gender and income, so any "consider fairness" instruction triggers that template, even with no demographic data in the question. And it's not just DEI: any framing instruction ("consider the cost," "consider the legal risks") can likewise make the model add unstated details and quietly lean on them.

Mechanism Blueprint

CONDITION: a question with no demographic data + the phrase "consider diversity/equity"
↓
STEP 1: The model adds demographic statistics → 33% of answers (vs. 0.7% without the phrase)
STEP 2: In most cases (91%) it's a harmless general statement → the answer doesn't change
STEP 3: In 0.25–2.4% of cases the model "sticks" an invented trait onto the specific patient → the answer choice changes
STEP 4: 99.8% of those changes go toward the WRONG answer

Example in Action

Task: An HR manager pastes anonymous descriptions of two candidates for a team lead role into ChatGPT (no name, photo or nationality) and asks for help choosing whom to promote, adding "apply diversity and inclusion principles in your assessment."

Prompt (risky version):

Here are two candidates for a department head role:

Candidate A: 6 years at the company, runs 3 projects, average soft skills ratings, 
strong technical background.

Candidate B: 2 years at the company, runs 1 large project, high ratings 
for soft skills, weaker technically.

Apply the principles of diversity, equity and inclusion in your assessment.

Result: Going by the mechanism in the study, the model is quite likely to start writing assumptions about the candidates' background, gender or age into its reasoning, details that weren't in the description. Most of the time it'll be a harmless background remark that doesn't affect the conclusion. But there's a chance (a fraction of a percent per the study, but not zero) that an invented detail "sticks" to a specific candidate and shifts the final recommendation, and far more often toward the wrong choice than the right one.

Why It Works

The model doesn't distinguish between "a statistical fact about a category of people" and "a fact about the specific person in the question"; it blends epidemiology with the case description, because in its training texts those things almost always sit side by side.

The model is great at picking up language patterns: once it sees the trigger "diversity/equity" in the instruction, it pulls out the whole linked cluster of associations (race, income, access to healthcare), simply because that's how these topics are usually discussed in the texts it was trained on.

Knobs that change the strength of the effect: - Instruction wording → a short equity phrase produces ~14% injection, a detailed "social determinants" wording up to 56%. The more compact and specific the framing instruction, the lower the risk. - Any framing instruction, not just DEI ("consider the cost," "consider the legal risks"), carries the same risk by the logic of the mechanism; the authors just didn't test it directly.

💡 Practical takeaway: whenever you add any "consider X in your assessment" instruction to a prompt for a real case (medicine, HR, legal review, credit scoring), pin down the boundaries of the data explicitly:

"Use only the facts stated above. Do not add or assume age, gender, race, income, social status or any other characteristics that were not explicitly mentioned. If data is missing, say so directly."

This is a logical consequence of the mechanism found, not a fix the authors tested, but it directly blocks the path the injection takes.

Limitations

⚠️ Only DEI wordings were tested: the "any framing instruction" hypothesis follows logically from the mechanism, but the authors directly tested only phrases about equity and diversity.

⚠️ The "changed answer" numbers are a conservative lower bound: the authors deliberately used a method that is more likely to underestimate than overestimate the share of cases where the fabrication changes the answer.

⚠️ Only multiple-choice tests were used: MedQA, MedMCQA, MMLU-medical and PubMedQA are "pick one of the options" or "yes/no/maybe" formats. It's unknown whether the model behaves the same way in a free-form consultation with no ready-made answer options.

⚠️ The mechanism is a hypothesis, not a proven fact: the authors suggest the model pulls the "equity ↔ demographics" association from its training texts, but didn't verify this directly by analyzing the model's internal activations.

⚠️ The antidote phrase wasn't tested in the study: the suggestion to add "don't invent unstated facts" is our logical extrapolation from the mechanism found, not a result the authors verified experimentally.

How They Tested It

The researchers took 47 models, from top closed models (GPT-5, Claude Sonnet 4.6) to small open ones (Llama-3-8B) and even medically fine-tuned ones (OpenBioLLM), and ran 500 questions from four medical tests through them in four variants: no additions, a nonsense addition, a neutral clinical addition of the same length, and the DEI phrase. That came to 376,000 answers.

The key design trick is the "neutral addition of the same length": it proves the issue isn't extra text as such but specifically the equity content. It worked: the DEI phrase beat the neutral control by a factor of 18.

To find where the model made something up, they used a two-stage judging system: a fast AI judge flagged suspicious answers and a precise arbiter (Claude Sonnet 4.6) double-checked them, which is why the final "changed answer" numbers are likely underestimates.

Remarkably, the effect showed up in literally all 47 models without exception, including models specially fine-tuned on medicine (their injection rate is lower, 19% vs. 49% for top closed models, but it doesn't vanish). And the wording of the phrase itself decides almost everything: the short version produced 14% injection, the detailed "social determinants" version 56%.

Resources

Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts: Diego Mardian, Frank Liu, Arizona State University. Benchmarks: MedQA, MedMCQA, MMLU-medical, PubMedQA. Judges: gemini-3.1-flash-lite (initial screening), claude-sonnet-4-6 (precise arbiter).