Practical prompting, backed by research

Key finding: when one prompt asks an LLM to verify a long checklist (45–174 criteria in the study), the model admits it checked only a fraction: Sonnet 4.6 reported checking 16% of 174 criteria and 27% of 45. Overloaded, it judges by impression instead of evidence: on honest submissions it rejected 74–88% of items experts had marked as met, and a persuasive cover note can swing it the other way. Splitting the list into small batches, each sent with the full document and a required quote, keeps every verdict tied to the text.

Finding: an LLM anonymizing text removes exactly what it was told to — not a word more. Institution-Specific Prompting lets you catch an organization's local slang — hospital abbreviations, building names, internal codes — that falls outside the standard 18 categories of personal data and therefore stays in the text untouched. Ask the model to look for specific local categories and give "don't remove this" examples — it applies the rule across the whole document instead of guessing. Result: about 4 out of 5 previously missed identifiers are found in a single pass.

Finding: an LLM reacts to a home neighborhood the same way whether the field is 100% useless for prediction or carries a real signal, like a gauge stuck on one reading. The method lets you test your chatbot for hidden bias via stand-ins for gender, race and age, without a single mention of the protected attribute in the prompt. The model grabs the word "neighborhood" like a hook instead of computing its real statistical significance — reliance on such fields doesn't grow with their actual usefulness.