TL;DR
Playbook induction is a method that turns an ordinary language model into a narrow specialist without fine-tuning. The model runs practice attempts at the task on its own, grades its misses against a checklist of criteria, turns each miss into a rule of the form "if situation X → do Y → get Z", and collects those rules into a compact list: the playbook. That list is then simply pasted into the prompt as an instruction.
The key finding: the model already knows the facts: what an ear infection is, how to read an audiogram, which diagnoses exist. The problem isn't knowledge; it's that the model doesn't keep the order of actions in its head: it forgets to ask about an important symptom, responds the wrong way to a patient's anxiety, or crams three questions into one reply. The researchers tested "throw more knowledge at it" (RAG, retrieving and inserting reference information), and it either didn't help or made things worse: the extra reference text got in the way of keeping the conversation structured.
The method fixes this not with knowledge but with procedure. Six steps: (1) build a checklist of 21 consultation-quality criteria, (2) run 73 practice attempts, (3) after each one, grade it against the checklist and write down what went wrong, (4) turn the misses into "trigger → action → expected effect" rules, (5) weed out useless rules, keeping only those that proved their value, (6) freeze the final 19 rules and paste them into the prompt as context for real work.
Method Blueprint
STEP 1: Build a rubric, a checklist of quality criteria (e.g., 15-20 items) → list of items
STEP 2: Do a practice run of the task (dialogue/text) → transcript
STEP 3: Grade the transcript on every rubric item + explain the low scores → "score + reason" table
STEP 4: From each low score, derive a rule: TRIGGER → ACTION → EFFECT → rule card
STEP 5: Add the rule to the master list, remove duplicates and anything not confirmed on new runs → compact list of rules (the playbook)
STEP 6: Paste the final playbook as a system instruction for real tasks → ready-to-use prompt assistant
Steps 2–5 repeat in a loop on different test scenarios (73 times in the study) until the list of rules stabilizes.
Example in Practice
Task: The founder of an online language school wants the chatbot on the website to run a sales consultation with a prospective student as crisply as a top sales rep: find out the learning goal, level and budget, handle price objections, and recommend a specific course.
Prompt (kicking off the loop):
I want to create a "playbook", a set of behavior rules, for an AI assistant
that will run a consultation: help someone choose an English course
and close the sale.
Let's do it as a loop:
STEP 1 — RUBRIC
Build a checklist of 10 quality criteria for this kind of consultation
(for example: found out the goal, clarified the level, answered "too expensive"
on the merits rather than with a discount, etc.)
STEP 2 — TEST RUN
Play the consultant and hold a dialogue with a client following this scenario:
"Adult, level between A2 and B1, wants it for work,
says 'too expensive' on their third message". Keep the dialogue natural
and don't peek at the rubric.
STEP 3 — GRADE AND LESSON
Grade the dialogue above against the rubric from step 1. Where the score is low,
write a rule:
TRIGGER: [situation]
ACTION: [what to do]
EFFECT: [what it achieves]
STEP 4 — ASSEMBLY
Add the rule to the "Consultation rules" list. Don't duplicate similar ones.
We'll repeat the loop 4-5 times on different client scenarios
until we have 10-12 stable rules.
Start with step 1.
Result: the model will first produce a checklist of criteria. Then, on each iteration, a dialogue with a "client", a breakdown against the checklist, and 1-3 new rules. After 4-5 loops you'll have a list of 10-15 specific instructions like "if the client says 'too expensive', don't offer a discount right away; ask what they're comparing the price to". You can paste that list into the bot as a system instruction, and the consultations become noticeably more consistent and structured.
Why It Works
The model is bad at keeping order and priorities straight inside a long task done "in one go": it knows all the necessary facts but doesn't arrange them into a disciplined procedure on its own. Just giving it more reference information (RAG) doesn't help: a reference is knowledge, not an instruction about "when to do what".
On the other hand, the model is excellent at grading text against a checklist and generalizing a conclusion into a structured form, which is exactly what it does well in reflection.
The method harnesses that strength: it makes the model analyze its own failed attempts and turn them into explicit, compact rules, rather than counting on the model to "just learn" to run the task correctly after a single instruction.
Control levers: - Number of practice runs (73 in the original) → for a simple task 5-10 is enough - Number of rules in the final playbook (19 in the original) → for a narrow task 5-8 is enough; otherwise the playbook bloats and loses focus - Rubric criteria → depend entirely on your task: sales gets one set of items, customer support another - Three tiers of rule selection (draft → verification → core) → can be simplified to a single "verified rules" list if you don't need that much rigor
Prompt Template
I want to create a "playbook", a compact set of rules, for an AI assistant
that will perform this task: {task}.
Let's do it as a 4-step loop:
STEP 1 — RUBRIC
Build a checklist of 8-12 quality criteria for this task.
Each criterion is a specific behavior that can be graded.
STEP 2 — TEST RUN
Play the role of {assistant role} and perform the task in this scenario:
{scenario}. Behave naturally and don't peek at the rubric.
STEP 3 — GRADE AND LESSON
Grade the result above against the rubric from step 1. Where the score is low,
explain why and write a rule:
TRIGGER: [the situation where this applies]
ACTION: [what needs to be done]
EXPECTED EFFECT: [what it will achieve]
STEP 4 — ASSEMBLY
Add the rule to the "Task execution rules" list.
If a similar rule already exists, don't duplicate it; refine the wording.
We'll repeat the loop (steps 2-4) several more times on different scenarios
until we have 10-15 stable rules. I'll use the final list
as a system instruction for real tasks.
Start with step 1. My task: {task}
Fill in: {task} is what the assistant should do (negotiate, write job applications, advise on a product), {assistant role} is who it acts as, {scenario} is a specific test situation with details.
🚀 Quick start: paste this into the chat:
Here's a method for building a "rules playbook" for an AI assistant through rubric-based
grading and reflection. Adapt it to my task: {my task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask about the assistant's role, typical scenarios and quality criteria, since without them it can't build a rubric or act out the test runs. It will take the loop logic from the template and plug in your task.
Limitations
⚠️ It takes effort: the method is a process of several "run → grade → rule" loops, not a quick prompt. Even in simplified form it will take 30-60 minutes.
⚠️ It doesn't solve a knowledge problem: the playbook teaches the ORDER of actions, not facts. If the model doesn't know the subject, rules won't help; you need a separate source of knowledge.
⚠️ You need a substantive rubric: if the grading criteria are vague ("be a good consultant"), the rules will come out as useless platitudes.
⚠️ Rules can overfit to the test environment: in the study, some rules reflected quirks of the simulator (a cap on the number of turns in the dialogue) rather than real practice. When moving a finished playbook into a real task, such rules need to be rechecked by hand.
How They Tested It
The researchers took 73 training cases from audiology (real clinical cases, standardized, in English and Chinese) and had an AI audiologist run a simulated consultation with an AI patient. After each attempt, a dedicated grader module checked the dialogue against a 21-item rubric, wrote explanations for each item and extracted lessons, and a curator module turned the lessons into rules and added them to a three-tier playbook, keeping only those that proved useful on repeat runs. The result was a compact list of 19 rules, which was frozen before the final evaluation.
Next, 58 new cases were evaluated blind: the same consultations were run both by the AI and by 17 practicing audiologists, and 4 independent experts graded the transcripts without knowing whether the author was a human or a machine. The AI won in all 58 cases and on 20 of 21 criteria. A curious detail: the gap was twice as large in Chinese as in English. Probably the human clinicians are more used to running appointments by Western standards in English, while the playbook doesn't care about language and works equally well in both versions.
A separate test showed that removing the playbook and leaving only access to a knowledge base (RAG) did almost no good, and sometimes made things worse: the model already knows the facts; what it lacked was the skill of running the appointment step by step, and the extra reference text threw off the structure of the conversation. That's the main practical takeaway: procedure beats knowledge when the model is already an expert on the facts but not on the workflow.