Instruction Retrieval: External Memory for Small Models via Ready-Made Instructions

STEP 1 (offline): Build the instruction corpus

• Cluster the training questions by similarity → groups of similar tasks
• For each cluster: generate an instruction with an LLM (GPT-4/5) → structure: Background Knowledge + Reasoning Steps
• Result: a library of ready-made instructions (in the paper: 8.4k for MedQA, 1k for MMLU Law, 29k for MathQA)

STEP 2 (inference): Retrieve and apply

• Embed the test question → find the top-5 nearest clusters by cosine similarity
• Retrieve 5 instructions → add them to the prompt before the question
• The model follows the instructions → generates the answer

Task: You have a small model (Qwen 7B) and need to help a user work through a medical question about a drug's side effects. On its own, the model mixes up contraindications and gives a shallow answer.

Prompt:

You are a medical assistant. Use the following instructions to answer.

INSTRUCTION 1: Anticoagulant contraindications
Background Knowledge:
- Anticoagulants reduce blood clotting, risk of bleeding
- Absolute contraindications: active bleeding, hemorrhagic stroke <3 months ago, severe thrombocytopenia
- Relative: pregnancy, recent surgery, GI ulcers
- Interactions: NSAIDs increase bleeding risk

Reasoning Steps:
1. Identify the drug class and mechanism of action
2. Check for absolute contraindications (if present → STOP)
3. Assess relative contraindications and the risk/benefit ratio
4. Check drug interactions
5. Choose the answer by priority: absolute > relative > interactions

[4 more similar instructions for context...]

QUESTION: A 67-year-old female patient with atrial fibrillation. History of a stomach ulcer 2 years ago (healed). Takes ibuprofen for arthritis. Can warfarin be prescribed?

ANSWER:

Result: The model sees structured knowledge (what anticoagulants are, which contraindications exist, the priorities) and a clear algorithm. Instead of hallucinating, it goes step by step: (1) warfarin = anticoagulant, (2) no absolute contraindications (the ulcer has healed), (3) relative — ibuprofen increases the risk, (4) weighs risk/benefit for atrial fibrillation, (5) gives a recommendation with a warning about ibuprofen. The answer is more accurate and safer than without instructions.

The weakness of small models: A 7B-parameter model lacks the capacity to hold specialized knowledge (thousands of drugs and contraindications) and procedural skills (how to weigh risks, which checks to run first) at the same time. When it tries to recall a rare fact, it often hallucinates. When it tries to build multi-step logic, it skips critical checks or mixes up priorities.

The LLM strength: Models are great at following explicit instructions and at applying ready-made frameworks. Give a model a clear structure like "first check X, then Y, if Z — choose A," and it executes more reliably than if you ask it to "think it through yourself." It's the difference between "invent a cake recipe" and "here's the recipe, follow the steps."

How the method uses the strength to work around the weakness: Instruction Retrieval moves knowledge and logic outside the model's parameters — into a text corpus. The model doesn't recall facts from its weights (unreliable); it gets them in the prompt (reliable). The model doesn't build a reasoning plan from scratch (risky); it follows a ready-made algorithm (safe). The model's parameters are freed up for language skills and instruction following, while the text corpus takes on the role of external memory.

Control levers:

• Number of retrieved instructions (top-k): 5 by default. Drop to 2-3 to save tokens on simple tasks, raise to 7-10 for complex interdisciplinary questions.
• Instruction detail (concise/verbose): Short instructions (500-750 tokens) work better — focus only on the essential steps. Long ones (1.3k-2.1k tokens) add noise and lower accuracy. For small models, always go concise.
• Audience level (high school/graduate): Simplified language vs. terminology. The difference is minimal, but for highly specialized domains (medicine, law) the graduate level preserves the precision of definitions slightly better.
• Clustering granularity: Small clusters (1-2 examples) → narrowly specialized instructions. Large clusters (5-10 examples) → more general ones. The study showed robustness to granularity — you can use broader clusters without losing quality.

You are a {specialist role}. Use the following instructions to answer the question.

INSTRUCTION: {task type name}

Background Knowledge:
{key facts, definitions, rules}
{patterns that help tell right answers from wrong ones}
{for each point: how strongly it determines the answer}

Reasoning Steps:
1. {what to identify in the question, which key signals to look for}
2. {how to apply the most important knowledge first}
3. {when which signals outweigh others}
4. {how to weigh different types of evidence}
5. {how to tell similar answer options apart}
6. {what should determine the final choice}

[Repeat the structure for 2-5 relevant instructions]

QUESTION: {your question}

ANSWER:

What to fill in:

• {specialist role} — medical assistant, legal consultant, tax expert
• {task type name} — "Drug contraindications", "Employment disputes", "Tax deductions"
• {key facts...} — domain-specific knowledge: definitions, classifications, important rules
• {Reasoning Steps} — a step-by-step algorithm: what to check, in what order, how to choose

🚀 Quick start — paste into the chat:

Here is the Instruction Retrieval template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your subject area (medicine? law? taxes?), typical questions and pain points — because it needs to understand which knowledge is critical and what check logic is right for that domain. It will take the structure (Background + Steps) from the template and fill it with concrete content for your task.

⚠️ Model size: Models with <3B parameters show unstable results — often a negative effect. The ~3B threshold is the minimum for reliable instruction following. Qwen 1.7B can gain +15% on one task and 0% on another.

⚠️ Task type: On math the gains are more modest (+5%) — models are already strong at symbolic reasoning thanks to pretraining. The biggest effect is on knowledge-intensive tasks (medicine +9%, law +8%), where the model lacks domain knowledge.

⚠️ Verbosity penalty: Long instructions reduce accuracy (up to -18% on some models). Excessive detail creates cognitive overload — the model loses focus. Rule of thumb: brevity > completeness.

⚠️ A corpus is required: The method needs pre-built instructions. In the paper they were generated with GPT-5 from training data. For your own tasks you either build a corpus (ask an LLM to write instructions for task types) or generate them on the fly (slower, but more flexible).

A team from the University of Michigan took three benchmarks requiring different types of reasoning: MedQA (medical exams — clinical knowledge + diagnosis), MMLU Professional Law (law exams — statutes + precedents + fact patterns), MathQA (math problems — symbolic reasoning). They tested 13 models from 1B to 14B parameters across 5 families (LLaMA-3, Gemma-2, Qwen-3, Mistral, DeepSeek R1).

First they built the Instruction Corpus: training questions were clustered by similarity (agglomerative clustering, cosine distance), and an instruction was generated for each cluster with GPT-5. The result: 8.4k instructions for MedQA, 1k for Law, 29k for MathQA. They tested 4 style variants: High School vs Graduate (language level) × Concise vs Verbose (length). Each instruction was rated on 5 quality criteria by GPT-5 (knowledge, relevance, reasoning accuracy, clarity) — average score 4.7+/5.

At inference, each test question retrieved its top-5 nearest instructions, which were added to the prompt. The comparison was against a zero-shot baseline (no instructions) and GPT-4o as the frontier model. The results were surprising: 14B models with instructions beat GPT-4o on MedQA (83% vs ~75%) and Law (65% vs ~60%). Gains appeared strictly above 3B — smaller models behaved unpredictably (+15% one time, -11% the next).

The key insight from mixed-effects regression: family matters more than size. Qwen-3 showed a +56pp marginal effect, DeepSeek R1 +38-43pp, while model size gave only +9pp per log-unit. In other words, architecture and pretraining (how the model was taught to follow instructions) matter more than raw parameter count. Verbosity showed a -8pp penalty per log-unit of tokens — more words = worse results.

For practice this means: don't chase the biggest model; pick a family with strong instruction following (Qwen, DeepSeek). Cut the wordiness in your instructions — every extra sentence hurts.

Context: The researchers tested the method on MedQA — medical exam questions. Here is an example instruction from the corpus (Graduate Concise variant):

## Background Knowledge

Female Athlete Triad (FAT) comprises three interrelated components: low energy
availability (with or without disordered eating), menstrual dysfunction
(amenorrhea or oligomenorrhea), and decreased bone mineral density (BMD). The
pathophysiology involves suppression of the hypothalamic-pituitary-ovarian (HPO)
axis due to chronic negative energy balance, leading to decreased GnRH pulsatility,
reduced LH and FSH secretion, and subsequently low estrogen levels.

Low estrogen is the primary driver of decreased BMD in this population. Other
hormonal changes include elevated cortisol (from stress) and decreased IGF-1,
both of which further impair bone formation and increase bone resorption. Physical
signs may include lanugo hair (marker of severe malnutrition), low body weight,
and orthostatic changes.

Key diagnostic considerations: negative pregnancy test (to rule out pregnancy as
cause of amenorrhea), low or inappropriately normal LH/FSH (indicating central
suppression rather than primary ovarian failure which would show elevated
gonadotropins), and low estradiol. TSH is typically normal unless concurrent
thyroid pathology exists.

## Reasoning Steps

1. Identify the diagnostic triad: amenorrhea + low BMI + intense training history
2. Prioritize the HPO suppression logic: chronic energy deficit → GnRH↓ → LH/FSH↓ →
   estrogen↓
3. Eliminate violated conditions:
   - High TSH would suggest hypothyroidism (different mechanism)
   - High estrogen contradicts the FAT pathophysiology
   - Positive hCG would indicate pregnancy (not FAT)
4. Apply the estrogen-BMD link: low estrogen is the strongest predictor of decreased
   bone density in this context
5. Select the answer that directly follows from low estrogen: decreased bone mineral
   density

This instruction was created for a cluster of questions about the Female Athlete Triad. The Background gives definitions (FAT = 3 components), the mechanism (HPO axis → low estrogen), physical signs and diagnostic markers. The Reasoning Steps give a clear algorithm: identify the triad → apply the HPO-suppression logic → eliminate options with violated conditions → link estrogen to BMD → choose the right answer.

💡 Adaptation for legal consultations (UK employment law):

Same Background + Steps principle, but for legal tasks — employment statutes instead of medical knowledge, a compliance-check algorithm instead of diagnosis:

You are a legal consultant on UK employment law. Use the instructions to answer.

INSTRUCTION: Unfair dismissal initiated by the employer

Background Knowledge:
- Employment Rights Act 1996, s.98: the list of potentially fair reasons for dismissal
- Misconduct (unauthorized absence): absence for >4 hours in a row without a valid
  reason
- Valid reasons: illness (sick note), car accident, death of a relative, force majeure
- Procedure: document the incident, request the employee's explanation (2 days to respond),
  issue a written decision
- A procedural breach = unfair dismissal, even if there was a valid reason

Reasoning Steps:
1. Check the formal ground for dismissal (is it a recognized reason?)
2. Check the factual basis (did the event actually happen?)
3. Check the procedure: incident report → request for explanation → decision (any step skipped?)
4. Assess whether the reason for absence was valid (sick note? force majeure?)
5. If the procedure was breached OR the reason was valid → the dismissal is unfair

QUESTION: An employee was absent from work for 5 hours due to a car accident (there is a police report).
The employer didn't ask for an explanation and fired him on the spot for unauthorized absence. Is the dismissal lawful?

ANSWER:

The model goes step by step: (1) the "unauthorized absence" ground is a recognized reason ✓, (2) the absence did happen (5 hours) ✓, (3) the procedure was breached (no explanation was requested) ✗, (4) the reason was valid (car accident + police report) ✗. Conclusion: the dismissal is unfair on two grounds. Recommendation: file a claim with an employment tribunal.

💡 Adaptation for business analysis (evaluating a startup idea):

Background = startup success factors, Steps = evaluation methodology:

You are a business analyst. Use the instructions to evaluate the idea.

INSTRUCTION: Evaluating a B2B SaaS startup

Background Knowledge:
- Unit Economics: LTV/CAC should be >3 for sustainable growth
- Time to Value: a B2B buyer expects results in <90 days, otherwise churn
- Switching Costs: if integration takes >2 weeks, it's a barrier to entry, but also to exit
- Market Size: TAM >$1B, SAM >$100M for venture interest
- Competitive Moat: network effects, data, technology (patents), brand
- Distribution: PLG (product-led) vs Sales-led (ACV >$10k needs a sales team)

Reasoning Steps:
1. Assess the problem: is it critical (top-3 pain in the segment?), measurable (is there a metric?),
   frequent (does it come up >1 time/month?)
2. Check the economics: LTV/CAC, payback period, gross margin (SaaS should be >70%)
3. Determine defensibility: what stops competitors from copying it in 3 months?
4. Assess GTM fit: does the distribution strategy match the ACV and segment?
5. Check timing: why now and not 2 years ago? (regulatory change? tech
   enabler?)

QUESTION: The idea — a platform that automates HR processes in retail (hiring,
onboarding cashiers). ACV $5k, 1-day integration, 20 paying customers right now,
LTV/CAC = 2.1. No competitors yet, but big HR suites could add the feature within
a quarter. Launching now because a new federal rule requires retailers to keep digital
employee onboarding records starting in 2025. Is this a promising idea?

ANSWER:

The model analyzes: (1) the problem is critical (compliance + cashier turnover) ✓, (2) LTV/CAC = 2.1 is on the weak side (target >3), payback is fast (low ACV) ✓, gross margin is probably good (SaaS) ✓, (3) the moat is weak (easy to copy), (4) GTM fit is OK (ACV $5k → PLG/inside sales), (5) timing is excellent ✓ (regulatory trigger).

Conclusion: the idea is risky — weak economics and no defensibility, but there is a window of opportunity (the new rule). Recommendation: move fast before 2025, build switching costs through integrations, improve unit economics to LTV/CAC >3 via upsell.

🔧 Technique: Confidence tiers in Background Knowledge

The original method says the Background should note "for each point: how strongly it determines the answer." You can label the tiers explicitly:

Background Knowledge:
- [CRITICAL] Anticoagulants reduce clotting → absolute contraindication
  with active bleeding (if present → STOP, no further checks needed)
- [HIGH] Relative contraindications require a risk/benefit assessment (GI ulcer,
  pregnancy → can be prescribed if the benefit outweighs the risk)
- [MEDIUM] Interactions with NSAIDs increase bleeding risk (take into account, but don't
  automatically block the prescription)
- [LOW] The patient's age by itself is not a contraindication (may affect the dose,
  but not the decision to prescribe or not)

Effect: The model understands the priority of checks more clearly — CRITICAL blockers first, then HIGH risk-benefit, then MEDIUM interactions. This cuts down on errors like "dropped the drug because of a LOW factor while missing a CRITICAL one."

Big Reasoning with Small Models: Instruction Retrieval at Inference Time Kenan Alkiek, David Jurgens, Vinod Vydiswaran
School of Information, University of Michigan
2025

Evaluations: MedQA (Jin et al., 2020), MMLU Professional Law (Hendrycks et al., 2021), MathQA (Amini et al., 2019)

Referenced methods: Chain-of-Thought prompting (Wei et al., 2023), Knowledge-Augmented Reasoning Distillation (Kang et al., 2023), Dense Passage Retrieval (Karpukhin et al., 2020)