REAP: Separate Recalling a Fact From Formatting the Answer So the LLM Doesn't Get Confused and Hallucinate

REAP is a technique for getting accurate facts out of an LLM without the internet or search. It splits the process into two separate steps: first the model "recalls" the answer in free text through a chain of thought, then a separate request converts it into the format you need (a list, JSON, a table).

When you ask a model in a single request to both recall a complex fact and immediately output it in a strict format, it gets confused. For long lists (every winner of an award over 100 years) it recalls only half. For quantitative facts (a stadium's capacity, an island's area) it mixes up similar objects: instead of the small stadium, it gives the capacity of a big neighboring one with a similar name. And if the fact doesn't exist at all (the person is alive, the company is private), the model makes up an answer instead of saying "I don't know."

The fix is to break the request down by question type: for long lists, ask in chunks (by decade, by compass direction); explicitly tell the model to first check whether the question even applies, and return an empty answer if the fact doesn't exist; and for numbers, check against a reasonable range before the final answer. The format (JSON) is required only at the very end, as a separate step.

Method Blueprint

STEP 1 (recall, free text, may be several sub-queries):
  → Applicability check: "is the person alive? is the company public?" → if not, the answer is "empty"
  → Query decomposition: by year / by compass direction / by category → draft list of facts
  → For numbers: check against a reasonable range of values

STEP 2 (formatting, separate request):
  → Convert the draft into the final format (JSON/table) without losing data

Example in Action

Task: You're writing a piece about the Booker Prize and want the full list of winners since 2006, but in a single request ChatGPT usually "loses" some of the years.

Prompt:

You are an expert on contemporary English-language fiction.

Step 1: Recall the Booker Prize winners for 2006–2012. 
For each year, give the author and the book. If you're not sure, say 
"not sure" outright; don't make things up.

[after the answer, repeat for 2013–2018]
[after the answer, repeat for 2019–2024]

Step 2: Combine all three lists into a single table: year | author | book.

Result: For each time slice the model gives a separate draft list in plain text, flagging uncertainty wherever it doesn't remember exactly. After merging, you get a table that's noticeably more complete than what you'd get from "list every Booker Prize winner" in one go, because focusing on a narrow slice forces the model to "scan" it more carefully.

Why It Works

A model has limited "attention" per answer. If you ask it to recall lots of facts and stick to a strict format at the same time, part of that budget goes to formatting, and recall (how complete the remembered set is) drops.

A model recalls facts well when the question is narrow and specific: "who won the prize in 1985-1995" works better than "list every winner over 100 years."

The method removes the format requirement from the recall stage and slices one big question into small, targeted sub-queries. On top of that, an explicit "does this question apply" check before answering is a direct way to cut hallucinations without relying on the model to "realize" on its own that it doesn't know the answer.

Control knobs: - Number of decomposition sub-queries → increase it for very long lists, drop it for simple questions - The phrase "if you're not sure, write 'not sure'" → reduces fabrication but may leave more blanks; use it when accuracy matters more than completeness - Plausibility range for numbers → set your own ("city population between X and Y thousand") for the specific task - Splitting "recall" and "format" into two requests → skip it if the task is simple and short, to save time

Prompt Template

Step 1. Applicability check
Before answering, check: {criterion for whether the fact exists, e.g. "is the person alive" 
or "is the company public"}. If the fact doesn't exist or you're not sure, 
write "NO DATA" and don't invent an answer.

Step 2. Extraction in parts
Topic: {query topic}
Split the question into {N} parts by {decomposition criterion: years/directions/categories} 
and recall the facts for each part separately. Write in free text, 
and flag uncertainty explicitly.

Part 1: {range 1}
Part 2: {range 2}
...

Step 3. Plausibility check (for numbers)
If the answer is a number, compare it with an object of a similar magnitude. 
Expected range: {range}.

Step 4. Formatting
Combine all the collected facts into {output format: table/JSON list} without losing data.

Fill in: {query topic} is what exactly you're recalling, {criterion for whether the fact exists} is the condition under which there is no answer, {N} and {decomposition criterion} are how you slice the question, {range} is the reasonable bounds for numbers, {output format} is how to shape the final result.

🚀 Quick start: paste this into the chat:

Here's a template for the REAP technique for accurately extracting facts from an LLM. Adapt it to my task: 
{your task}. Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask which "the fact doesn't exist" criterion fits your topic and how best to split the question into parts, because without that the template won't work for your specific task. It will take the pattern and adapt it to your case.

Limitations

⚠️ Rare entities: for obscure facts the method won't save you. The model simply doesn't know the answer, and no prompt structure can make up for that.

⚠️ Exact numbers: even with a range check, exact values (capacity, area) can be off by more than is acceptable, especially for little-known objects.

⚠️ The cost of decomposition: splitting into parts = more requests = more time and tokens. For short lists it's overkill; use it only when you risk losing data.

How They Studied It

The researchers took a competition on building a knowledge base from LLMs: entrants had to answer 6 types of questions (neighboring countries, place of death, stadium capacity, award winners, stock exchanges, territory area) using a model of no more than 32 billion parameters, with no internet and no additional training. They tested three models, first with a simple "single-shot" request, then with the two-stage REAP pipeline.

The plain request gave about 40% accuracy, and REAP raised it to 60–65%, with the effect stronger on the larger model (24 billion parameters), from 43% to 65%, while on the smaller model (9 billion) the gain was more modest. They also checked which element mattered most: without splitting the query into parts, accuracy dropped by 6 points (the model forgot half the list of award winners); without the "does an answer exist" check, by 8 points (the model started making things up); and without the chain of thought at all, by a whopping 22 points. That turned out to be the key element.

The main insight: the technique doesn't replace the model's knowledge; it helps extract what's already there. That's why larger models gain more from structured prompts than small ones.

Resources

REAP: Relation-Aware Elicitation and Parsing for Closed-Book Knowledge Base Construction from LLMs. Thanh-Dan Bui, Thanh-Trung Do, Tuan-Phong Nguyen, VNU University of Engineering and Technology, Hanoi. Code: github.com/yammdd/AKBC-Shared-Task-2026