TL;DR
This method combines three tricks: the LLM writes its own working prompt (after studying prompting best practices and domain terminology), several prompt drafts get merged into one "master prompt," and then extraction runs five times in a row — with the answer that shows up most often winning. On top of that, results from several different models (GPT, Claude, Gemini, etc.) get cross-checked — wherever the models disagree, that's your signal: this case is contested, a human needs to step in.
The core pain point: LLMs are inconsistent. The same prompt on the same paper can produce a correct answer in three out of five runs and a wrong one in the other two. In the study, only 76% of extracted values matched across all five consecutive runs, and for some models accuracy swung from 95% in one run to 78% in another. Interestingly, models find raw numbers correctly almost every time (99%), but figuring out which exact condition a number belongs to (which sample, which temperature, which measurement method) turns out noticeably harder (81%).
The method fixes this not with a smarter prompt, but through repetition and cross-checking: five runs of the same prompt plus majority voting cancels out random noise, while comparing multiple models catches systematic errors — including mistakes in the source data that humans themselves missed.
Method Blueprint
STEP 1 (separate model request): Ask the LLM to study prompting best practices for itself + terminology for your topic → draft prompt
STEP 2: Repeat step 1 two more times (same model or different ones, your call) → 3 drafts
STEP 3 (separate request): Give the model all 3 drafts → "merge the best parts of these into one master prompt"
STEP 4: Run the master prompt on your document 5 times (5 separate chats/runs) → take the answer that matched in at least 3 out of 5 runs
STEP 5 (optional, for important decisions): Run the same master prompt in 2-3 different LLMs → if answers disagree, that's a contested case — check it yourself
Example in Action
Task: A venture fund analyst is going through 15 competitor press releases and annual reports to pull data into a table: revenue, margin, number of active customers, and reporting currency/period. Same problem as in the study: the LLM easily finds the raw numbers, but tying them to the right period and currency (say, euros vs. dollars, quarter vs. year) is trickier.
Prompt (steps 1-3, self-authoring):
Study best practices for writing prompts that extract financial data from reports.
Study standard financial reporting terminology (EBITDA, margin, ARR, reporting period).
Based on this, write a prompt that extracts from a financial document:
revenue, margin, number of customers, currency, and reporting period,
with a mandatory quote from the source text for each extracted value.
Output format: a table.
Repeat this 3 times (new chats are fine), then:
Here are three prompts for extracting financial data: [insert 3 drafts].
Merge the best elements of each into one final prompt.
Result: You end up with a ready-made master prompt tuned for financial reports. Run it 5 times on each report (ideally in fresh chats, to avoid context bleed), compile into a table, and take the value that matched at least 3 times. Wherever answers disagreed across all five runs — that's your signal that the document is ambiguous and needs a manual look.
Why This Works
LLMs are inconsistent across runs — the same model can give different answers to the same question, especially when the task requires more than just finding a number: it needs to understand context (which condition that number belongs to). This isn't a bug, it's just how text generation works: the model assembles its answer slightly differently each time.
But LLMs are great at mechanical search and pattern matching — they find numbers with near-perfect accuracy. Majority voting across five runs cancels out random noise: if an error is random, it won't repeat 3+ times, but if it's systematic (say, the model keeps confusing two tables), it'll repeat consistently — and that's useful information too.
Levers you can adjust: - Number of runs (5→3) — dial it down for simple tasks, up for high-stakes decisions. - Number of models for cross-checking — more models means more chances to catch an error, but costs more time. - Requiring a quote next to each extracted value — forces the model to show its sources, making verification easier. - The "contested case" threshold — you can swap majority voting for "any disagreement = human review" for a more conservative approach.
Prompt Template
STAGE 1 — self-authoring the prompt:
Study prompting best practices for the task "{task_type}".
Study domain-specific terminology in the field "{domain}".
Write a prompt to extract the following data from a document: {field_list}.
For each extracted value, provide a quote from the text it was taken from.
Output format: a table with columns {field_list}.
(repeat 3 times, then:)
STAGE 2 — merging:
Here are three prompt variants: {prompt_1}, {prompt_2}, {prompt_3}.
Merge the best elements of each into one final prompt.
STAGE 3 — extraction (repeat 5 times on the same document):
{final_prompt}
Document: {text_or_file}
STAGE 4 — aggregation:
Here are 5 extraction results for one document: {result_1...5}.
For each field, report the value that matched in at least 3 out of 5 cases.
If fewer than 3 matches — flag it as "contested, needs human review."
🚀 Quick Start — paste into chat:
Here's a template for the self-prompting and majority-voting method for data extraction.
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.
[insert template above]
The LLM will ask what data fields you actually need and from which documents — because without that, it can't build a working extraction prompt. It'll take the template's structure and fill it in with the specifics of your task.
Limitations
⚠️ Contextual classification is weaker than numeric extraction: the model almost always finds the number itself correctly, but makes noticeably more mistakes figuring out which condition (sample, method, period) that number belongs to. Don't trust context classification without checking it.
⚠️ Self-written prompts underperform expert-written ones: even the best model, when writing its own prompt, lost a few percentage points of accuracy compared to a prompt written by a human expert. Self-prompting gets you to "pretty good," not "just as good."
⚠️ Autonomous source-finding isn't reliable: if you ask the model to find the relevant papers/documents itself (rather than just processing the ones you give it), it'll either miss most of the relevant sources or start inventing citations that don't exist. It's better to hand it the documents to analyze yourself.
⚠️ More output doesn't mean better output: some models produce a lot of "extra" data rows that don't directly answer the question. This creates an illusion of thoroughness but adds manual filtering work.
⚠️ The method takes time: five runs per document plus the self-prompting stages add up to a series of steps, not a single request. For quick one-off tasks, this is overkill.
How It Was Studied
The team took 18 real scientific papers from the 1980s-2000s on calcium binding to the protein troponin — a complex topic with outdated terminology, data scattered across tables and text, and even scanned images instead of text. Seven top models (GPT, Claude Opus, Gemini, Qwen, Kimi, GLM, DeepSeek) were tested through the regular browser interface — meaning the way most people actually use them, not via API.
Each of four scenarios (expert prompt → self-written prompt → autonomous source-finding → building a new dataset in a different field) was run five times, and the results were compared against a reference dataset assembled by humans. A telling detail: the human experts themselves didn't always agree with each other — their assessments matched in only 79% of cases, which shows that even "ground truth" in complex scientific tasks can be partly subjective.
The biggest surprise: accuracy for the same model could swing by nearly 17 percentage points between runs, and that's exactly what justifies majority voting across five runs as a practical standard — not overkill.
Resources
Romanov V., Bax M., Niederer S. "Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models". Imperial College London, University of Cambridge, Stanford University.