TL;DR
Quote-Grounded Extraction (“grounding by quote”) is a way to pull facts out of text so you can actually trust them. First the model casts a wide net for candidates and has to attach a verbatim quote to every one of them. Then each fact runs through narrow checks: the quote is searched for in the text as a literal string, and if the quote names only one side of the relationship, the model rereads the whole page. At the end, five independent calls vote on “yes/no” questions.
The core pain: ask “does this page confirm that customer X uses product Y?” and the model is way too eager to say “yes”. It judges the very same sentence differently: one and the same testimonial on four pages got three “yes” votes and one “no”. The more text the model reads and produces in one go, the worse the result. And a bogus “customer” on the list does more damage than a missing one: it creates false confidence.
The method splits up the jobs. One call only proposes (generously, with a quote). Other calls only verify (narrowly, one criterion each). If the evidence is split (the testimonial names the hospital, the headline names the product), a “read the whole page” step kicks in. The final call is made by a vote, and the share of “yes” votes doubles as a confidence score.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Collect pages (the vendor's entire site + web search on a budget) → text cache
STEP 2: Extraction (generous, 1 call per page) → relationship + verbatim quote
STEP 3: Quote repair (if it was stitched together from fragments) → quote from the original text
STEP 4: Referent resolution: if the quote doesn't name both sides,
the model reads the WHOLE page → argument + yes/no + evidence sentence from the page
STEP 5: Form checks (no model) → is the quote in the text? is it an organization, not “hospitals” or a person?
STEP 6: Panel: 5 identical calls × yes/no questions → a criterion passes if “yes” ≥ half
STEP 7: Accept with quote, URL and timestamp — or reject with a reason code
Steps 2–6 are separate requests. One do-it-all request can't replace them, because the whole method relies on each call staying narrow.
Example in Action
Task: A marketer at a SaaS startup is putting together a comparison against HubSpot. They need a list of the competitor's real customers, with evidence. Sources: case study pages, press releases, the blog. Mistakes are costly here: if the deck names a company as a customer when it isn't one, the sales team ends up with egg on its face.
The typical problem with these pages: the headline names the product (“How X rolled out HubSpot”), while the testimonial below names only the company (“We cut our lead response time in half”). No single quote contains both sides at once.
Prompt 1 (extraction, generous):
You are extracting facts about customers of the product “HubSpot” from the text of a page.
Find every organization that might be using “HubSpot”.
If the wording is ambiguous, include it anyway — we'll decide later.
For each relationship, return:
- organization
- quote: ONE continuous passage copied from the text verbatim.
Grab the neighboring heading or sentences so that the quote
contains both the organization's name and a sign that it uses the product
- evidence type: case study / testimonial / logo / press release / other
Do not count as usage: investments, awards, association membership,
partnerships without an actual deployment.
Page text:
[paste text]
Prompt 2 (referent resolution, reads the whole page):
Here is the full page and a claim: “The company {organization} uses
HubSpot.”
Read the page and answer:
1. Argument: what on the page speaks for or against it (2-3 sentences).
Keep in mind that “we” in a testimonial may refer to the testimonial's author,
and the product name may appear only in the headline.
2. Verdict: YES or NO
3. Evidence sentence: a sentence copied from the page verbatim.
If there is no direct confirmation, the verdict is NO.
Page:
[paste text]
Result: Prompt 1 returns a list of candidates with quotes, some of them shaky. For each shaky candidate, Prompt 2 gives an argument, a YES/NO, and a sentence you can verify with a text search (Ctrl+F). Only candidates with a YES and a sentence that's actually found get accepted. Then five runs of a short judge (template below) weed out the remaining doubtful cases.
Why It Works
The model's weakness. The question “does the page confirm this?” invites agreement, and the answer drifts from run to run. When the model reads a lot and spits out a lot in one go, quality drops. Ask it to “find and verify at the same time” and you get both done worse.
The model's strength. It does well on a narrow task with a clear format: copy a passage, answer “yes/no” to a single question. Copying can be checked with a plain string search. Results become reproducible once you average the answers of several independent calls.
How the method exploits this. The quote turns “do I trust the model?” into “is this sentence on the page?”, and a text search answers that. Splitting the roles removes the conflict: the extractor can err on the side of generosity because filters stand behind it. The referent-resolution step plugs the biggest hole in narrow checks: they lose evidence that's scattered across the page. In the experiments, this step more than doubled the number of relationships found.
Control knobs: - Number of votes (5) → drop to 3 to save money, raise it if answers still drift. - The “half the votes” threshold → raise it to “all five” if a false positive costs more than a miss. - The criteria list → add one question for each error you spot, e.g. “is this an award rather than usage?” - How much text the judge sees → wider means fewer true facts lost, narrower means less noise. In the study, a narrow slice of the page produced lots of false rejections. - Extraction strictness → “generous” for hunting opportunities, “strict” for a report that's going to a client.
Prompt Template
A template of three prompts, one per stage. The paper lists the judge's criteria in Table 8, which isn't in the available part of the text. So the criteria below are examples based on the paper's description — adapt them.
Prompt A — Extractor (generous):
<Role>You extract relationships of the form “{subject} {relation_type} {object}” from text.</Role>
<Task>
Find every relationship that might be present in the text. When in doubt,
extract it — the decision will be made at the next step.
</Task>
<EvidenceRules>
- For each relationship, give ONE continuous quote copied from the text verbatim.
- Grab the neighboring heading or sentences so that the quote contains
both names and the sign of the relationship.
- Do not alter, stitch together or shorten the quote.
</EvidenceRules>
<NotARelation>
{what does NOT count as a relationship: e.g., investment, award, membership}
</NotARelation>
<Output>
List: [subject | relationship | object | quote]
</Output>
Text:
{text}
Prompt B — Referent resolution (reads everything):
<Task>
Claim: “{subject} {relation_type} {object}”.
Read the whole page.
</Task>
<Hint>
Pronouns (“we”, “our team”) refer to the author of the testimonial.
The product name may appear only in a heading or caption.
{rule_for_page_type: the vendor's own page or a third-party one}
</Hint>
<Output>
1. Argument: 2-3 sentences
2. Verdict: YES / NO
3. Evidence sentence: verbatim from the page
No direct confirmation means the verdict is NO.
</Output>
Page:
{page}
Prompt C — Judge (run it 5 times independently, each time in a new chat or call):
Claim: “{subject} {relation_type} {object}”
Page fragment: {fragment_around_quote}
Additional info (if available, from the previous step): {argument_and_sentence}
Answer only YES or NO for each criterion:
1. {criterion_1: does the page present the organization as a user of the product?}
2. {criterion_2: is the subject a specific organization, not “clinics” in general?}
3. {criterion_3: is the relationship something other than an investment, award or membership?}
4. {criterion_4}
5. {criterion_5}
Format: number — YES/NO — one-line reason.
Final rule: a criterion passes if “YES” gets at least half the votes. A relationship is accepted if it passes every applicable criterion. The share of “YES” votes is your confidence score: send anything below the threshold to manual review.
What to fill in: {relation_type} — what you're looking for (“uses”, “integrates with”, “supplies”); {NotARelation} — an explicit list of things that look similar but don't count; criteria — one checkable idea each.
🚀 Quick start — paste this into the chat:
Here is a template for three-step fact extraction with quotes (extractor → referent resolution → 5-vote judge).
Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which relationships you're looking for, what looks like them but doesn't count, and where the evidence tends to be split up. You need those answers to write the “doesn't count” list and the judge's criteria — the method's precision rests on them. For everything else, it'll borrow the pattern from the template.
Limitations
⚠️ Quote ≠ truth: a quote only proves that the page claimed it. It doesn't check whether the relationship is current or real. The vendor may have written about a customer who left long ago.
⚠️ Losing true facts: even the final pipeline misses roughly one in three true facts. Most often they get cut by the judge, who sees only a slice of the page and misses the “Trusted by” heading above a row of logos.
⚠️ The score depends on the reference: measured against the company's manual list, the method got low precision because the list is incomplete. Most of the “errors” turned out to be real relationships missing from the list. Check your results against the pages, not against someone else's spreadsheet.
⚠️ Labeling was done by models: the “correct answers” were assigned by one strong model and rechecked by another. Some verdicts had to be flipped, and a human read only a sample.
⚠️ Integrations are harder than customers: “product — product” relationships are verified noticeably less accurately than “organization — product” ones.
⚠️ Cost and scale: five votes per relationship plus several calls per page add up to a lot of requests. Done by hand in a chat, the method works for dozens of pages; thousands need automation.
⚠️ Risky sources: three aggregator sites produced fabricated relationships and had to be blocked. Third-party pages give lower recall, and often there's no evidence on them at all.
⚠️ Part of the paper is unavailable: the text cuts off at the section on the confidence score. The findings on long context, run-to-run variance and prompt tuning (Section 6) aren't covered here.
How They Studied It
The idea was simple: a relationship gets into the knowledge graph only with a quote. The team took 61 products for healthcare and the catalog of the Elion marketplace, with its manual list of customers and integrations. The pipeline went through all the pages: 3284 pages, 6428 candidates. Each candidate was labeled by a strong model (which read the whole page), and the verdict was rechecked by another model. Because of that, 14% of the verdicts were flipped.
There are two metrics. Against the page-level labels, precision is 0.94 and recall is 0.71. Against the catalog's manual list, just 0.34 and 0.41. But of the 803 pairs missing from the list, 716 (89%) were confirmed by the page. So the list is incomplete — the method isn't bad.
The most telling experiment ran on 266 pages from 21 vendors. Without the referent-resolution step, the method found 133 relationships; with it, 291. Recall jumped from 0.36 to 0.81, and precision went up too (0.89 → 0.93). The manual list didn't register this gain at all. The practical takeaway: measure with the wrong yardstick and a useful step will look useless.
The web-search agent cost $2.19 for all 61 vendors and found evidence for 37% of the customers that aren't on the vendor's own site. From the panel's vote shares, the authors built a confidence score (a simple logistic regression). At the same 0.94 precision, it keeps 0.77 of the true relationships versus 0.63 for the hard rule. The share of “yes” among the five calls alone gives almost the same result as the full model. For practitioners this means counting votes matters more than inventing fancy formulas.
Surprises: the criterion that loses true relationships most often is “the page presents the organization as a user.” It depends on context outside the quote. And roughly one quote in eleven turned out to be stitched together from two fragments: the model skipped the characters between them.
Adaptations and Extrapolations
🔧 Technique: widen the judge's context → fewer true facts lost
Among the relationships the panel rejected, 594 had already been confirmed by the referent-resolution step on the full page, and 391 of those were true. If the judge rejects things the full-page read already confirmed, give it this along with the fragment:
Additional info: the previous step read the whole page and confirmed the relationship. Its argument: {argument}. Its evidence sentence: {sentence}. Take this into account, but check that the sentence actually appears in the fragment or is consistent with it.
That's exactly what the authors did: they passed the argument and the sentence to the judge, labeled as “the previous step's claim.”
Extrapolation: research-agent rules in a system prompt or CLAUDE.md
The paper didn't test this prompt; it carries the ideas of the agentic step over to instructions:
You are researching the customers of {company} on the web.
Limits: no more than 25 searches and 15 page reads.
Start every tool call with a single line: “Reason: …”.
Read only pages whose URLs were returned by search.
Do not read: social networks, job sites, app stores, aggregators {list}.
Record every relationship you find with a verbatim quote and the URL.
If there is no quote, don't record the relationship.
At the end, return the pages worth saving and the reason for each one.
Resources
- Paper: Every Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents — Juan Felipe Duran, Arthur Tobler, Martin Vallières (McGill University, Montreal; Toboggan Labs). Preprint.
- Catalog: Elion — elion.health
- Attribution: Rashkin et al. (the idea of tying claims to sources)
- ReAct agent: Yao et al.
- Tools: LangGraph, OpenRouter, Braintrust, Firecrawl, Graphiti, Neo4j