TL;DR
PARCEL is a benchmark (a test set) for checking claims against a document. The model gets the entire document and a single claim (in legal style: "the court held that…"). It has to pick one of three labels: supported, refuted or not in the document. This is how the authors find out whether a model can be trusted in the role of "checker."
The headline finding: models are good at catching a direct contradiction ("the court reversed" instead of "the court affirmed"), but noticeably more often miss a lack of support. If a claim sounds plausible and fits the document's topic, the model answers "supported" even though the text says no such thing. The most dangerous case is a made-up but fitting citation: to a case or a rule that isn't in the document. High overall accuracy hides this. One model scored about 80% accuracy yet "supported" almost half of the claims that weren't in the document.
The practical takeaway: checking "does the document support this" is a separate and weaker operation than "does the document contradict this." You should judge a checker by its rate of false "supported" verdicts, not by average accuracy. And always give the model an honest third option: "not in the document."
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Take the full document text (premise) + ONE claim (hypothesis)
STEP 2: Short instruction: the task + three labels
SUPPORTED / REFUTED / NOT_FOUND
STEP 3: The model picks one label (one request per claim)
STEP 4: Scoring: count not only accuracy,
but the rate of false "supported" separately:
— on refuted claims
— on claims that aren't in the document
It's all one request per claim, no chains.
Example in Practice
Task: An in-house lawyer at a small company is leasing space in a shopping mall. A paralegal sent over a memo with four points, each citing a clause of the lease. The lawyer wants to quickly see which points the lease doesn't actually back up before taking them into negotiations with the landlord. A point like "Section 7.4 caps annual rent escalation at five percent" can easily sound plausible, while the lease may contain no such clause.
Prompt:
<Task>
Check the claim against the text of the lease. Rely ONLY on the lease text below.
Do not use general knowledge of how such leases are usually written.
</Task>
<Document>
[full text of the lease]
</Document>
<Claim>
Section 7.4 caps annual rent escalation at five percent.
</Claim>
<Labels>
SUPPORTED — the lease text contains enough grounds for the claim to be true.
REFUTED — the lease text directly contradicts the claim.
NOT_FOUND — the lease text does not contain enough information to support or refute the claim.
</Labels>
Pick exactly one label. Then explain your choice in one line.
Check each of the four claims in a separate response.
Result: For each claim the model returns one label and a short explanation. Claims with a wrong outcome or a swapped condition are more likely to be caught as REFUTED. Claims that simply aren't in the lease are the danger zone: some of them may get SUPPORTED. So the lawyer still checks everything marked SUPPORTED against the lease text.
Why It Works
The model's weakness. It reads the document and the claim together and judges how well they "match in meaning." If the claim is written in the document's style, uses its vocabulary and resembles something that could have been there, the similarity is high. A missing fact leaves an "empty spot," and an empty spot is harder to notice than an outright mismatch. On top of that, models tend to agree and fill in the picture. That's why "supported" on an absent claim shows up more often than "supported" on a refuted one.
The strength. The model finds direct contradictions confidently: when the text says one thing and the claim says another, the comparison is clear-cut. The best models handle the third label almost perfectly too. So with a well-framed task and a strong model, checking works.
What the method gives you. Three explicit labels with a defined NOT_FOUND turn a vague "check this" into a choice among concrete options. Counting false "supported" verdicts separately exposes a weak spot that average accuracy hides. Knobs: - The NOT_FOUND definition: the stricter it is ("enough grounds" vs. "anything similar"), the fewer false supports. The paper didn't test this. - "Rely only on the document": the paper notes that the model's internal knowledge can override strict adherence to the text, so banning general knowledge makes sense. - One claim per request: that's how the benchmark does it. A batch of claims in one request wasn't tested in the paper.
Prompt Template
The exact prompt text isn't given in the available part of the paper. The authors describe it as a minimal system prompt with the task and the labels. Below is a reconstruction from that description.
<Task>
Determine how the document relates to the claim.
Rely only on the document text.
</Task>
<Document>
{document}
</Document>
<Claim>
{claim}
</Claim>
<Labels>
SUPPORTED — the document contains enough grounds for the claim.
REFUTED — the document directly contradicts the claim.
NOT_FOUND — the document does not contain enough information to support or refute it.
</Labels>
Pick exactly one label.
Fill in:
- {document}: the full source text (a contract, a court ruling, a policy, a report);
- {claim}: one statement to check.
Limitations
⚠️ The paper text is incomplete: the available version lacks the tables breaking results down by perturbation strategy and the analysis of "hard" claims. The conclusion that fabricated citations are the hardest category comes from the abstract.
⚠️ No fix was tested: the authors measure the problem but don't test prompts that reduce it (quote-as-evidence, self-checking, multiple passes). Everything in the "Adaptations" section below is a hypothesis.
⚠️ Legal domain and synthetic claims: all perturbations were generated by an LLM following set strategies. Real mistakes by real people and models may look different.
⚠️ The whole document in context: the tests used short and medium texts fed in full. For huge corpora, where you first have to find the right passage, the finding isn't proven.
⚠️ Weak models: one of the seven models showed a heavy skew toward "supported" and missed almost half of the absent claims. The risk is higher for less capable models.
How They Studied It
The authors took 85 recent decisions of a New York State appellate court from 2024–2025. First they collected about 700 parenthetical explanations attached to citations by real lawyers (in legal writing this is how you briefly summarize what a case held) and picked 180 "substantive" ones as style samples. Then for each decision they generated three types of claims: supported (849), refuted (1,698) and "not in the document" (849). That made 3,396 in total. Refuted claims were made in four ways: entity swap, outcome reversal, dropping a condition, and attributing the ruling to the wrong judge. The "absent" ones were made in three ways: a phantom citation, "the court did not address the argument," and a made-up detail. The key design detail: perturbations were generated with the full text of the decision in view, so they sound like the judge and fit the topic.
Then seven models solved the task with no examples and no clever prompts. The best model reached about 97% accuracy, almost all the others were above 90%, and Llama-3.3-70B only 81%. But the authors introduced a separate metric: the false endorsement rate (FER). The gap held up: GLM-4.7's false supports on refuted claims were 1.18%, but on absent claims 9.55%. Llama's false-support rate on absent claims was 48.76%. Across all models, 15.4% of absent claims were accepted as supported, versus 4.3% of refuted ones.
The surprise was that ranking models by accuracy and by safety doesn't give the same order: Kimi is slightly more accurate than GLM, but GLM has fewer false supports. The practical takeaway: a checker's average accuracy hides its most expensive mistake, so you need to measure that mistake separately.
Adaptations and Extrapolations
🔧 Technique: require a quote for SUPPORTED → harder to "support" thin air
Add to the end of the template:
If you choose SUPPORTED or REFUTED, give a verbatim quote from the document that your conclusion rests on.
If there is no verbatim quote, choose NOT_FOUND.
This is a hypothesis, not from the paper: it hasn't been measured, but it targets exactly the weak spot that was found.
Extrapolation: a mini-test of your own checker. Before trusting a checker prompt on real work, prepare 15–20 claims about one of your own documents. Some of them should be fabricated, using the authors' techniques.
Here is a document: {document}.
Write 18 claims to test a checker:
— 6 true (different types: outcome, argument, fact, reference to another document);
— 6 refuted (swapped party, reversed outcome, dropped "if" condition, attribution to the wrong participant);
— 6 "not in the document" (a citation to a nonexistent rule, "the court did not address argument X", a plausible detail that isn't in the text).
All of them must match the document's style. Output a list marked with the correct answer.
Then run the claims through your checker prompt and count how many of the "not in the document" ones got SUPPORTED. That's a mini version of the authors' metric.
Resources
- Paper: When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection (ICAIL 2026)
- Authors: M. Mikail Demir, M. Abdullah Canbaz — University at Albany, SUNY
- Dataset: https://github.com/mmikaildemir/PARCEL
- Court hallucination trackers: AI Law Tracker (Polaris Lab), Damien Charlotin's collection
- Related work: Dahl et al. (hallucinations in legal LLMs), LegalBench, LexGLUE