TL;DR
NumTrace is a benchmark that checks every number in an agent's answer with three separate verdicts: value (the digits match the source), binding (the number refers to exactly what's claimed) and support (the cited source backs it up). A binding is five fields: who (company or group), what (the metric), when (the period), in what (the unit) and at what scope (group or single legal entity, plan, segment). This yields a simple technique: require the model to produce a "passport" for every number and check the binding separately.
The headline finding: agents almost never get the digits themselves wrong; they get wrong what the number refers to. A model honestly finds 256 shares in a document and passes them off as the holding of the entire fund group, when the real group total is 2,854,645. A "the number is in the source" check will miss this. A "compare with the reference answer" check will say "wrong" but won't explain why. The model latches onto a similar-looking row in the table and doesn't notice that it has a different scope, period or unit. Between 68 and 82% of all incorrect claims from the three tested agents are exactly this kind.
The method requires recording, for each number, the source, five binding fields and how it was obtained. Then you run three checks: value, binding, support. The binding is checked by a human or by a second model pass over the document.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Agent extracts a number → quote from the source (document, location)
STEP 2: Agent states the binding → who / what / when / unit / scope
STEP 3: Agent states the method → "copied" or "computed" (formula + input numbers with quotes)
STEP 4: Value audit → do the digits and arithmetic match the quote?
STEP 5: Binding audit → does the cited number have the same who/what/when/unit/scope?
STEP 6: Support audit → is the source primary (not outdated) and does it confirm both the number and the binding?
Steps 1–3 can fit into one prompt or the agent's instructions. Steps 4–6 are better done as a separate request, ideally in a new chat.
Example
Task: A financial analyst at an investment firm is preparing an investment memo on Capital One Financial. He has uploaded the annual 10-K report and the investor presentation into the chat. He needs to pull net income for the year and compare it to the prior year. The reports have different scopes (the holding company or the bank subsidiary alone), different periods (full year or Q4) and different units (millions or billions of dollars). Here it's easy to mistake a plausible number from a neighboring table row for the right one.
Prompt:
You are helping prepare an investment memo. Attached are the annual report and the investor presentation.
Task: find the group's net income for 2024 and for 2023 and calculate the growth rate in percent.
Answer rules. For EVERY number that goes into the answer, fill in a passport:
<NumberPassport>
<Value>the number exactly as printed, with its unit</Value>
<Source>document, page/section, verbatim quote of the table row</Source>
<StatedBinding>
<Entity>whose figure this is (group / bank / segment)</Entity>
<Concept>which metric exactly (e.g., "net income available to common stockholders" or "net income for the period")</Concept>
<Period>for what period</Period>
<Unit>unit and scale ($ millions, $ billions)</Unit>
<Scope>scope: consolidated or standalone, GAAP or regulatory filing</Scope>
</StatedBinding>
<Derivation>copied OR computed (formula and input numbers, each with its own quote)</Derivation>
</NumberPassport>
If the document has several similar numbers (different scopes, periods, metrics), list them in an <Alternatives> line and explain why you chose this one.
If the required number is not in the document, write "not found in the document." Do not substitute something close in meaning.
Result: The model will produce a passport for each number: 2024 net income, 2023 net income, plus the growth formula with both inputs. Each passport will have a verbatim quote and five binding fields. Where the report has competing numbers (for example, income before and after noncontrolling interests), a list of alternatives with an explanation of the choice will appear. The analyst can quickly check the quote against the PDF and spot a mismatch in scope, period or unit if there is one. If the number isn't in the documents, the model will say so.
Why It Works
Weakness. The model searches the document for a chunk of text that resembles the question and copies the number. Copying it does almost perfectly. But the meaning of a number is often not printed anywhere next to it: units depend on the filing date, an amendment replaces rows or adds them, a group total depends on the list of members. The model can't tell "similar" from "correct," and the answer looks convincing.
Strength. If you ask it to explicitly name which "who, what, when, in what, at what scope" the number refers to, and to provide a quote, the model is forced to compare what it claims with what's written. Checking a match is easier for it than picking from a dozen similar rows on its own. And a human compares two lines faster than rereading a report.
How the method uses this. The passport turns an invisible error ("the number is right, but it's the wrong one") into a visible mismatch: the quote says one thing, the stated binding says another. Three separate verdicts show exactly where the break is. If the value is right but the binding isn't, the number came from the wrong place. If the value is wrong too, the error is in copying or arithmetic.
Control knobs:
- Remove fields the task doesn't need (e.g., Unit for dimensionless metrics) to keep the passport shorter.
- Add your own binding fields for your domain: "currency," "sales tax: included or excluded," "GAAP/IFRS" in accounting, "before/after fees" in sales.
- Tighten the refusal rule ("not found") if errors are costly, and loosen it if you just need a draft.
- Split the roles: one model fills in the passports, another, in a new chat, checks the three verdicts.
Prompt Template
Template 1 — for the agent or chat that extracts numbers (paste into the system prompt or at the start of the request):
For EVERY number in your answer, fill in a passport. Do not include numbers without a passport in the answer.
<NumberPassport>
<Value>{number as printed, with unit}</Value>
<Source>{document, section, verbatim quote of the row}</Source>
<StatedBinding>
<Entity>{whose figure this is}</Entity>
<Concept>{which metric exactly}</Concept>
<Period>{for what period}</Period>
<Unit>{unit and scale}</Unit>
<Scope>{{scope: e.g., group/legal entity/segment, GAAP/IFRS}}</Scope>
</StatedBinding>
<Derivation>copied | computed: {formula}; inputs: {number 1 + quote}, {number 2 + quote}</Derivation>
<Alternatives>{similar numbers in the document and why they don't fit}</Alternatives>
</NumberPassport>
Rules:
- Derive the binding from the document, not from the wording of my question.
- If the required number is not in the document, answer "not found" and explain what exactly is missing. Do not substitute a number close in meaning.
- Check units and scale against the table header and the document date, not out of habit.
Template 2 — auditor (a separate request, ideally in a new chat):
You are an auditor of numerical claims. Below are the number passports from an analyst's answer and the source documents.
For each passport, give three separate verdicts: pass, fail or not applicable.
<Audit>
<Value>do the digits and arithmetic match the quote? (ignore scale for now)</Value>
<Binding>does the number in the source have the same who/what/when/unit/scope as stated in StatedBinding? Check each of the five fields separately and state which one didn't match</Binding>
<Support>is the source primary (not outdated, not a paraphrase) and does it confirm both the number and the binding?</Support>
</Audit>
Flag in particular the case "Value=pass, Binding=fail": the number was copied correctly but refers to the wrong thing.
Passports: {passports}
Documents: {documents}
What to fill in. {number as printed} and the other fields are filled in by the model. You need to put the scopes typical for your field into {{scope...}}. In template 2, paste the output of the first step into {passports} and the same files into {documents}.
🚀 Quick start — paste into the chat:
Here is the "number passport" template and the auditor template. Adapt them to my task: {your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what documents you're using and which numbers you need. It will also ask where numbers most often get mixed up in your field: scope, period, units or document versions. That way the binding fields reflect the real traps in your documents instead of generic filler.
Limitations
⚠️ No validated prompt: The authors did not test whether a "number passport" reduces an agent's errors. They built a benchmark and measured how often binding errors occur. The templates above carry over the record structure the authors used to grade answers. The passport's effect needs to be tested on your own documents.
⚠️ Checked with code, not a prompt: Grading in the paper was done by a deterministic script with reference answers based on accounting rules. In a chat, the audit is done by another model or a human, and they can make mistakes too. The passport doesn't replace checking the quote against the original.
⚠️ Narrow material: The documents are US SEC filings. Other fields have other traps, and you'll have to pick the binding fields yourself.
⚠️ Some errors aren't about binding: The agents searched documents with weak retrieval and a limited number of queries. For some questions, the needed passages simply weren't found. The passport doesn't fix that.
⚠️ The model comparison isn't apples to apples: Two agents ran through coding interfaces, the third through an API. You can't say "model X is better than Y." The conclusion about the error class is robust; the ranking isn't.
⚠️ The automated check made mistakes too: Expert review found scorer errors in both directions. According to the expert, the share of correct bindings is lower than the script showed (88.0% vs. 90.6%).
How It Was Studied
The researchers assembled a frozen snapshot of 2,163 SEC EDGAR documents and wrote 540 questions. 120 require ordinary financial statements, 200 require executive deferred compensation tables, and 220 require 13F reports on who owns shares. The last two types had barely appeared in benchmarks before. Traps were built in deliberately: units depend on the filing date, amendments sometimes replace and sometimes add rows, a group total depends on the list of members.
Three agents went through the same four-step pipeline: retrieval, extraction, calculation, final report. Every number in the report got a record with its source, stated binding and three verdicts. The authors also verified their own reference: a financial analyst with eight years of experience checked 200 questions (96.1% field-level accuracy), and a programmer reimplemented the rules from scratch and matched in 443 of 444 cases.
Results: numbers are copied almost perfectly (the value is correct for 95–99% of claims), but binding is worse, especially in 13F (65–83% correct). For group totals, about 91% of claims pass the value check, but fewer than half pass the binding check. RNWA accounted for 67.5–81.8% of all incorrect claims. Most surprising, the check itself turned out to be a source of errors: after fixing 18 classes of scripting artifacts, Claude's share of correct bindings rose from 70.8% to 93.8% without a single change to the model's answers. Practical takeaway: if a check doesn't separate "value" from "binding," it either misses errors or creates false ones.
Adaptations and Extrapolations
🔧 Technique: add a mandatory refusal → the model won't substitute a nearby number
In the benchmark, some questions deliberately have no correct number: the field isn't disclosed or the needed table doesn't exist. The right answer in such cases is "not disclosed," not a guess. Carry this into the prompt:
If the metric is not disclosed in the document, or is disclosed at a different scope/period, answer "not found in the document" and list which numbers are nearby and how they differ. Using them in place of the required one is not allowed.
This guards against the model grabbing the nearest row just to produce a number.
Extrapolation: "document traps" before extraction
The authors knew three traps in their documents in advance: units by date, amendment type, group membership. Do the same for your files: first get the model to list the places where a number is easy to mix up, then extract.
Before searching for numbers, study {documents} and list where confusion is possible:
— which units and scales are used in different tables;
— which periods and scopes (group/legal entity, GAAP/IFRS) appear;
— which metrics have similar names but differ;
— where there are versions and amendments to the document.
For each trap, write which signal in the text tells it apart. Then answer the question: {question}, filling in a passport for each number.
Resources
- NumTrace: Auditing the Number Provenance of Financial Analyst Agents — Tianlei Zhu, Matt White, Xue Liu, Xiao-Yang Liu (Columbia University, UC Berkeley, Linux Foundation, MBZUAI, McGill University, Mila).
- Related work cited by the authors: FinGround, VeriFin, FinFIRST, BigFinanceBench.
- Data sources: SEC EDGAR, Form 13F, Item 402(i), XBRL.