TL;DR
OSVerify is a two-stage check for a text or a solution. First, one request hunts for suspicious spots: across the whole document, then within its parts, then within individual steps. Then each suspicion gets verified by its own separate request with a verdict: “doesn’t exist,” “minor,” or “substantial error.” The solution is rejected if even one substantial error turns up.
Ask a model “check whether everything is correct” and you get a blanket verdict, and a noisy one. On the very same work it sometimes misses the error and sometimes rejects a perfectly fine piece. The reason: hunting for a flaw across the entire text at once is hard. But checking one specific claim like “step 4 calculates sales tax wrong” is something the same model does noticeably better. In the paper, separation quality at the level of an individual error (AUROC, a measure of “how well the verdict tells bad from good”) rose from 0.68 to 0.90. Same model, same data.
The method turns one hard task, “evaluate everything,” into two easy ones: “throw out suspicions” (generously, extras welcome) and “check one suspicion” (strictly, with context). The second stage weeds out the extra and invented errors. The real finds stay.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1 (proposer): split the solution into parts →
look for errors in the whole, in the halves, in the small chunks →
list of “suspicions” (where, what’s wrong, why it breaks the solution)
STEP 2 (validator, a SEPARATE request for each suspicion):
gets the task + solution + ONE suspicion →
NOT-REAL (made up) / MINOR (nitpick) / SUBSTANTIAL (breaks the solution)
STEP 3 (verdict): at least one SUBSTANTIAL → reject / send to manual review
none → pass it on
(optional: run the validator several times and look at the share of “SUBSTANTIAL” votes)
Steps 1 and 2 are separate requests (different chats or different calls), so the validator doesn’t pick up the proposer’s confident tone.
Example in Practice
Task: The founder of a quick-delivery startup in Austin used ChatGPT to put together unit economics for an investor pitch: average order $18, delivery cost, margin, an 11-month payback per location. Nobody has checked the 40-row spreadsheet. Before the meeting, they need to make sure the math has no holes an investor would spot in five minutes.
Prompt (step 1, proposer):
<Role>You are a tough financial-model auditor. Your job is to find places
where the calculation might be wrong. Better to flag an extra suspicion
than to miss a real error.</Role>
<Task>Below are the unit economics of a quick-delivery service in Austin.
{paste the calculation}</Task>
<Decomposition>
1. Split the calculation into two meaningful halves (draw the line by logic, not by length).
2. Split each half into two parts again.
3. Check for errors at three levels: the whole calculation (logic, assumptions),
the halves, the individual chunks (arithmetic, units, sources of figures).
</Decomposition>
<Output>
A list of suspicions. For each:
- Where: quote the problematic spot
- What’s wrong: be specific
- Why it breaks the conclusion: how it affects the bottom line (payback, margin)
If there are no suspicions at a given level, write “none.”
</Output>
Prompt (step 2, new chat, one suspicion at a time):
<Role>You are an independent reviewer. You’ve been given a calculation and one claim
that it contains an error. Check that specific claim.</Role>
<Calculation>{paste the full calculation}</Calculation>
<Suspicion>{one suspicion from step 1}</Suspicion>
<ReasoningLogic>
1. Check whether the claim is true at all. If not, the answer is NOT-REAL.
2. If the error is real, assess whether it changes the final conclusion
(payback, margin, the decision to invest).
Changes it substantially → SUBSTANTIAL. Doesn’t change it → MINOR.
</ReasoningLogic>
<Output>Verdict: NOT-REAL / MINOR / SUBSTANTIAL + 2–3 sentences of justification.</Output>
Result: Step 1 will produce a list of several suspicions, some of them a stretch. Step 2 will return one of three verdicts for each, with a short justification. Made-up complaints get filtered out as NOT-REAL, nitpicks go to MINOR. What’s left are the ones that actually change the payback. Those are what the founder fixes before the pitch.
Why It Works
Weakness: when a model evaluates the whole thing, it has to hold the entire text in mind while searching for what’s wrong with it. The verdict comes out fuzzy. The same error gets caught one time and missed the next. Run the check 64 times and the result improves, but it quickly hits a ceiling. Simply “more checks” doesn’t help.
Strength: the model is good at checking a narrow, specific claim once it’s been shown where to look. The paper builds on an observation: models often fail to find an error on their own, but fix it when the spot is pointed out. Searching and checking are tasks of different difficulty.
How the method exploits this: the search stage is deliberately generous, high coverage, false alarms and all. The check stage is strict and narrow and throws out the extras. That lowers the overall error rate: what matters gets found, and the rest gets tossed.
Control levers: - Split depth → in the paper, most of the gain comes with two levels (whole → halves → quarters). For short texts, one is enough. - Number of validator runs → the stricter your “don’t lose a good solution” requirement, the more repeats per suspicion you need. For everyday work, 3–5 is enough. - Definition of a “substantial error” → spell out in the prompt what counts as minor and what’s critical for your task. In a proof, small inaccuracies are tolerable; in production, they can cost you an outage. - Repeat search at the root → running the proposer on the whole document two or three times boosts coverage.
Prompt Template
Proposer:
<Role>You are a nitpicky reviewer. Find places where the {material_type}
might be wrong. Better an extra suspicion than a missed error.</Role>
<Task>{task_or_goal}
Material to check: {text}</Task>
<Decomposition>
1. Split the material into two logical parts (draw the line by meaning).
2. Split each part into two more.
3. Look for errors at every level: the whole (logic, assumptions),
the parts (consistency), small chunks (facts, figures, wording).
</Decomposition>
<Output>
A list of suspicions. Each one:
- Where: {quote}
- What’s wrong: {specifically}
- Why it breaks the result: {how it affects {success_criterion}}
No suspicions at a level → write “none.”
</Output>
Validator (new chat, one suspicion at a time):
<Role>You are an independent reviewer. Check ONLY the stated claim
about an error.</Role>
<Material>{full text}</Material>
<Goal>{task_or_goal}</Goal>
<Suspicion>{one suspicion}</Suspicion>
<ReasoningLogic>
1. Is the claim true? No → NOT-REAL.
2. True → does it change the {success_criterion}?
Yes, substantially → SUBSTANTIAL. No → MINOR.
</ReasoningLogic>
<Output>Verdict + 2–3 sentences of justification.</Output>
Final rule: at least one SUBSTANTIAL → the material fails.
What to plug in: {material_type}: a calculation, contract, report, code; {success_criterion}: what makes an error substantial (money, legal risk, whether the thing works at all).
🚀 Quick start: paste this into the chat:
Here’s the OSVerify template (two-stage check: hunt for suspicions → check each one separately).
Adapt it to my task: {what I’m checking}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what you’re checking and what counts as a substantial error in your case. Without that, the validator can’t tell a nitpick from a gaping hole. It’ll take the two-request structure from the template and tailor it to your task.
Limitations
⚠️ Tested only on mathematical proofs: every benchmark is olympiad-level or research-level math. The paper didn’t test transfer to financial models, contracts, code, or prose. It’s a reasonable guess, but still a guess.
⚠️ The full method takes many calls: the authors ran 16 repeat checks per suspicion and tuned the threshold on a calibration set. You can’t replicate that by hand in a chat; you need a script or automation. A simplified version works in chat.
⚠️ A human stays in the loop: the method doesn’t replace an expert. It cuts the number of bad candidates that reach a human, but doesn’t guarantee that the ones that pass are correct.
⚠️ Generous search breeds invented errors: without a strict second stage, the method turns into a false-alarm generator. A validator with a fuzzy definition of “substantial” will let them through.
⚠️ The “don’t lose good ones” guarantee requires calibration: to almost never reject correct solutions, the threshold is tuned on a set of known-correct samples. Without them, you don’t get that tuning.
⚠️ Fine-tuning the validator (RL) is out of reach for readers: it lifts the gain from 46% to 57%, but you can’t replicate it without training a model. The results below are described without fine-tuning.
How They Tested It
Researchers from Stanford tested the method on three math datasets: Putnam100 (olympiad problems), Hard2Verify (solutions with expert step-level annotations), and FirstProof-Eval (research level, hand-annotated). They deliberately picked relatively weak models as verifiers, gpt-oss-120b and gpt-5-mini. This mimics the situation where the problem is harder than the model can solve on its own. Reference solutions and formal checking programs were hidden from the verifier.
They compared against three approaches: a plain “LLM-as-judge,” voting (16 independent verdicts), plus multi-agent debate and one more verification method. They measured two things: how many bad solutions got filtered out and how many good ones were lost. In practice it boils down to one question: how many wrong candidates will a human have to look at if you lose no more than 5% of the correct ones.
Result: at a 5% loss of correct solutions, the share of bad ones filtered out for gpt-oss-120b rose from 38% to 70% on Putnam100 and from 36% to 58% on FirstProof-Eval. The number of bad candidates reaching a human dropped by 46% on average. Fine-tuning the validator pushed that to 57%.
A curious detail: simply adding more votes quickly hits a ceiling. The gain came from something else: checking a specific error instead of the whole solution. Splitting deeper than two levels is barely needed.
The method was also tested on real papers. In a paper from Annals of Mathematics it found 45 confirmed issues, and in a fresh preprint, 88. The papers’ authors acknowledged the errors and accepted the fixes. It also helped settle three open conjectures from the Langlands program, with experts confirming the proofs.
Adaptations and Extrapolations
🔧 Technique: drop “better too many” → fewer false alarms, but a higher risk of misses
If wading through suspicions wears you out, replace “Better to flag an extra suspicion than to miss” in the proposer’s role with “Only propose suspicions you’re at least half sure about.” You’ll get fewer suspicions, but some real errors won’t make the list. For critical documents, stick with the original.
Extrapolation: a rule for a reviewer agent in your instructions file
This is my idea; it’s not in the paper. If you have Claude Code or Cursor producing code or documents, add this to CLAUDE.md:
## Pre-delivery review
Before reporting that the work is done:
1. Make a list of specific suspected errors (where, what’s wrong, how it breaks the result), separately for the whole change, for each file, and for each function.
2. Check each suspicion separately, as an independent reviewer: NOT-REAL / MINOR / SUBSTANTIAL.
3. If there’s at least one SUBSTANTIAL, fix it and repeat. If not, ship it.
Resources
- One-Sided Verification Under Limited Expert Review — Fan Nie, Haotian Ye, Shurui Liu, Zihao Wang, Stefano Ermon, James Zou, Stanford University
- Project site: https://osverify.github.io/
- Code: https://github.com/OSVerify
- Benchmarks: Putnam100, Hard2Verify, First Proof (FirstProof-Eval)