TL;DR
The Toolmaker approach is a division-of-labor rule. The model doesn't eyeball your data. It writes code (a script, a parser, a checker) that does the analysis for it. You then test that code on examples where the correct answer is known in advance. The author tested this on reverse-engineering iPhone data formats: when models read disassembled code directly, most of the results contained errors, while a tool written by Claude recovered everything exactly.
The core pain: model errors are invisible. The output looks flawless: the syntax is valid, the names are plausible, everything compiles and even decodes something. But inside you get swapped types, shifted field numbers, dropped values. Almost 90% of the wrong fields never showed up in real traffic at all, so a "we ran it on real data and it seems to work" check would have missed them. The model never says "I'm not sure." It hands you a coherent but wrong picture.
The method has two parts. First: ask for a tool, not an answer, and check it against independent ground truth (data with a known answer, previously published results, real measurements). Second: if you truly need direct analysis at scale, cut the job into small units and hand each one to a fresh agent with a clean context. A single agent doing everything failed almost completely.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
PART A — Toolmaker (the paper's main takeaway)
STEP 1: Do NOT ask the model to analyze the data directly
STEP 2: Ask the model to write a tool (a script) → code + tests
STEP 3: Gather independent ground truth: data with a known answer,
prior results, external measurements
STEP 4: Require the tool to match the ground truth exactly
STEP 5: Only then run it on real data
PART B — when direct analysis is unavoidable (the author's original prompt)
STEP 1: Split the task into units (one class = one unit)
STEP 2: Give each unit to a new agent with a clean context
STEP 3: In the prompt: "not sure — mark it UNRESOLVED, don't guess"
STEP 4: Check the result against ground truth outside the model
Example in Practice
The method shines wherever you can check the result on examples with a known answer: reconciliations, parsing, data exports, counts. It's weaker where the "right answer" is subjective (tone of voice, creative work): you can't build ground truth for those.
Task: The finance manager at an Amazon seller has pulled a 30,000-row settlement report and an export from QuickBooks. They need to find amount mismatches. If you just drop both tables into the chat and ask "find the mismatches," the model will give you a plausible list. But it will miss some mismatches and invent others. You won't notice.
Prompt:
I'm reconciling an Amazon settlement report (file amazon.xlsx) against a QuickBooks export (file quickbooks.xlsx).
Don't analyze the data yourself. Write a Python script reconcile.py that:
— matches rows by order ID;
— outputs amount mismatches, rows missing from either table, and duplicates;
— saves the result to mismatches.xlsx.
Then build a test set where the answer is known in advance:
— 20 rows where everything matches;
— 5 rows with an amount mismatch (including cents and negative amounts — refunds);
— 3 rows that exist only in Amazon;
— 3 rows that exist only in QuickBooks;
— 2 duplicates.
Write down up front what result the test set should produce.
Run the script on the test set and show whether the result matches the expected one exactly.
If it doesn't, fix the script and repeat.
If any matching rule is unclear to you, don't guess — ask me.
Only after an exact match, run it on the real files.
Result: The model first delivers the script and a test set with the expected answer. Then it shows the run: what the script found versus what it should have found. If there are differences, it fixes them and reruns. On the real files you get a mismatch table produced by verified code, not by the model's "gut read." As a bonus step, pull 10–20 mismatches from the output and check them by hand.
Why It Works
The model's weakness. When a model reads data itself, it tends to produce a consistent picture. Small errors (a swapped type, a skipped row, shifted numbering) don't break the overall coherence, so they go unnoticed. One early mistake quietly corrupts everything that depends on it. A "seems to work on real data" check is unreliable here: the errors hide in rare cases that real data barely touches.
The model's strength. Writing code against a clear spec is something models do well. A tool is deterministic: same input, same output, every time, and you can verify it. The author got a 21,000-line tool with tests, written entirely by the model. It produced results identical to the ground truth and ran in a couple of minutes with zero model calls.
How the method sidesteps the weakness. It moves trust from the model to the check. You don't trust the model, you trust the match with ground truth. The more independent the ground truth is from the model itself, the more reliable the check.
Levers: - Ground truth. The more rare and edge cases your tests cover, the lower the odds of missing hidden errors: that's exactly where errors live. - Unit size. Smaller unit, cleaner agent context. A single agent doing everything failed: it stopped before reaching most of the nested structures, possibly because its context overflowed. - Permission to say "I don't know." An explicit flag instead of a guess cuts down on made-up values. - Independent sources of truth. Checking against previously published results and real measurements catches what tests can't see.
Prompt Template
A. The original per-unit agent prompt (based on the author's description). The full text of the author's prompts lives in their GitHub repository. The paper describes the logic; below it's turned into a template.
You are analyzing one unit of the task: {unit}.
Source: {data_source}. Only these tools are available: {tools}.
External hints, ready-made answers and web search are not available.
For this unit, recover: {what_to_recover_for_each_element}.
If an element references another unit — don't analyze it yourself,
return a reference to it in the "next units" list instead.
If something is NOT visible from the source — mark it as
// UNRESOLVED: <reason>
Don't guess and don't make things up.
If an element is simple by nature ({simple_type}) — keep it simple,
don't invent a complex type for it.
Return: the list of elements + the "next units" list.
Orchestration: you hand each unit from the "next units" list to a new chat or new agent with the same template. You keep a list of units already processed so no work is repeated. In the paper a Python script did this. In everyday work you can keep the shared list by hand or ask an agent with subagents (for example, Claude Code) to maintain it.
B. Adaptation: a Toolmaker request (in the spirit of the paper's conclusion).
Task: {analysis_task}. Data: {data_description}.
Don't analyze the data directly. Write a tool: {language_and_format} that {what_the_tool_does}.
Verification:
1. Build a test set with a known answer: {case_types},
including rare and edge cases: {edge_cases}.
2. Write the expected result first, then run the tool.
3. Require an exact match. If it doesn't match — fix and repeat.
4. If the spec is unclear — ask a question, don't guess.
5. Only after a match, run it on the real data and give me
10 random results to check by hand.
Fill in: {analysis_task} — what you need to get, {data_description} — format and size, {edge_cases} — the rare situations in your data (refunds, empty fields, duplicates, nonstandard encodings).
🚀 Quick start — paste into the chat:
Here's a Toolmaker approach template. Adapt it to my task: [your task].
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what data you have, what result you need and which rare cases come up, because without them it can't build a test set with a known answer. It will take the structure from the template and assemble a ready-to-use prompt for your task.
What the Numbers Showed
- Two models reconstructed the schemas of 17 messages (about 7,800 fields). Claude got 1,069 fields wrong, Codex 1,980. Many of the errors weren't visible.
- On fields that appear in real traffic, accuracy was 94–96%; across all schema fields, 74–86%. A test on real data would have missed almost 90% of the wrong fields.
- One agent for the whole task: 8.9% of fields correct. A fresh agent per unit: noticeably better, but expensive.
- The agent-per-unit approach took about five days and millions of tokens. The tool did the job in roughly 2 minutes with no model calls.
Limitations
⚠️ Ground truth isn't free: Verifying the tool requires an independent source of truth. The author built ground truth through their own manual reverse-engineering. If you have no way to get ground truth, the approach loses its main pillar.
⚠️ One domain: The experiment covered a single task: recovering schemas from Apple binaries. Carrying it over to your tasks (reconciliations, parsing, exports) is a logical leap, but the paper didn't test it separately.
⚠️ Subjective criteria: If the result has no verifiable correct answer (style, creative work, idea evaluation), you can't build ground truth, and the method won't help.
⚠️ Direct analysis is expensive and unreliable: If you do run the model over a large volume anyway, it's costly in tokens. A single agent over the whole volume failed almost entirely.
⚠️ Real-data tests lie: "Ran it on live data, seems to work" doesn't prove correctness. Errors hide in rare cases that real data rarely contains.
⚠️ Incomplete text: The description of the tool itself and its verification is truncated in the available version, so some details aren't visible.
Resources
- Decoding What iOS Location Services Send to Apple: An LLM Is an Unreliable Analyst but a Useful Toolmaker — D. J. Leith, Trinity College Dublin, Ireland.
- The tool (Extractor) and ios_decoder: https://github.com/doug-leith/ios-protobufdecoding
- AI analysis data, including the prompts used, and network measurements: https://github.com/doug-leith/ioslocation-services-study-data