TL;DR
This method splits data extraction into two steps: first the model identifies the document type (e.g., "invoice" or "contract"), then it gets a narrow prompt built specifically for that type — only the relevant fields and 2-4 examples, with no instructions for all the other types cluttering things up. Instead of one giant "one-size-fits-all" prompt, the model works with a clean, focused rule set.
When a prompt packs in instructions for 16 different document types at once, the model's attention gets spread thin across relevant and irrelevant parts alike. It sees rules for types that don't even appear in this particular document, and that gets in the way of accurately extracting the fields it actually needs — formats get mixed up, details slip through, and error rates climb. The authors call this "attention dilution": the more clutter in the prompt, the weaker the signal from the instruction that actually matters.
The method fixes this by classifying before extraction: first have the model identify the task type from a predefined list, then assemble (or have the model assemble) a prompt using only the rules relevant to that specific type. The model gets a narrow, purpose-built context instead of an all-you-can-eat buffet of instructions.
Method Blueprint
STEP 1: Classify the document/task type → a type label (e.g., "Invoice")
STEP 2: Assemble a prompt for that type: general format rules + fields specific to this type + 2-4 examples → finished prompt
STEP 3: Have the LVLM extract data using the assembled prompt and the image → structured JSON
In the original setup, steps 1 and 2 are handled by separate systems (a classifier + a templating engine). In a regular chat, it's easier to combine them into a single request, where the model first determines the type "internally" and then applies only the relevant block of rules.
Example in Action
Task: A small business owner receives a mixed bag of documents from vendors — service reports, invoices, bills of lading, delivery notes — and wants to use ChatGPT/Claude with photo uploads to automatically pull key data into a spreadsheet, without manual entry.
Prompt:
You are a document-processing specialist.
STEP 1 — Identify the document type in the photo from this list:
- Invoice
- Service Completion Report
- Universal Transfer Document (UTD)
- Delivery Note
Don't write an explanation, just determine the type and move to step 2.
STEP 2 — Extract the data using the rules below.
General rules for all types:
- Output clean JSON only, no comments
- Dates in "year-month-day" format
- If a field is missing or unreadable, write "not determined"
- Extract ONLY the fields listed for the type identified in step 1
Type-specific rules (use only the ones matching the identified type):
Type "Invoice": extract these fields — invoice number, date, vendor, amount, VAT
Sample output: {"number": "145", "date": "2024-03-11", "vendor": "Acme Corp", "amount": "45000", "vat": "7500"}
Type "Service Completion Report": extract these fields — report number, date, contractor, client, list of services, amount
Sample output: {"number": "22", "date": "2024-02-01", "contractor": "John Smith Contracting", "client": "Acme Corp", "services": "equipment installation", "amount": "80000"}
[add similar blocks for UTD and Delivery Note]
Document to analyze: [attach photo]
Result: The model first determines the document type within its response (or silently, if you ask it not to show that step), then outputs clean JSON containing only the fields relevant to that type — with no attempt to cram in fields from other document categories.
Why This Works
When rules for several task types are jumbled together in one prompt, the model spreads its attention across the entire instruction text — including parts that have nothing to do with the current document. That weakens the "signal" coming from the rules that actually apply.
LLMs excel at narrow, specific instructions paired with clear examples — that's their sweet spot: the tighter and shorter the task, the more precise the execution.
Classification "narrows" the task down to one specific case before the model starts extracting data. Only the relevant slice of instructions makes it into the prompt — the model doesn't waste "attention" on the rest.
Levers you can pull: - The type list in step 1 → expand it for your own use case (not just documents — this works for email types, customer inquiry categories, or report formats too) - Number of examples (2-4) → more examples boost accuracy but lengthen the prompt - The "not determined" rule → swap in your own behavior for missing fields (e.g., "leave blank" or "flag for review")
Limitations
⚠️ Dependency on step one: if the model misidentifies the document type in step 1, everything downstream is built on the wrong prompt — extraction proceeds using the wrong rules. This method only pays off when classification is accurate.
⚠️ Requires a closed list of types: this method works when you have a finite set of known categories with clearly defined fields. For one-off, totally unpredictable documents with no clear category, it doesn't add much — you'll be back to a single generic prompt.
⚠️ Poor photo quality: the no-fine-tuning version (the one usable in a regular chat) handles stamps, watermarks, and low contrast noticeably worse than the version with additional model training — and that requires code and GPUs, which isn't available in chat.
How It Was Tested
Researchers assembled a real-world production dataset from an e-procurement portal — nearly 100,000 photographed documents across 16 types (licenses, diplomas, social insurance certificates, and others), complete with real-world noise: stamps, watermarks, and camera-capture distortions. They compared the method against a strong baseline — a separately trained OCR + entity-extraction pipeline specifically tuned for each document type.
The result was surprising: the method with zero fine-tuning (just classification + a narrow prompt run on a plain Qwen2.5-VL-7B model) beat the specially trained baseline by almost 18 percentage points in accuracy. The logic is simple — the trained pipeline is tuned for each type separately, but it suffers from cross-type transfer issues and error accumulation along the OCR → extraction chain. Classification plus a narrow prompt gives the model less noise for the same amount of information.
Additional fine-tuning of the model itself pushed results even higher and especially helped with stamps and low-quality photos — but that requires serious infrastructure (8 GPUs, several days of training) and falls outside what's practical in a regular chat.