TL;DR
When you hand an agent a Word file or PDF, the converter (docling, "plain text," and the like) keeps the words but throws away text color and highlighting. To the agent, a red requirement, a yellow exception, and an ordinary paragraph all look the same. The paper's method: serialize the document to text so that color and highlighting stay visible, e.g., [COLOR:#FF0000] and [HIGHLIGHT:YELLOW]. Then you need to tell the agent what those labels mean (critical, warning, deprecated).
The painful part is that the pipeline doesn't break. The text is all there, there are no errors, and the agent answers briskly. On the task "list the requirements the author highlighted," quality drops from 0.69 to 0.08–0.15. The agent doesn't say "I can't see any markings"; it guesses. With docling, the false-positive rate on non-highlighted requirements is roughly four times higher than with the labeling method. Other models do the opposite and go almost completely silent. Neither failure is visible without reference answers.
The method has three moves. First: preserve color and highlighting as labels in the text. Second: type the labels, i.e., give each one a role (critical / warning / deprecated). Third: don't label things that appear everywhere (bold, italics, headings). Of all formatting types, only color and highlighting work. Bold, italics, monospace, and strikethrough gave no gain.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Read the .docx/PDF while preserving formatting → text blocks where every run has color/highlight/strikethrough
(code, not an LLM)
STEP 2: Turn formatting into inline labels → **bold** in Markdown, [COLOR:#FF0000], [HIGHLIGHT:YELLOW]
STEP 3: Add a role to the rare labels → red/red highlight = CRITICAL_WARNING,
any other highlight = WARNING, strikethrough = DEPRECATED
STEP 4: Give the agent the labeled text + a legend of what the roles mean → in one prompt
STEP 5 (check): A test document with known markings → verify the agent saw them
Steps 1–2 are done by a script before the agent. Steps 3–5 are written in words in the system prompt.
Example
Task: A bank lawyer maintains an anti-money-laundering policy (along the lines of the Bank Secrecy Act) in Word. In it, mandatory requirements are marked in red, exceptions in yellow highlight, and repealed rules are struck through. A no-code automation agent (n8n, Dify) needs to answer tellers' questions: "What must be checked on a $15,000 transfer?" If the policy goes through an ordinary converter, the colors vanish, and the agent can't tell what's mandatory from what's just reference.
Prompt (system instruction for the agent):
You are an assistant to bank tellers. Answer only based on the policy below.
<MarkupLegend>
The policy text preserves the author's formatting marks:
- [COLOR:#FF0000]...[/COLOR] or red highlight = CRITICAL_WARNING:
the author marked a mandatory requirement. It takes priority over ordinary text.
- [HIGHLIGHT:any color]...[/HIGHLIGHT] = WARNING: an exception or special condition
the author wants you to pay attention to.
- ~~text~~ = DEPRECATED: the rule has been repealed; don't apply it or call it current.
- **bold** is just structure or a term. Do NOT treat it as a sign of importance.
- If there are no marks next to an item, it's an ordinary item. Don't call it
highlighted by the author if you don't see a label.
</MarkupLegend>
<Rules>
1. When asked "what's mandatory," list the CRITICAL_WARNING items first.
2. For each item, give a verbatim quote from the policy.
3. If the policy has no marks at all, say so directly.
</Rules>
<Regulation>
2.4. Transactions over $10,000 [COLOR:#FF0000]are subject to mandatory review
with the basis recorded in the log[/COLOR].
2.5. The customer may confirm the source of funds with a bank statement. [HIGHLIGHT:YELLOW]Exception:
transfers between accounts of the same customer within the bank[/HIGHLIGHT].
2.6. ~~Verbal confirmation of the source of funds is permitted.~~
2.7. The teller maintains the **Review Log** electronically.
</Regulation>
Question: what must be done for a $15,000 transfer by a customer who brought a bank statement?
Result: The model will flag item 2.4 as mandatory in its answer and quote it. It will name exception 2.5 as a separate condition. It won't offer verbal confirmation (2.6) as a valid option. It won't move item 2.7 about the log into "mandatory" just because a word is in bold. Remove the labels and the legend, and the model will very likely start listing everything or miss the key point.
Why It Works
The weakness. An LLM reads tokens, not a page. Everything the author encoded with formatting (red = "don't violate," highlight = "look here") disappears when the converter keeps only the words. The task is nearly unsolvable from words alone: even a classifier trained on the markup achieves results only slightly better than chance from text (AUC 0.66). Without cues, the model is left guessing.
The strength. If the signal is visible in the text, the model uses it. It reads [COLOR:…] and [HIGHLIGHT:…] labels without any training. Its second strength: it follows a legend well. When it's written right there that "red = mandatory," the model uses the label as a priority signal. Typing added another +0.085 on top of bare labels.
How the method connects the two. First the signal is put back into the text. Then it's narrowed down to rare, meaningful features. That's the lever: - What to label. Color and highlighting, because they're rare and deliberate. Every heading is in bold, so bold doesn't "highlight" anything. In the ablation (removing one formatting type at a time), bold, italics, monospace, and strikethrough contributed zero. - How many roles to give. A short set (critical / warning / deprecated). In the full set, about 62% of roles described structure (heading, emphasis). The model took any role as a flag and started marking everything as highlighted. - What not to label. Anything ambiguous, such as green text. The authors excluded such rules: a wrong hint does more harm than no hint. - Highlight by presence, not by shade. The rule "yellow = warning" missed blue and gray highlights. Switching to "any highlight except white" raised paragraph coverage from 79% to 98%.
Prompt Template
<Role>
You are an assistant who answers strictly based on the document below.
</Role>
<MarkupLegend>
The document preserves the author's formatting marks. Read them as priority signals:
- {critical_label} = CRITICAL_WARNING: {what it means in my document}
- {warning_label} = WARNING: {what it means in my document}
- {deprecated_label} = DEPRECATED: {what it means and how to handle it}
- **bold**, *italics*, headings — structure only. They are NOT a sign of importance.
- If an item has no mark, it's an ordinary item. Don't call it highlighted by the author.
</MarkupLegend>
<Rules>
1. For questions like "what's important / mandatory / critical," first list the items
with CRITICAL_WARNING, then those with WARNING.
2. For each item, give a verbatim quote from the document.
3. Do not apply DEPRECATED items as if they were current.
4. If the document has no marks at all, say so — don't guess.
</Rules>
<Document>
{document_text_with_labels}
</Document>
Question: {question}
What to fill in:
- {..._label}: exactly the labels that appear in your text. For example, [COLOR:#FF0000] or "red highlight."
- {what it means...}: the meaning from your practice. The paper stresses that the same formatting means different things in different departments. Don't guess — ask the document's author.
- {document_text_with_labels}: the document after conversion with labels. How to get them is shown in the "Quick start" below.
🚀 Quick start — paste into the chat:
Here's a prompt template for reading documents with formatting marks. Adapt it to my task: [what documents I have and what's marked in them with color, highlight, strikethrough].
Ask me questions to fill in the fields.
[paste the template above]
Also write a short Python script (python-docx) that reads my .docx and outputs the text with [COLOR:#RRGGBB], [HIGHLIGHT:COLOR], and ~~strikethrough~~ labels around the corresponding runs.
The LLM will ask what each color and each highlight means in your documents, who applies them and how, and what questions people will ask the agent. That's because the method rests on the legend: without it, labels remain a bare signal, and typing provided an extra gain. It'll take the pattern from the template and plug in your meanings.
Limitations
⚠️ Only color and highlighting: Bold, italics, monospace, and strikethrough gave no gain in testing. There's nothing to be won by labeling them "just in case." Worse, extra labels clutter the text.
⚠️ Typing gives a modest gain: Preserving labels delivers most of the effect. The legend adds +0.085, and its significance is borderline (p = 0.049), with a wide confidence interval. On two of the six models, the legend didn't help.
⚠️ Models fail in different ways: Without labels, some models list too much, while others nearly refuse to answer (one gave very precise but rare answers, with recall around 0.09). Neither failure is visible without reference answers.
⚠️ You need code or a script: The labels won't appear on their own. Ordinary converters (docling, plain text, some upload tools in chat interfaces) lose color. You need to build a small .docx parser — an LLM can do it — but without this step the method doesn't work.
⚠️ Only one task was tested: Finding requirements the author highlighted. Other document questions, such as exact quoting or comparison, weren't tested.
⚠️ Color meaning isn't universal: The same formatting means different things in different departments. You can't hard-code "red = danger" in advance: you have to ask the author. The authors deliberately excluded ambiguous rules, such as green text.
⚠️ PDFs and scans are harder: PDFs have no "text properties"; color and highlighting are reconstructed from glyph geometry. Results on PDF roughly match the best existing converters, and scans need OCR, with its own errors.
⚠️ It's not just formatting that gets lost: Headers and footers, footnotes, document properties, and accepted tracked changes (the text that's actually in force) aren't reached by an ordinary walk through the document body. The authors discovered this themselves during the work — their first version missed it too.
How They Tested It
The researchers scanned 6,462 real .docx documents from public policies. 32.8% have deliberate red text or highlighting, and in 10.3% it's applied to a sentence containing an obligation. This isn't exotic — it's the normal state of corporate documents. From these, they built a task of 84 documents (332 highlighted requirements and 4,192 non-highlighted ones). The model had to name the requirements the author "marked for special attention." The ground truth came from the file's own properties, not from anyone's opinion. Scoring used no LLM judge: a requirement counts as found if the answer contains four consecutive meaningful words from its beginning.
They compared six ways of preparing the text with the same question. Plain text scored 0.075, docling 0.150, docx2python (which preserves formatting as HTML) 0.586, and the authors' method 0.690. The cleanest conclusion came from the layer-by-layer experiment: one formatting type at a time was removed from the text, and the entire effect came from highlighting (+0.302) and color (+0.262). Bold, italics, monospace, and strikethrough stayed within ±0.003 of zero. Surprisingly, a single document can contain more than a thousand bold marks. They're uninformative precisely because they're too common: an author who bolded every heading hasn't highlighted anything.
The question "is a legend needed on top of preservation" was tested in a pre-registered test: hypotheses and the analysis method were fixed before the run. It was confirmed, but with p = 0.049, right on the edge. The authors honestly showed that after fixing a bug in their own parser (it inserted spurious black labels), the significance got worse while the effect stayed the same. They also tested on Microsoft's external benchmark (326 questions, 59 documents) and on six models from two developers. Plain text failed on all of them (0.007–0.075), meaning the issue is how the document is presented, not the model. The full pipeline with retrieval and package assembly confirmed the gap: 0.503 versus 0.258 (docx2python) and 0.084 (docling).
Adaptations and Extrapolations
🔧 Technique: add "what not to treat as a signal" to the legend → fewer false positives
The paper typed only rare features. The rule "bold is structure" is easy to write straight into the prompt:
- **bold**, headings, italics — just structure. Not a reason to treat an item as important.
- If there's no label nearby, write "not highlighted by the author," not "seems important."
It's the same idea as in the original (a narrow set of roles), but expressed as an explicit prohibition for models that tend to treat any mark as a flag.
Extrapolation: audit the pipeline before launching the agent. The paper doesn't describe this; it's my transfer of the principle "check against ground truth taken from the file itself."
Take one document where you know for sure what's highlighted, and ask the agent:
In this document, the author marked some items with color or highlighting.
List exactly the items that are marked, and for each one state
which mark told you so. If you can't see any marks in the text, write "I don't see any marks."
If the model answers with a confident list while there are no marks in its input, it's guessing. If it honestly says "I don't see any," the pipeline is losing the signal and it's time to fix the conversion. It's a cheap test for "silent loss." It catches what no monitor ever will.
Resources
- Paper: What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion. Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra, Amazon.
- Mentioned in the paper: Microsoft Office Comprehension Benchmark (File-Fidelity, Word), ParseBench, OmniDocBench; converters docling, docx2python, python-docx, Unstructured, pymupdf4llm, mammoth.
- Related: PASTA (steering attention by text style), Dillitzer et al. (attention ≠ accuracy), TexTAR (recognizing bold, italics, underline, and strikethrough in images), Yang et al. (USENIX Security, visual tricks for evading moderation).