The Format Tax: Separate Reasoning from Formatting to Get Answer Quality Back

Option A — TWO REQUESTS (more reliable, +6.8 pp accuracy on average):

REQUEST 1: [Task] — no mention of format
         → Free-form, detailed answer

REQUEST 2: "Take the answer above and format it as [format: JSON / table / XML]"
         → Structured result

---

Option B — ONE REQUEST (simpler, works for ChatGPT/Claude with thinking):

REQUEST: "First work through the task in detail.
         Then, in a separate block, format the conclusion as [format]."
         → Reasoning, then structured output

Both options work in a regular chat, no tools required.

Task: Emma is a product manager at DigitalOcean. She needs to hand her team a structured analysis of three competitors in JSON so it can be loaded into a shared system. She used to ask for JSON right away and got shallow results; important details got lost.

The two-step approach:

Request 1:

Analyze three of DigitalOcean's competitors in the B2B cloud market
in the US: Linode (Akamai), Vultr and Hetzner.

For each: key strengths, weak spots,
pricing position, target segment.
Be specific — examples, numbers, real products.

Request 2:

Great. Now take this analysis and format it strictly as JSON
with the following structure:

{
  "competitor": "name",
  "strengths": ["..."],
  "weaknesses": ["..."],
  "pricing_position": "...",
  "target_segment": "..."
}

Return an array of three objects. Don't add anything extra.

Result: In the first request the model delivers a detailed analysis with concrete details: products, examples, positioning nuances. In the second it neatly packs that same content into JSON without losing meaning. The final JSON is substantive, not an empty template.

An LLM doesn't think and then write — it generates everything at once, word by word. When the prompt says "answer in JSON," the model keeps the format requirement in mind from the very first token. It's like trying to compose a text while making sure every sentence starts with the next letter of the alphabet.

Formatting compresses reasoning. The study showed that structured answers are on average 162 tokens shorter than free-form ones. The model literally trims its logic to fit the structure. The stronger the compression, the higher the chance of a wrong answer. On a chart this looks like a U-shaped curve: errors grow at both extremes — under strong compression and under strong expansion.

Separation removes the competition. When reasoning and formatting happen in sequence, the second step receives a finished, high-quality answer as input. Formatting turns into a simple transformation instead of a mixed game of "think and format at the same time."

Control levers: - Step order — never ask for a format in the first request if the task is complex - Format strictness — the more detailed the JSON schema, the heavier the tax; simplify the schema when you can - The "think first" instruction — works even in a single request; just add an explicit sequence - Thinking mode (o3, Claude with extended thinking) — removes the tax automatically, because reasoning happens separately from the visible answer

Option A: Two requests

Request 1:

{task} — give a detailed, free-form answer.

Request 2:

Take the answer above and format it as {format}.

Structure:
{format_schema}

Return only the structured result, with no extra commentary.

Placeholders: - {task} — your question or assignment - {format} — JSON / table / XML / list with headings - {format_schema} — an example structure or list of fields

Option B: One request

Task: {task}

Produce two blocks:

BLOCK 1 — ANALYSIS:
Work through the task in detail. Think out loud, give arguments, don't cut corners.

BLOCK 2 — RESULT IN FORMAT:
Take the conclusions from block 1 and format them strictly as {format}:
{format_schema}

🚀 Quick start — paste into the chat:

Here is the Format Tax template (separating reasoning from formatting).
Adapt it to my task: {your task}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about the output format you need and the schema structure, because without that it can't build a proper second step. It will take the pattern from the template and adapt it to your task.

⚠️ Complex tasks with thinking mode: "Thinking" models (o3, Claude with extended thinking) already solve the problem internally. No need to split things manually; they separate reasoning from formatting on their own.

⚠️ Simple tasks with a simple format: If you're asking for "a list of 5 bullet points" or "a short one-sentence answer," the tax is minimal. Not worth overcomplicating.

⚠️ Speed and cost: Two requests = roughly twice the tokens and the wait. For quick tasks that can be overkill.

⚠️ Modern closed models: Claude Haiku 4.5, GPT-4o (newer versions) and Grok barely suffer from the tax. If you work with them, the two-step approach gives a smaller boost. For open models (Qwen, LLaMA and others) the effect is strong.

⚠️ Very complex format schemas: The more detailed the JSON structure, the bigger the tax. The "think first" principle helps but doesn't cancel it completely with extremely rigid schemas.

A team from UC San Diego asked a simple question: where does the quality drop come from when you request structured output? Most researchers assumed the culprit was the technical generation mechanism that blocks "invalid" tokens. The researchers tested this directly: they took the same tasks, ran models with and without the technical constraint, but kept identical prompts — and saw that quality dropped equally in both cases. So the culprit isn't the generation technology but the mere presence of the words "answer in JSON" in the prompt text.

They tested ten models (six open, four commercial APIs), four formats (JSON, XML, Markdown, LaTeX) and four task types — math, science questions, logic puzzles and writing. That's 144 experimental cells in total. Of the 39 cells with a statistically significant effect, 92% of the degradation came from the prompt, and only 38% was added by the technical mechanism (and even then overlapping with cases already degraded by the prompt). Three cells showed degradation purely from the mechanism — out of 72 possible. They call this "the prompt dominates."

A curious detail: the researchers initially expected to come back with an answer about the generation mechanism, and instead found the root of the problem several levels higher. The practical takeaway turned out simpler than expected: you don't need to improve generation technology — you just need to change the order of steps in your prompt.

The most effective mitigation is to decouple reasoning from formatting—
either by generating freeform first and reformatting in a second pass,
or by enabling extended thinking within a single generation.
Both recover most of the lost accuracy; varying prompt content alone does not.

The tax is not a fundamental limitation. Some models already resist it,
and decoupled generation reverses it without retraining, suggesting that
the underlying reasoning capability remains intact.

Context: From the Conclusion section — the authors sum up the main practical principle of the work.

💡 Adaptation: a client report

You're writing a marketing report for a client that has to be delivered in a strict template with H2 headings, tables and bullet lists.

REQUEST 1:
Analyze the results of {brand}'s ad campaign for {period}.
Data: {data}
What worked, what didn't, why, and what to do next.
No formatting — just write.

REQUEST 2:
Take this analysis and format it using the template:

## Campaign Results
[key numbers in a table: metric / plan / actual / variance]

## What Worked
[3-5 points with a short explanation]

## What Didn't Work
[3-5 points with a short explanation]

## Recommendations
[3 concrete actions for the next period]

🔧 Technique: an explicit pause in one request

Just add a divider between thinking and formatting:

Task: {task}

Step 1. Think out loud — break the task down in detail, with no format constraints.
---
Step 2. Now, based only on step 1, fill in the template:
{template}

The dashes --- create a psychological and structural pause. The model "closes" the reasoning stage and moves on to formatting with a clean context. Especially handy when you don't feel like making two separate requests.

🔧 Technique: check whether you're paying the "tax" right now

Take any prompt you use regularly. Strip every mention of format out of it. Run it. Compare the quality of reasoning with the original.

If the model answers noticeably better without the format, you're paying the tax. Switch to the two-step approach.

Paper: "The Format Tax" — Ivan Yee Lee, Loris D'Antoni, Taylor Berg-Kirkpatrick (UC San Diego). Preprint, under review.

GitHub: https://github.com/ivnle/the-format-tax

Related work mentioned in the paper: - Tam et al. (2024) — first documentation of accuracy drops under structured output - Park et al. (2024) — grammar-aligned decoding as an alternative approach - Banerjee et al. (2025) — a theoretical argument for why separating reasoning from formatting is necessary