Structured output and JSON prompting: what the research shows

What structured output prompting is

Whenever an answer goes into a script, a database or a spreadsheet, you need it in a fixed shape: a JSON object, a table with known columns, a numbered list that ends with a score. Structured output prompting covers the ways to get there: naming the fields, giving a schema with types and allowed values, showing a filled-in example, and checking the result against the schema.

The naive version is one line at the end of a prompt: "answer in JSON". The studies on this page explain why that line often costs more than it seems, and which additions make the structure both valid and correct.

Why format instructions tax the answer

A model writes token by token and keeps every instruction in view from the first token. A format requirement is one more thing to track, and it competes with the actual task.

• The Format Tax study (ten models, four formats, four task types) found that most of the quality loss came from the format instruction in the prompt, not from constrained decoding: of the cells with a significant effect, 92% of the degradation came from the prompt. Structured answers were on average 162 tokens shorter, which the authors read as reasoning being cut to fit the format. Newer closed models such as Claude Haiku 4.5 and GPT-4o barely paid the tax; open models paid it heavily.
• The same pattern appears for formatting rules in general. In two-pass generation, a single "no commas" rule made GPT-4o-mini lose 31% of its content, and the model's internal state predicted the shorter answer before the first word.
• In long chats the cost grows: the output-stalling study estimates that JSON markup adds +15% to the volume and HTML +20%, budget a crowded context can no longer spare.
• Terminal formats are the first to slip under load. Salience Reminder reports o4-mini following one format requirement 99% of the time and five requirements only 47.5% of the time.

Variants that tested well

• Think first, format second. Ask for a free-form answer, then a separate step that only repackages it. The Format Tax authors report +6.8 percentage points on average for the two-request version; the two-stage prompting case study reports up to 4x higher task success for an agent once reasoning and JSON generation were split. REAP applies the same split to recall from memory: list facts in free text, in small slices, then format at the end. Accuracy on their knowledge-base questions rose from about 40% to 60-65%.
• A schema written for the model, not for a parser. "price": {"type": "string"} leaves the model guessing about currency symbols and separators. PARSE iteratively added field descriptions and regex patterns to schemas and checked each extraction for completeness, grounding in the source and pattern compliance. On HTML extraction (SWDE) accuracy went from 24.5% to 93.14%; the structured checks caught 92% of errors on the first fix iteration.
• Closed value lists and explicit nesting. Schema-guided extraction gave Claude Opus 3 a 14-field schema with allowed values and nested lists of unknown length; 12 of the 14 fields exceeded 90% accuracy, and the two weak fields were ones human experts also found ambiguous.
• Templates as checklists. An empty field acts as a question the model must answer. Customized prompting (role, focus list, output template, "write 'not mentioned' if absent") raised extraction recall by 15%. Schema plus examples shows a filled-in example instead of describing the format.
• Restate the format at the end. A tag at the start plus a reminder at the end brought compliance back from 70-87% to 90-100% in the Salience Reminder tests.

When JSON is the wrong tool

Two studies push back on making the model read or write raw JSON at all.

• Tool calls. Natural Language Tools replaced JSON tool calls with a plain list such as Search - YES, parsed by code. Across 10 models accuracy went from 69.1% to 87.5%; open-weight models gained the most (58.7% to 84.8%).
• Reading large JSON. In the tool-output study, even GPT-4o reached only 77% answering questions over API responses directly. Asking for a Python function that parses the JSON beat direct answers in 13 of 15 models, and adding the schema gave +12%. For a single simple value, a direct answer was sometimes better.
• Editing big JSON. JSON Whisperer asks for patch operations (add, remove, replace) instead of regenerating the document, and replaces list indices with keyed items so deletions do not shift positions.
• Long quotes inside fields. Index addressing has the model return line ranges for long fields and copies the text from the source, avoiding paraphrase.

Limits worth knowing: the Format Tax authors note that models with built-in extended thinking already separate reasoning from the visible answer, so a manual split helps less there. Two-step pipelines also cost extra calls, and PARSE adds an average of +10 seconds per extraction for its checks.

How to apply it

1. If the task needs real reasoning, do not mention the format in the first step. Ask for the analysis, then ask to convert it.
2. Describe every field: what goes in, what does not, the exact format, allowed values. Mark which lists have unknown length.
3. Tell the model what to write when a value is missing instead of letting it guess.
4. Repeat the format requirement right before the answer starts when the task is long.
5. For tool selection, filtering large JSON or long quotes, let code do the structural work.

For the input side of the same problem see delimiters and input formatting; for the failure pattern itself see the problem page on unreliable JSON output.

Task: {task}

Produce two blocks:

BLOCK 1 - ANALYSIS:
Work through the task in detail. Think out loud, give arguments, don't cut corners.

BLOCK 2 - RESULT IN FORMAT:
Take the conclusions from block 1 and format them strictly as {format}:
{format_schema}

You are a {expert role in your field}. Extract information from the text strictly according to the following structure (schema).

SCHEMA:
{field_name_1}: {data type, e.g. string/date/number}
  - description: {what exactly it means}
  - allowed values: {if it's a restricted list, enumerate them}

{field_name_2}: [list, number of items not known in advance]
  - {subfield_1}: {description}
  - {subfield_2}: [nested list, if you need to go deeper]

Return the result strictly as JSON following this schema, with no explanations or comments.

TEXT:
{source_text}

Extract {data_description} from the text according to the optimized schema.

Text:
{input_text}

Optimized schema:
{json_schema_with_descriptions_and_patterns}

Perform the extraction with checks:
1. Are all required fields filled?
2. Do the extracted values appear in the source text?
3. Do the values match the schema's patterns and rules?

If a check fails, state the specific error and fix it.