How to get reliable JSON and structured output from an LLM

What unreliable structured output looks like

Structured output fails in two ways, and the second is easier to miss.

• The structure itself breaks. Unclosed brackets, stray text around the JSON, missing required fields, the same field filled in different formats from call to call, or a closing JSON block that never appears. A study of terminal format requirements saw up to 50% violations of "end with a JSON block"-type instructions on hard tasks.
• The JSON is valid but the content got worse. Asking for a format in the same request as the reasoning lowers answer quality. The Format Tax study found structured answers were on average 162 tokens shorter than free-form ones, with the lost reasoning showing up as errors.
• The model reads JSON badly, too. When the input is a large API response, a JSON-processing benchmark found GPT-4o topped out at 77% accuracy reading it directly.

Why format instructions fail

Three mechanisms recur across the papers.

Format competes with thinking. The model generates token by token and keeps the format requirement active from the first token. The Format Tax authors separated the prompt from the decoding mechanism and found the instruction itself caused most of the damage: 92% of the significant degradation came from the prompt. A compliance-agent case study describes the same conflict in agent loops, worst on weaker models.

Schemas written for programmers are ambiguous to models. A field typed only as "string" leaves the model to guess whether a price includes the currency symbol or whether a car model includes the trim. Constrained decoding guarantees valid syntax, not the right values, as the PARSE paper shows.

Deferred commitments fade. "Finish with JSON" has to be held in mind across the whole answer. In the salience study, o4-mini followed one format requirement 99% of the time and all five only 47.5% of the time, and its math accuracy fell from 93% to 27% with active format requirements.

Fixes ranked by measured effect

Ordered by the size of the measured effect, on the authors' models and data.

1. Write the schema as instructions: descriptions, patterns, allowed values, then a self-check. PARSE adds a description and regex pattern to each field and has the model check completeness, grounding in the source and pattern compliance. On HTML extraction (SWDE), accuracy rose from 24.5% to 93.14%, and structured error feedback fixed 92% of errors on the first iteration. Schema-guided extraction used the same levers for nested lists without examples and got over 90% accuracy on 12 of 14 fields with Claude Opus 3.
2. Ask for code instead of reading JSON. For filtering or aggregating a large response, have the model write a Python function and run it. The JSON-processing study reports up to 95% accuracy this way, and adding the JSON schema to the prompt added 12%.
3. Reason first, format second. Get the analysis in free text, then convert it. Two-stage prompting reports up to 4x improvement in task success in an agent setting; the Format Tax study measured +6.8 pp accuracy on average for two requests. REAP applies the split to recall: draft the facts in slices, format at the end.
4. Let the model choose tools in plain language. Natural Language Tools replaces JSON tool calls with "Tool – YES/NO" lines parsed by code; accuracy went from 69.1% to 87.5%.
5. Frame the format requirement at both ends. An "IMPORTANT FORMAT REQUIREMENT" tag at the start and a reminder at the end raised compliance from 70–87% to 90–100% in the salience study. With several requirements, list them in the reminder.
6. Box the model in. Exact field names, fixed order and explicit bans ("do not add any steps beyond those listed"). In the hyper-specific prompt study, GPT-4o produced an exactly correct YAML config in only 3 cases out of 100 from ordinary prompts. One filled-in example of text-to-JSON also helps, per the schema-plus-examples study.
7. Edit with patches, not full regeneration. For large documents, ask for RFC 6902 add/remove/replace operations and apply them in code, as in JSON Whisperer; replace list indices with keyed entries so the model does not miscount positions.

A combined prompt stack

A pipeline that combines the strongest pieces:

1. Request 1, no format: the task, answered in detail, with no mention of JSON (Format Tax).
2. Request 2, format only: "take the answer above and format it as JSON", with a schema whose fields carry a description, an allowed-value list or pattern, and nesting written out.
3. Self-check inside request 2: are all required fields filled, do the values appear in the source, do they match the patterns; if not, state the error and fix it (PARSE).
4. Bracket the requirement: the format tag first, a specific reminder last (Salience Reminder).
5. Downstream: validate with code, edit with patches, and process large JSON with generated code rather than by reading it.

In a long chat, the split also saves output budget: the output-stalling study estimates JSON markup adds +15% to the volume of an answer.

What does not work

• "Analyze and return JSON" in one request for anything that needs reasoning. This is the pattern every paper above was built to replace.
• Relying on constrained decoding or JSON mode alone. It fixes syntax, not meaning; ambiguous fields still get guessed values (PARSE).
• A generic "follow all requirements" with many constraints. Compliance still falls as requirements stack; name them in the reminder or split them across requests (salience study).
• Pasting huge JSON and asking questions about it. Mistral-large dropped by 91% as inputs grew to 80K tokens in the JSON-processing study.
• Over-tuning the schema. PARSE accuracy dropped after the 6th optimization iteration.

The size of the format tax differs by model. The Format Tax study found Claude Haiku 4.5, newer GPT-4o and Grok barely suffer and thinking modes remove it, while open models are hit hard; the two-stage split matters most for weaker and open models.

Task: {task}

Produce two blocks:

BLOCK 1 — ANALYSIS:
Work through the task in detail. Think out loud, give arguments, don't cut corners.

BLOCK 2 — RESULT IN FORMAT:
Take the conclusions from block 1 and format them strictly as {format}:
{format_schema}

Extract {data_description} from the text according to the optimized schema.

Text:
{input_text}

Optimized schema:
{json_schema_with_descriptions_and_patterns}

Perform the extraction with checks:
1. Are all required fields filled?
2. Do the extracted values appear in the source text?
3. Do the values match the schema's patterns and rules?

If a check fails, state the specific error and fix it.

IMPORTANT FORMAT REQUIREMENT: {format_requirement}

{main_task}

{additional_instructions_if_needed}

Make sure to follow ALL the format requirements stated above.