Processing JSON with LLMs: When Code Beats Words

The study compared two main approaches:

APPROACH 1: Answer generation

Query → LLM reads the JSON → Outputs a text answer

APPROACH 2: Code generation ← works better

Query → LLM writes a Python function → Code runs → Result is returned

Ways to improve it:

• Add the JSON schema to the prompt
• Provide a trimmed version of the response if the full one doesn't fit
• For complex tasks: add reasoning instructions (CoT)

Every approach works in a single call to the model.

Task: You pulled Chicago apartment listings from a real-estate listings API as JSON (50+ objects, nested structure: neighborhood → building → unit → features). You need every two-bedroom under $750,000 with a balcony.

Prompt (basic approach — works poorly):

Here's a JSON with apartments. Find every two-bedroom under $750,000 with a balcony and list the addresses.

[JSON]

Prompt (improved — code generation):

Here is a JSON response from a real-estate listings API with apartments, plus the schema of that JSON.

Write a Python function that:
1. Takes the entire JSON as a dictionary
2. Filters apartments: rooms == 2, price <= 750000, has_balcony == true
3. Returns the list of addresses as a comma-separated string

Code requirements:
- Starts with "def "
- Checks that keys exist before accessing them
- Compares strings in lowercase
- Returns only the result (a string), with no extra text

JSON schema:
{
  "apartments": [
    {
      "address": "string",
      "price": "number",
      "rooms": "number",
      "has_balcony": "boolean",
      ...
    }
  ]
}

JSON:
[paste the data]

Python function:

Result: The model produces Python code with a def filter_apartments(data): function that handles the nesting correctly, checks the conditions and returns the list of addresses. You can run the code right away (in local Python or via the chat's code execution tool). Accuracy is 3-50% higher depending on the task's complexity.

The LLM's weakness: Models get lost in the structure when processing JSON directly. They confuse similar keys (name, room_name, name_without_policy), pull values from the wrong objects, and stumble over nesting. With long responses (50K+ characters) accuracy collapses. It's like eyeballing your way through a 100-sheet Excel workbook to find one row.

The LLM's strength: Models know code structure extremely well and can write precise instructions. Python for parsing JSON is home turf for them. Code is unambiguous: if item['price'] <= 750000 can't be read two ways. Plus code is more compact — a 20-line function instead of a sprawling 200-word text answer.

How the method exploits this: You move the task out of "understanding the structure" and into "writing code". Instead of "find it and tell me" → "write a function that finds it." The model generates the code, the code runs, and the result is exact. The JSON schema works as a map of the terrain — the model sees data types, nesting and relationships. A structural sample (even a trimmed one) shows real values, so the model understands the format ("USD 250" vs 250).

Control knobs:

• JSON length → if it's >50K characters, use a trimmed version (one example of each object type)
• Task type → for extracting a single value ("What is form type?") a direct answer is sometimes better; for filtering/aggregation, always use code
• Schema → always add it if you have one — it gives +12% accuracy
• CoT instructions → for aggregation (sums, averages) add "Think step-by-step" before code generation

Here is a JSON response from {source} and its schema.

Write a Python function that:
1. Takes the entire JSON as a dictionary (parameter data)
2. {task description: filters/extracts/counts}
3. Returns the result as a string

Code requirements:
- Starts with "def "
- Checks that keys exist and values aren't None
- Compares strings in lowercase
- Cleans numeric strings (strips "$", spaces) before converting
- Returns only the result, with no extra text

JSON schema:
{schema}

JSON:
{data}

Python function:

Placeholder guide:

• {source} — where the data comes from (a listings API, a CRM export, etc.)
• {task description} — the specific operation ("find all X where Y", "compute the average Z")
• {schema} — the JSON schema of the structure (generate one online or ask the model)
• {data} — the JSON response itself (full or trimmed)

🚀 Quick start — paste into the chat:

Here is a template for processing JSON through code. Adapt it to my task: [your task].
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask where the JSON comes from, what you need to extract and whether there's a schema — because that information is critical for accurate parsing. It will take the function pattern from the template and adapt it to your data structure.

⚠️ Task type: For simple extraction of a single value ("What is the price?") a direct answer is sometimes more accurate than code — 10 out of 15 models showed this. Code pays off for filtering and aggregation.

⚠️ Context size: With JSON >80K tokens, accuracy drops for every model. Even with code. Use trimming (one example of each object type).

⚠️ Code syntax: ~5-10% of errors come from incompatible syntax (wrong data type, traversal bugs). Top models (GPT-4o, Claude-4-Sonnet) produce runnable code 95%+ of the time.

⚠️ Semantic ambiguity: If the JSON has similar keys (name, room_name, full_name), the model may pick the wrong one. Specify the exact path in the task or clean up the JSON before feeding it in.

The team collected 1,298 real question-answer pairs over API responses from RapidAPI (hotels, cars, flights, SEC filings, products). The JSON was long — from 24K to 74K characters. Questions were split into three types: extractive (find a value by key), filtering (return all records matching a condition), aggregation (compute a sum/average/minimum).

They tested 15 models (from Llama-3.1-8B to GPT-4o) with different prompts: answer generation vs. code, with and without a schema, with and without reasoning (CoT). Accuracy was measured with three metrics: exact match, contains (the result appears in the answer), and LLM as a judge (semantic match).

The results were surprising: Even GPT-4o reached only 77% with the best approach. Code beat direct answers in 13 of 15 models. The schema added +12% for code and +7% for direct answers. When JSON grew from 10K to 80K tokens, accuracy dropped by anywhere from 7% (GPT-4o) to 91% (Mistral-large). Recency bias (the answer's position in a list) had a 5-75% effect depending on the model.

The critical insight: When they stripped the JSON down to the bare minimum (oracle projection — keeping only the needed fields), accuracy jumped by 8-38 points. That means noise and structural complexity are accuracy's main enemy. Models can cope if you give them a clean, simplified version.

Context: The researchers used this prompt for code generation with the full schema and response.

You will be given a JSON object as data which is a response from a REST API containing information returned from the API call.
You will be given a JSON schema of the response from the REST API returned from the API call.
Your task is to extract and return information from the JSON object which follows the JSON schema and answers the user query: <<task_prefix>>

You need to write a Python function that:
- Starts the function with "def".
- Takes only the entire api response as input and doesn't have any other input.
- Identifies the structure of the input data, ensuring it checks for relevant keys and data types.
- When comparing strings, it should always convert both sides of the comparison to lowercase.
- Processes the provided data.
- Iterates through the data to extract relevant information.
- Cleans numeric strings by removing non-numeric characters before converting them to integers.
- Performs proper checks to ensure a key exists and is not None before querying its value.
- Returns only the requested data as a string and no other extra information or words.
- Do not add any extra keys or terms to the output.

Final Check:
- The function must be formatted in Python markdown for direct execution.
- No explanations, comments, or additional text should be included.
- Do not include any example usage data.

The JSON schema of the object given as data is as follows: <<json_schema>>

data = <<json_obj>>

Python Function:

💡 Adaptation for tables in PDFs:

The same logic works for pulling data out of tables. Instead of JSON — an HTML or Markdown table extracted from a PDF.

Here is a table from a PDF report in Markdown format.

Write a Python function that:
1. Parses the table (you can use pandas)
2. Filters rows where {condition}
3. Returns the result as a list

Table:
{markdown_table}

Python function:

💡 Adaptation for nested dictionaries:

If the structure is really complex (the API returns 5 levels of nesting), ask the model to simplify it first, then process it:

Here is a complex JSON with 5 levels of nesting.

Step 1: Write a flatten_json() function that simplifies the structure down to 2 levels
Step 2: Write an extract_data() function that finds {what you need} in the simplified JSON

JSON:
{data}

🔧 Technique: Add debug output → see the logic

Add this to the prompt template:

- Add print() at every processing step for debugging

You'll get code with print(f"Found {len(items)} items") — so you can see how the model is reasoning. Handy for checking the logic.

🔧 Technique: Swap generic variable names for descriptive ones → fewer errors

Instead of:

for item in data:
    if item['key'] == value:

Ask for:

- Use descriptive variable names (apartment instead of item, price instead of value)

The model generates more readable code and gets confused less.

How Good Are LLMs at Processing Tool Outputs? https://github.com/LongFuncEval/toolJSONprocessing

Kiran Kate, Yara Rizk, Poulami Ghosh, Ashu Gulati, Tathagata Chakraborti, Zidane Wright, Mayank Agarwal — IBM Research, USA; Persistent Systems; Grab