Key Aspects of the Research:
The study shows that LLMs, especially less powerful ones, often make mistakes when asked to reason in natural language and generate structured data (for example, JSON) at the same time. The authors propose splitting this process into two sequential stages: first the model generates only its reasoning (its "thoughts"), and then, in a second step, it builds the structured output based on those thoughts.
Key Result: Splitting one complex step into two simpler ones (first "think," then "structure") can raise the task success rate by up to 4x by sharply reducing formatting errors and "hallucinations."
The Core of the Method, Explained:
The method boils down to decomposing a complex task that demands two different kinds of thinking from the LLM at once: creative-analytical (reasoning, drawing conclusions) and formal-logical (generating code or JSON that follows strict syntax).
Imagine asking someone to write an essay and fill out their tax return at the same time. Most likely they'll keep switching between the two and end up making mistakes in both the prose and the numbers. An LLM runs into a similar problem. When you ask it in a single response to "analyze the text and output the result as JSON," it tries to do two things at once, which often leads to "broken" JSON or an incomplete analysis.
The method described in the study offers a more effective approach that can be summed up like this: "Finish one task completely first, then move on to the second."
In practice, this is done through two-stage prompting: 1. Stage 1: "Thinking" (Think1). You give the model the source data and ask it to do a full analysis, draw conclusions, make a plan, and lay all of that out as plain text. At this stage you don't require any formatting. 2. Stage 2: "Acting/Structuring" (Think2). You take the text output from the first stage and, in a new request, ask the model: "Based on this analysis of yours, extract the data and present it as JSON/a table/a list."
This approach lowers the model's "cognitive load," letting it focus on one thing at each step, which dramatically improves the reliability and accuracy of the final result.
Practical Applicability Analysis:
Direct applicability: Very high. Anyone can implement this two-stage approach by hand in any chatbot (ChatGPT, Claude, etc.). Instead of one complex prompt, you send two simpler ones in sequence, using the answer to the first as context for the second. Or, even better, you can make the model follow this pattern within a single prompt by explicitly splitting the instructions into "Step 1" and "Step 2."
Conceptual value: Huge. Users start to understand that an LLM isn't a "black box" but a system with specific limitations. The key conceptual idea: LLMs handle a sequence of simple tasks better than one complex task that requires processing different kinds of information in parallel. Knowing this helps you avoid common mistakes and design complex requests more effectively.
Adaptation potential: The adaptation mechanism is simple and universal. Any prompt of the form "Analyze X and output the result in format Y" can, and should, be rephrased as: "You are an expert. First, do a detailed analysis of X and describe your conclusions in text. Then, based on your own analysis, produce the result in format Y." This template works for text analysis, planning, data extraction, and many other tasks.
Practical Example:
Let's say you're a marketer and want to analyze a customer review to get both a quick summary and structured data for a report.
You are an experienced marketing analyst. Your task is to analyze a customer review and extract the key information from it. Follow the steps strictly.
**Context: Customer review**
> "Overall, your new grocery delivery service isn't bad. The courier showed up on time, and the packaging was top-notch. But I was really disappointed that the milk was missing from my order and the avocados were completely unripe. I had to call support, where I waited 15 minutes for an agent to pick up. She did help in the end and processed a refund, but it left a bad taste."
---
**Your task:**
**Step 1: Analytical summary**
First, write a short but substantive summary of the review. Describe the customer's overall impression and highlight the main strengths and weaknesses of the service mentioned in the text. Don't use lists or special formatting; just lay out your thoughts in 2-3 sentences.
**Step 2: Extract the data into JSON**
After you write the summary, based on **your own analysis and the review text**, create a JSON object with the following structure. Don't add any comments before or after the JSON object.
{
"sentiment": "",
"positive_points": [""],
"negative_points": [""],
"mentions_support": "",
"support_wait_time_minutes": ""
}
Why It Works:
This prompt works by explicitly separating the cognitive tasks for the LLM, a direct implementation of the study's findings:
- A clear sequence: The instructions "Follow the steps strictly," "Step 1," and "Step 2" force the model to perform the operations one after another rather than simultaneously.
- Lighter load in Step 1: In the first stage (
Analytical summary), the model only has to do one thing: analyze the text and state its conclusions in natural language. It doesn't have to think about JSON syntax, quotes, and brackets. That lets it build a solid text foundation. - Focus in Step 2: In the second stage (
Extract the data into JSON), the model gets a clear instruction: "based on your own analysis... create a JSON." Now its job isn't analysis but structuring information it has already made sense of. That's a simple fill-in-the-template task LLMs handle much better, which minimizes the risk of syntax errors in the JSON.
Another Practical Example
Let's say a user wants to plan a short trip and get both an inspiring description and a clear itinerary.
You are a creative travel agent. Your job is to help me plan a 3-day trip to Istanbul for a couple who love history, great food, and leisurely walks.
Work in two stages.
**Stage 1: Inspiring description**
First, write a vivid, captivating paragraph (5-6 sentences) that conveys the atmosphere of the proposed trip. Describe the emotions and experiences awaiting the travelers and build a sense of anticipation. Don't list specific places; focus on the feelings.
**Stage 2: Structured plan**
After the paragraph, building on the ideas you put into it, create a detailed trip plan as a markdown table. The table should have the columns: "Day," "Morning (history and culture)," "Afternoon (walks and food)," "Evening (atmosphere)."
**Important:** The inspiring paragraph must come first, followed immediately by the itinerary table. No other text.
Why This Example Works: The Mechanism.
This prompt is effective because it separates two fundamentally different tasks: a creative one and an organizational one.
Splitting by type of thinking:
- Stage 1 (
Inspiring description) taps the LLM's "creative" potential. The model focuses on generating vivid, emotional text without getting distracted by logistics and structure. - Stage 2 (
Structured plan) taps its "logical" potential. The model switches into planner mode and organizes the information. It uses the creative text it already generated as a source of ideas, which keeps the two parts of the answer coherent.
- Stage 1 (
Preventing format conflicts: Without this separation, the model might try to drop table elements right into the narrative or, conversely, make the description in the table too dry and formal. A clean split into "paragraph" and "table" ensures each element comes out in the format best suited to it, which matches the study's main finding that splitting tasks improves reliability.
Usefulness Score: 85
Main Evaluation Criteria
- A. Relevance to prompting techniques: Yes, the study reveals a specific pattern (separating thinking from data generation) that can be applied to structure complex requests.
- B. Better conversational answers: Yes, the case study demonstrates a 4x performance improvement by reducing tool-calling errors, which directly affects answer quality and reliability.
- C. Direct practical applicability: High, but it needs adapting. Users can't use the PDL language itself, but they can apply the key "separation of tasks" principle in their prompts in any chat interface.
- D. Conceptual value: Very high. The study gives a fundamental understanding of an LLM limitation: models (especially small ones) struggle to generate natural-language reasoning and structured data (JSON) at the same time. This explains frequent errors and "broken" output formats.
- E. A new useful practice (clusters): The work falls into several clusters at once:
- #1 (Phrasing techniques): Offers a form of task decomposition (splitting the ReAct loop).
- #2 (Behavioral patterns): Exposes LLMs' weakness at generating mixed content (text + JSON).
- #5 (Extraction and structuring): Directly solves the problem of reliably extracting structured data.
- #7 (Reliability and stability): The method's main goal is to increase reliability and reduce failures.
- Practicality checklist (+15 points): Yes, the work shows how to structure complex requests, reveals non-obvious LLM behavior, and offers a way to improve answer accuracy.
Numerical Usefulness Score
Arguments for a score of 85: The study offers not just "yet another trick" but a fundamental principle for working with LLMs that applies across dozens of scenarios. The main takeaway, "split the cognitive load," is extremely valuable for any user who wants more than plain text from a model: a structured result (JSON, a table, a list). The demonstrated 4x gain in effectiveness clearly proves the power of this approach. This knowledge transfers directly into prompt-writing practice without any tools.
Counterarguments (why the score could be lower): * High barrier to entry: The paper is about building a specialized language (PDL) and agent frameworks (CrewAI), which may put off everyday users. The focus is on development, not simple chat. * Indirect application: Users can't just pick up and use PDL. They have to "translate" the paper's idea into a regular prompt format themselves, which takes some thought.
Counterarguments (why the score could be higher): * A universal principle: The principle of "separating thinking from data generation" is one of the most powerful and universal in advanced prompt engineering. Mastering it fundamentally changes how you tackle complex tasks with LLMs. It may be one of the most practically significant concepts for users moving from simple requests to complex ones.