TL;DR
PARSE is a two-component system for reliably extracting structured data from text. ARCHITECT optimizes the JSON schema once, specifically for how LLMs read it — adding detailed field descriptions and validation patterns, and restructuring it for clarity. SCOPE does the actual extraction with a multi-stage check: are all fields filled, do the extracted values appear in the source text, are the schema rules respected. RELAY generates Python code to transform the output back to the original schema format — so the system stays compatible with existing processes.
Models are bad at extracting data against schemas written for programmers. A field like "price": {"type": "string"} looks obvious to a human, but an LLM gets confused: should it include the currency symbol, separate thousands with commas, include cents? With HTML markup, the model may extract "2010 Subaru Legacy 2.5 i 4dr Sedan" instead of "2010 Subaru Legacy" — the schema never said what exactly counts as the model. Even with constrained decoding and self-correction, models hallucinate or skip fields when the schema is ambiguous to a machine.
ARCHITECT turns the schema into an instruction for the LLM. It generates synthetic test examples, runs extraction, analyzes the errors and iteratively improves the schema: it adds regular expressions ("pattern": "^(19[5-9][0-9]|20[0-2][0-9]) [A-Za-z0-9 -]+$"), sharpens descriptions ("full model name including year, make and model"), and flattens nested structures for clarity. After 5-6 iterations the schema becomes an unambiguous contract between the task and the LLM. SCOPE uses the optimized schema and checks at every step: are the required fields filled, do the values appear in the text, do they match the patterns. If a check fails, it generates a structured reflection pointing to the specific error, and the model fixes it and tries again.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
PREPARATION PHASE (once):
STEP 1: ARCHITECT receives the original JSON schema
STEP 2: Generates synthetic test examples based on the schema and a seed dataset
STEP 3: Runs extraction with the current schema → analyzes the errors
STEP 4: Improves the schema (adds descriptions, patterns, restructures) → iterates 5-6 times
STEP 5: RELAY generates Python code to transform output back to the original schema format
APPLICATION PHASE (every time):
STEP 1: SCOPE receives the text + the optimized schema
STEP 2: The model extracts the data → constrained decoding guarantees syntactically valid JSON
STEP 3: Static checks:
- Are all required fields filled?
- Do the extracted values appear in the source text? (grounding)
- Do the values match the schema's patterns and rules?
STEP 4: If a check fails → generates a reflection describing the error → the model fixes it
STEP 5: RELAY transforms the result back to the original schema format (if needed)Example in Action
Task: Extract the specs of a used car from a Craigslist listing to auto-populate a catalog
Prompt:
Extract the car's specs from the HTML listing according to the optimized schema.
Listing HTML:
<title>2015 Toyota Camry 2.5 V6 Luxury Edition black</title>
<div class="specs">Mileage: 54,000 mi | Engine: 2.5L V6 gas | Fuel economy: 22 mpg city</div>
<div class="price">$18,900</div>
Optimized schema:
{
"model": {
"type": "string",
"description": "Full model name including year, make and model name WITHOUT trim level or color",
"pattern": "^(19[5-9][0-9]|20[0-2][0-9]) [A-Za-z]+ [A-Za-z0-9 ]+$",
"minLength": 10
},
"price": {
"type": "string",
"description": "Currency symbol + numeric value with thousands separators",
"pattern": "^[$₽€£¥] ?[0-9]{1,3}([ ,][0-9]{3})*$",
"minLength": 2
},
"engine": {
"type": "string",
"description": "Engine configuration: displacement + type (V6/inline/turbo) + fuel",
"pattern": "^[0-9]\\.[0-9]L? (V[0-9]|inline|turbo) (gas|diesel|hybrid|electric)",
"minLength": 10
},
"fuel_economy": {
"type": "string",
"description": "Fuel economy with unit of measure and driving conditions",
"pattern": "^[0-9]+\\.?[0-9]* mpg (city|highway|combined)",
"minLength": 5
}
}
Perform the extraction with checks:
1. Are all required fields filled?
2. Do the extracted values appear in the HTML?
3. Do the values match the patterns?
If a check fails, state what's wrong and fix it.Result:
The model will run the extraction and check itself right away. The first attempt might return "model": "2015 Toyota Camry 2.5 V6 Luxury Edition black" (too much detail). Grounding verification confirms the value is in the text, but rule compliance spots the pattern violation — the trim level and color are included. The reflection will say: "The model field includes the trim 'Luxury Edition' and the color 'black', which violates the pattern. The description requires only the year, make and model name." The model corrects it to "model": "2015 Toyota Camry".
The final JSON will be accurate:
{
"model": "2015 Toyota Camry",
"price": "$18,900",
"engine": "2.5L V6 gas",
"fuel_economy": "22 mpg city"
}
```
If the downstream system works with the old schema, RELAY automatically converts the output back.
---
## Why It Works
**An LLM can't guess what a schema expects**. When the schema says `"type": "string"` for a price — the model picks between `"18900"`, `"$18,900"`, `"18,900 USD"` **at random**, based on frequency in its training data. For the "model" field the model may or may not include the year, trim, color — the schema didn't define the **boundaries of the concept**. Constrained decoding forces valid JSON, but doesn't explain **which** JSON is correct. Unstructured reflection degenerates into "try again" — and the model repeats the same mistake.
**LLMs are great at understanding explicit instructions and patterns**. The model was trained on millions of examples where a description + a regex pattern + examples form an **unambiguous contract**. `"pattern": "^[$€] ?[0-9]{1,3}([ ,][0-9]{3})*$"` leaves no room for interpretation. The description "Full model name including year, make and model name WITHOUT trim level or color" turns an abstract "model" into a **concrete instruction**. The model sees: "I need to take year + make + name, but stop before 'V6' and 'Luxury'".
**ARCHITECT automates turning a human schema into an LLM contract**. The system generates synthetic test examples, runs extraction, sees that the model confuses the trim with the model — and adds a clarification to the description and regex. It sees prices extracted in different formats — and adds a strict pattern. After 5-6 iterations the schema becomes **a set of clear rules** with no ambiguity. **SCOPE turns checking into feedback**. Three checks (all fields filled? values present in the text? matching the patterns?) catch 92% of errors on the very first iteration. When a check fails, the model doesn't just get "error", it gets *"Field X violates pattern Y because Z"* — a **concrete instruction for the fix**, not an abstract "try again".
**Control levers:**
- **Number of ARCHITECT iterations (5-6 by default)** → cut it to 3 for simple schemas (saves prep time), raise it to 10 for complex nested structures
- **Accuracy threshold for stopping optimization** → raise it from 80% to 95% if reliability is critical, lower it to 70% if you need speed
- **Seed dataset for synthetic examples** → add more real examples if the synthetic data doesn't cover edge cases
- **Strictness of pattern validation** → loosen the regex if the schema is too rigid and rejects valid data, tighten it if junk gets through
- **SCOPE reflection mode** → drop the grounding check if you work with paraphrased content (not exact quotes), add custom validators if you have domain rules
---
## Prompt Template
```
Extract {data_description} from the text according to the optimized schema.
Text:
{input_text}
Optimized schema:
{json_schema_with_descriptions_and_patterns}
Perform the extraction with checks:
1. Are all required fields filled?
2. Do the extracted values appear in the source text?
3. Do the values match the schema's patterns and rules?
If a check fails, state the specific error and fix it.
```
**What to fill in:**
- `{data_description}` — what you're extracting (product specs, order details, meeting details)
- `{input_text}` — the raw unstructured text (HTML, email, a customer message)
- `{json_schema_with_descriptions_and_patterns}` — the schema after ARCHITECT optimization, with detailed descriptions, pattern rules, minLength
**Key elements of an optimized schema:**
- `description` — a detailed description of what goes in the field, what does NOT, examples
- `pattern` — a regex for the exact format (especially important for dates, prices, ID numbers)
- `minLength/maxLength` — length limits where applicable
- `enum` — the list of allowed values if the field is categorical
- `required` — an array of required fields at the object level
**Structure of the three-stage check:**
1. **Completeness check** — are all fields from `required` present?
2. **Grounding check** — can each field's value be found in the source text?
3. **Compliance check** — do the values satisfy `pattern`, `enum`, length constraints?
🚀 **Quick start** — paste into the chat:
```
Here's the PARSE template for extracting structured data. Adapt it to my task: {your extraction task}.
Ask me questions to optimize the JSON schema.
[paste the template above]The LLM will ask:
- Which fields need to be extracted?
- Which of them are required?
- Do you have sample input text?
- What data formats are expected (dates, prices, IDs)?
These questions are needed to build an optimized schema: add detailed descriptions for every field, write regex patterns for validation, and define the required fields. The model will take the PARSE pattern (optimized schema + three-stage check) and adapt it to your specific extraction task.
Limitations
⚠️ Schema prep takes time: ARCHITECT runs 5-6 optimization iterations (generate synthetic examples → extract → analyze errors → improve the schema). It's a one-time process, but for complex nested schemas it can take 10-20 minutes. Not a fit for ad-hoc tasks where you need the schema "right now."
⚠️ Depends on the seed dataset: The quality of schema optimization depends heavily on having real examples. For brand-new domains (with not a single example of correct extraction), ARCHITECT may miss edge cases. You need at least 20-30 real examples for solid optimization.
⚠️ Latency overhead from complex checks: SCOPE adds an average of +10 seconds per extraction because of reflection iterations (model → check → fix → check). Using ARCHITECT-optimized schemas cuts that to +6 seconds (fewer errors = fewer iterations), but for real-time scenarios that can still be a dealbreaker.
⚠️ Over-optimizing the schema: If you run ARCHITECT for >10 iterations, the schema can overfit to the synthetic data and perform worse on real data. The study shows accuracy dropping after the 6th iteration — you have to stop in time.
⚠️ Not for dynamic schemas: PARSE assumes a relatively stable schema structure. If your data requirements change every week (new fields get added, formats change), you'll have to rerun ARCHITECT constantly. The system is optimized for "debug it once, and it runs for a long time."
How They Tested It
The team took three different datasets to test how general the approach is: the internal Retail-Conv with real business conversations (6 schemas × 40 examples = 240 tasks), the public Schema-Guided Dialogue with 20,000 annotated conversations across 20 domains, and SWDE — a classic benchmark for extraction from HTML (1,600 examples across 8 verticals). The baseline was an agent with constrained decoding and the original user-provided schemas. They tested on five different models (Claude 3.5/3.7 Sonnet, Claude 3.5 Haiku, Llama 4-Maverick, DeepSeek-R1) to make sure the gains weren't specific to one architecture.
The most dramatic result came on the SWDE dataset. Baseline models scored 24.5% accuracy (correctly extracting structured data in only 49 of 200 examples!). Adding SCOPE raised that to 89.2%, and ARCHITECT + SCOPE together delivered 93.14% — a 3.8x jump. Why such an explosive effect on SWDE specifically? HTML markup creates visual noise for an LLM — the model sees tags, attributes and nested structures and gets confused about what counts as "the car model" versus "an engine spec." The original schemas were too abstract for that context ("model": {"type": "string"} with no clarification), and ARCHITECT tuned them for HTML: it added the pattern "^(19[5-9][0-9]|20[0-2][0-9]) [A-Za-z0-9 -]+$" (year + make + name WITHOUT extra details) and the description "Full model name including year, make and model" — which turned ambiguity into an instruction.
On cleaner data (Retail-Conv, SGD), baseline accuracy was higher (~75-92%), but the gains were still significant: +15-18 absolute points on Retail-Conv, +2-4% on SGD. An interesting finding: ARCHITECT schemas optimized on one model (Claude) worked on other models (Llama, DeepSeek) almost as well (89.1% vs 93.7% on the same model). That suggests the system optimizes universal properties of schemas (clarity, lack of ambiguity) rather than fitting the quirks of a particular architecture. The reflection mechanism in SCOPE cut errors by 92% on the very first fix iteration — not the second, not the third, but immediately. That means structured feedback ("field X violates pattern Y") is far more effective than an abstract "try again."
The researchers uncovered the optimization patterns: in 55% of cases ARCHITECT restructured schemas (flattening nested objects for clarity), in 34% it improved descriptions (adding context and concept boundaries), in 3% it added pattern rules (regexes for strict formats), and in only 0.08% did it touch the basic validation constraints. This shows that the main problem with schemas isn't the technical constraints but their lack of clarity for an LLM — the model can output valid JSON but doesn't understand WHICH data counts as correct.
Resources
PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction — Anubhav Shrimal, Aryan Jain, Soumyajit Chowdhury, Promod Yenigalla, RBS Tech Sciences, Amazon. The study isn't openly published (it's an internal Amazon development), but the methodology is described in the full technical report.
Key links to the datasets and methods mentioned:
- Schema-Guided Dialogue (SGD): https://github.com/google-research-datasets/dstc8-schema-guided-dialogue
- SWDE (Structured Web Data Extraction): https://swde.codeplex.com/
- ReAct framework: Yao et al. 2023, "ReAct: Synergizing Reasoning and Acting in Language Models"
- Chain-of-Thought: Wei et al. 2022 (the foundational step-by-step reasoning technique)