TL;DR
The method breaks one complex task into two separate steps: first the model simply pulls facts out of the text in structured form (JSON), with no conclusions of any kind. Then those facts are run through clear rules — separately, without the source text — to reach the final decision.
The headline finding: even if you give the model all the classification rules right in the prompt along with the text, it still gets confused — accuracy swings from 65% to 96% depending on the model. The reason is simple: the model has to read raw text and keep a multi-step decision tree in its head at the same time. Those are two different loads, and together they get in each other's way — the model either drops a condition or invents something that wasn't in the text.
Splitting the task into two steps almost entirely eliminates these errors: accuracy rises to 95–99% across all tested models. In the first step the model does a narrow job — "find feature X in the text" — and it handles that superbly. In the second step the rules are applied to ready-made, clean data — without the noise of the source text.
Method Blueprint
STEP 1: The LLM reads the source text → extracts features into JSON
(facts only, no category conclusion)
STEP 2: Rules (hard "if-then" logic) are applied to the JSON
→ final category + rationale
In the original study, step 2 was done by program code (a deterministic "rules engine"). In a chat you can emulate it with a second, separate prompt where the model applies the rules only to the structured data — with no access to the source text.
Example in Action
Task: At a customer support operation (think Amazon or Chase), incoming tickets need to be classified by priority P0–P3 according to an internal SLA policy — team leads do this by hand, and their decisions differ from person to person.
Prompt (Step 1 — fact extraction):
You are a support analyst. Read the ticket text below and extract only
the listed features in JSON format. Do not draw conclusions about priority,
only extract facts.
Features to extract:
- affected_users: how many users are affected (a number or "all"/"one")
- feature_down: which feature is not working (text)
- workaround_exists: whether there is a workaround (yes/no)
- financial_impact: whether financial losses are mentioned (yes/no, amount if given)
- outage_duration: how long the problem has lasted
Ticket text:
{ticket_text}
Output only JSON. If a feature is not mentioned, put null.
Prompt (Step 2 — applying the rules):
You have structured data and ticket prioritization rules.
Apply the rules in order and determine the final priority.
Rules:
P0: the service is down for all users, no workaround
P1: a critical feature is not working for some users, financial losses
P2: a workaround exists, but users are inconvenienced
P3: a minor issue, no business impact
Data:
{json_from_step_1}
Output: the final priority and which rule assigned it.
Result: Step one returns clean JSON with facts — no interpretations. Step two returns the ticket's priority with an explicit reference to the rule that fired, which makes it easy for a team lead to audit the decisions.
Why It Works
A model struggles when it has to simultaneously read unstructured text and keep a complex decision tree in its head — it drops conditions along the way, confuses similar categories, and alternately under- and over-rates.
But with a narrow, simple task — "find a specific fact in the text: yes/no/number" — the model performs consistently and almost without errors.
The method exploits this difference: it breaks a complex task into a series of simple ones. Extraction becomes mechanical ("is ascites mentioned in the text?"), and rule application is done either by code or by a separate prompt where the model isn't distracted by the source text and sees only clean data.
🔧 Control knob: the list of features to extract in step 1 is what determines the quality of the whole result. The more concrete and narrow each feature is (not "assess severity" but "financial losses mentioned: yes/no"), the more accurate the extraction.
🔧 Another knob: you can ask the model in step 2 to show which rule fired — that gives you transparency and the ability to check the logic by hand, instead of a black box that "just gives an answer."
Prompt Template
STEP 1 — Extraction:
You are an analyst who extracts facts from text. Read {text}
and extract only the listed features into JSON. Do not draw conclusions
and do not classify.
Features:
{feature_list_with_descriptions}
Text:
{text}
Output only JSON. If a feature is not found, null.
---
STEP 2 — Applying the rules:
You have structured data and classification rules.
Apply the rules strictly in order and find the first one that fires.
Rules:
{rules_in_if_then_format}
Data:
{json_from_step_1}
Output: the final category and the rule that assigned it.
Fill in: {text} — the source document (report, ticket, résumé, contract), {feature_list} — the specific facts to pull out, {rules} — the policy or classification checklist.
🚀 Quick start — paste into the chat:
Here is a template that splits "fact extraction + rule application".
Adapt it to my task: {your classification task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask which features need to be extracted and what classification rules you have — because the method doesn't work without them: you need a clear line between "facts from the text" and "decision logic."
Limitations
⚠️ The method doesn't work without an explicit rulebook. If the classification rules are themselves vague or subjective (e.g., "rate the creativity of the text"), splitting into two steps won't help — there are no clear "if-thens."
⚠️ Extraction accuracy drops on rare and complex features. The model finds simple facts (cyst present yes/no) almost flawlessly, but sometimes misses or confuses subtle nuances (e.g., "ascites" mentioned only implicitly).
⚠️ The method requires prep work. You have to write out the full list of features and formalize the rules in advance — it doesn't work "off the cuff" on the very first request; the template needs tuning on your own examples.
⚠️ Tested on one document type and language. The study used Chinese medical ultrasound reports from a single institution. On texts of a different style and format the effect may be weaker, especially if the source data is unstructured in a very different way.
How They Studied It
The researchers took 310 patients with 390 ovarian tumors and free-text ultrasound reports in Chinese. Eight different LLMs (GPT, Claude, Gemini, Grok, DeepSeek, Qwen) were given the same task — classify risk under the O-RADS system — in three ways: with no explicit rules at all, with the full text of the guidelines in the prompt, and with the "facts first, then rules" split. The reference standard was the assessment of radiologists with 10–30 years of experience.
The most interesting part: even when the models were given the entire official guideline right in the prompt text, accuracy still swung from 82% to 96% depending on the model — simply "providing the rules" doesn't solve the problem. With the task split into two steps, accuracy rose to 95.6–99.2% for all models, regardless of how "strong" the base model was. That's the key insight: prompt architecture matters more than the raw power of the model.
Also telling: the doctors' original reports tended to overstate risk (a known problem called overstaging in radiology), and the hybrid method eliminated this systematic error.
Resources
Tan X., Qiu C., Liu X. et al. Improving O-RADS Risk Stratification from Ultrasound Reports: A Comparative Evaluation of Hybrid versus End-to-End LLM Reasoning Strategies. Shenzhen University, Qilu Hospital of Shandong University. Code and pipeline: github.com/Xiaotong66/ORADS-LLM-Pipeline