2 Why This Example Works.
This example runs on the same fundamental principle as the study, adapted for a task that demands accurate calculations, not just accurate facts.
- Grounding in exact data: LLMs often slip up on math and mix up the calorie counts of foods. The prompt makes the model first "look up" and lock in the exact numbers in a table. That creates a foundation of verified information.
- Simulating a "tool" (calculator/database): Step 1 mimics using an external tool, a calorie calculator. The model can't just pull numbers out of thin air.
- Keeping the math separate from the creativity: The prompt cleanly splits the task into "find the numbers" and "build the menu". The instruction "Using the exact data you just found" forces the model to calculate from the locked-in values, which sharply improves the accuracy of the final plan and keeps it from fudging the numbers to fit the desired result. It's the equivalent of the agent in the study using the code interpreter's output to make its decision.
Key Aspects of the Study:
The study looks at whether the quality of LLM answer evaluation (when one model grades another) can be improved by giving the judge external tools: web search for fact-checking and a code interpreter for checking math and programs. The results show this approach significantly increases evaluation accuracy, especially on tasks that require factual accuracy and correctly executed code.
Key Result: External verification through tools is a powerful way to ground an LLM, making it rely on real data instead of its internal "knowledge", which drastically cuts errors and hallucinations.
The Core Idea of the Method:
The heart of the method, which you can adapt for everyday prompting, is forcing the answer-generation process into two stages: gathering/verifying information and synthesizing the answer.
Instead of just asking a question and hoping for the best, you use your prompt to make the LLM simulate the agent approach from the study:
- Assess the task and pick a "tool" (in your head): You recognize that your request requires exact facts (prices, dates, specs, scientific data) or calculations.
- Instruct it to "use the tool" (in the prompt): You explicitly order the model to perform a verification step first. Use phrasing that mimics external tool use:
- "First, run a web search on..."
- "Before answering, find up-to-date data on..."
- "Write and mentally execute Python code to calculate..."
- Instruct it to make the "final decision" (in the prompt): You order the model to build its final answer based solely on the results of the previous step.
- "Based on the facts you found, and only those, answer my question."
- "Use the result of the calculation to form your conclusion."
This technique turns the LLM from a "know-it-all fantasist" into a "diligent research assistant". You're not asking the model to know; you're asking it to find out and report back. That dramatically increases the reliability and trustworthiness of the answer.
Practical Applicability Analysis:
-
Direct applicability: You can start using this method right away by building two-step logic into your prompts. For example, instead of "Which smartphone is better for photos?" you can write: "1. Find comparative camera reviews of the iPhone 15 Pro and Google Pixel 8 Pro from the last 3 months. 2. Based on those reviews, conclude which phone is better for photos and explain why, quoting the key arguments from the reviews."
-
Conceptual value: The main idea the study gets across is don't trust, verify (and make the LLM verify). It gives you a critically important understanding: LLMs are prone to fabrication by nature, and the best way to fight that is to ground their answers in external, verifiable data sources. That shifts your prompting from "asking" to "setting a task with mandatory verification".
-
Adaptation potential: The method is universal. You can adapt it to any task where accuracy matters:
- Travel planning: "First check the current opening hours and ticket prices for the Louvre and the Eiffel Tower, then put together a plan for the day."
- Medical information (with caution!): "Find information about the side effects of [drug] in authoritative medical sources (for example, Mayo Clinic, WebMD), then summarize them."
- Essay writing: "Gather 3-5 key facts about the Industrial Revolution from academic sources, then use them to write the introduction to an essay."
The adaptation mechanism is simple: figure out which information in the request is critical and add a preliminary step to the prompt for finding and verifying it.
1 A Practical Example:
You are an experienced marketing analyst. Your job is to help me choose a CRM system for a small company (10 people) in the online education space.
**Task methodology (follow strictly):**
1. **Step 1: Gather and verify data.**
* Simulate a web search to find 3 popular CRM systems that are a good fit for small businesses and integrate with education platforms (for example, Kajabi, Teachable).
* For each of the three systems, find current information on the following points:
* Approximate plan price for a team of 10 (per month).
* Key features useful for an online school (for example, funnel automation, messaging app integrations, sales analytics).
* One main pro and one main con according to user reviews from the past year.
2. **Step 2: Synthesis and analysis.**
* Present the information gathered in Step 1 as a comparison table.
* Based **only on the data in this table**, write a short conclusion and give your recommendation on which system is the best choice and why.
**Answer format:**
First present the table, then your conclusion and recommendation.
2 Why It Works:
This prompt works because it directly implements the principles from the study, making the LLM act like an agent with external tools:
- Forced verification: The prompt doesn't let the model "make up" an answer right away. It forces it through a mandatory data gathering and verification stage ("simulate a web search"). That grounds the answer in fresh, relevant information instead of stale training data.
- Task decomposition: The complex task "recommend a CRM" is broken into simple, checkable subtasks: find, extract specific facts (price, features), compare, recommend. That lowers the model's cognitive load and reduces the chance of error.
- Control over the source of the answer: The instruction "Based only on the data in this table" forces the LLM to build its final conclusion not on its internal "impressions" but on information it just "gathered". That significantly boosts the trustworthiness and objectivity of the recommendation.
1 Another Practical Example
You are an experienced nutritionist and fitness coach. I need a 1-day meal plan for weight loss (target: 1800 kcal). I want the plan to include salmon and avocado.
**Your methodology (mandatory):**
1. **Step 1: Fact-checking and calculations.**
* Using public nutrition databases (simulate a search), find the exact calories and protein, fat and carbs (macros) per 100 g for the following foods:
* Salmon (fillet, baked)
* Avocado
* Brown rice (cooked)
* Chicken breast (boiled)
* Olive oil
* Present this data as a small reference table.
2. **Step 2: Build the plan.**
* Using **the exact calorie data you just found**, design a menu for the day (breakfast, lunch, dinner) with the weight of each food.
* The plan's total calories should be as close as possible to 1800 kcal.
* Be sure to include salmon and avocado.
* For each meal and for the day as a whole, calculate total calories and macros.
**Output format:**
First the calorie reference table, then the detailed meal plan by meal.
Usefulness Score: 85
Main Evaluation Criteria
- A. Relevance to prompting techniques: High. Although the study doesn't give users ready-made phrasing, it reveals a fundamental principle: to make answers more reliable, you have to make the LLM perform external verification (fact-checking, code execution). That directly shapes how you should build complex prompts.
- B. Better conversational answers: High. Applying the study's principles (simulating fact-checking or code execution in the prompt) leads directly to more accurate, trustworthy and factually correct answers, especially in complex domains.
- C. Direct practical applicability: Medium. You can't directly use the agent framework with tools described in the paper. However, you can adapt and simulate the approach in your prompts, making the model verify information first and answer second.
- D. Conceptual value: Very high. The study brilliantly illustrates a key limitation of LLMs: they're unreliable with facts, math and code without an external "anchor". It gives you the right mental model: an LLM isn't a knowledge base but an information processor that needs to be fed verified data or made to go find it.
- E. New useful practice (clustering): The work falls into several key clusters at once:
- Cluster 1 (Phrasing techniques): Shows the value of task decomposition (verify, then answer).
- Cluster 2 (Behavioral patterns): Shows that without external verification LLMs are prone to errors and bias (for example, they favor code that looks stylistically like their own, even when it's wrong).
- Cluster 6 (Context and memory): Essentially proposes an advanced form of RAG (Retrieval-Augmented Generation), where "retrieval" means active search or code execution.
- Cluster 7 (Reliability and stability): This is the core of the study. The whole method is aimed at reducing hallucinations and increasing factual accuracy.
- Practicality checklist (+15 points): Yes, the work shows how to structure complex requests, reveals non-obvious LLM behavior and offers ways to improve answer accuracy.
2 Numeric Usefulness Score
The score of 85 reflects the study's enormous conceptual value for any advanced LLM user. It doesn't hand you ready-made "magic spells", but it equips you with a fundamental understanding of why and how to build reliable prompts for tasks that demand accuracy.
Arguments for the score: * A fundamental principle: "Verify before you answer" is one of the most powerful prompt-engineering techniques for fighting hallucinations. The study gives it a scientific basis. * Setting the right expectations: Once you understand this paper, you'll stop blindly trusting LLM answers and learn to phrase requests that minimize the risk of misinformation. * Direct adaptation: The principle adapts easily into prompts via instructions like "First find the facts online, then answer based on them".
Counterarguments (why the score could be lower): * A high barrier to entry: The paper is written in dense academic language and focuses on a specific task (LLM-as-a-Judge), not on helping the everyday user. To get value from it, you have to "translate" the findings into the language of practical prompting. * No ready-made solutions: You don't get copy-paste templates. You have to digest the principle yourself and build it into your requests.
Counterarguments (why the score could be higher): * Strategic value: Understanding this principle matters more than knowing a dozen small tricks. It's the shift from "guessing" a working prompt to deliberately designing reliable requests. For someone aiming to use LLMs professionally, the value is close to 100.