AgentOnto: A Business Glossary Instead of Guesswork, So Your Analyst Agent Stops Fudging the Numbers

STEP 1. BUILD THE DICTIONARY: database + documents + example tasks
        → entities / metrics / table mappings
        (every entry has: source + "confirmed" status)
        Confirmation is a separate model pass.

STEP 2. WARM-UP (2 rounds × 4 tasks): the model invents its own questions,
        solves them using the dictionary, a separate pass checks the answers
        → any gaps found get added to the dictionary

STEP 3. HANDLING A REQUEST:
        request → decision: execute / clarify / "not supported"
        → plan with references to the dictionary
        → check the plan against the dictionary (entities exist? metric follows the rules? mapping exists?)
        → SQL from the mappings → execution

STEP 4. UPDATE from confirmed feedback:
        edit a single entry
        → "should change" checks + "should stay the same" checks
        → accept the edit only if both groups pass; otherwise roll back

In the original, this is a coded system. Steps 1–4 run automatically, and the model is called at each step.

The method shines where your data has non-obvious business rules that aren't reflected in column names. There's no point using it for a one-off simple export.

Task: You sell on Amazon and Etsy. Your sales export sits in Google Sheets or a database, and an agent (Claude Code, Cursor or a custom GPT) is looking at it. The owner asks: "What's our revenue for the Home category in April, and how many active buyers do we have?" The tables have statuses like "delivered", "returned" and "canceled", and prices include tax. The agent has no idea that your revenue counts delivered orders only, minus returns, net of tax.

Prompt (put it in the system prompt or CLAUDE.md):

<Ontology>
  <Entities>
    Order(orders): order_id, status ∈ {delivered, returned, canceled}, price_gross, vat_rate, sku_id, buyer_id, delivered_at
    SKU(skus): sku_id, category
    Buyer: buyer_id (only inside orders; no personal data)
  </Entities>
  <Metrics>
    Revenue = SUM(price_gross / (1 + vat_rate)) over orders with status='delivered' MINUS the same sum over status='returned'.
      Source: "Reporting Policy v3", section 2. Status: confirmed.
    Active buyer = buyer_id with ≥1 delivered order in the last 90 days before the report date.
      Source: example task #14. Status: NOT confirmed (needs review).
  </Metrics>
  <Mappings>
    Revenue → orders.price_gross, orders.vat_rate, orders.status; category → JOIN skus ON orders.sku_id = skus.sku_id
  </Mappings>
</Ontology>

<Procedure>
For every request:
1. Decide the action: EXECUTE / CLARIFY / UNSUPPORTED.
   CLARIFY — if the request is missing information (period, channel).
   UNSUPPORTED — if the request needs data that isn't in the dictionary (e.g., margin: there's no cost data).
2. Write a PLAN: which Ontology entries you use, and which choices you made yourself (period, filters).
3. CHECK THE PLAN: are all entities in Entities? Is the metric computed exactly per its definition? Is there a Mapping for every field?
   If not — fix the plan or change the action.
4. Only after the check, write the query and run it.
5. In the answer show: the action, the plan, which Ontology entries were applied, the result. Mark metrics with "not confirmed" status as preliminary.
</Procedure>

Result: The agent first states its decision ("execute" or "clarify") and a plan that cites the definitions. Then it shows the plan check: returns and tax are accounted for. Only after that do you get the query and the final number. Ask about margin and the agent should reply "not supported: no cost data". It will flag "active buyers" as preliminary, because that definition isn't confirmed.

The weakness. An LLM writes a plausible query based on column names and guesses the business meaning. If the query runs, nobody notices the number was calculated "not your way". On top of that, "an answer was returned" ≠ "the answer is right": a high answer rate doesn't guarantee correct answers.

The strength. The model is good at following explicitly written definitions and at checking a plan against a checklist. It can also come up with test questions from documents and work through discrepancies.

How the method exploits this. Knowledge moves out of the analyst's head into one auditable document, and the plan gets checked before execution. Edits are tested for "regressions": you fixed the "active buyer" metric without breaking "revenue".

Control knobs: - Confirmation status → treat only confirmed definitions as "hard" and mark the rest as preliminary. - Number of warm-up rounds and tasks (2×4 in the paper) → increase for a big database, decrease for a small one. - Three actions (execute / clarify / decline) → add your own, e.g., "hand off to a human". - "Will change" and "will stay" checks → keep building a set of questions with known answers; it becomes your test suite.

The authors don't publish prompts; they describe a coded system. Below is the idea translated into a text instruction following its structure. It's not a verbatim prompt from the paper.

<Role>
You are a data analyst. You do not write a query until you have checked it against the Ontology.
</Role>

<Ontology>
  <Entities>
    {entities: tables, key fields, allowed values}
  </Entities>
  <Metrics>
    {metric} = {formula and conditions of use}. Source: {document/example}. Status: {confirmed / not confirmed}.
  </Metrics>
  <Mappings>
    {concept} → {table.column, JOIN relationships}
  </Mappings>
</Ontology>

<Procedure>
For every request {request}:
1. action = EXECUTE | CLARIFY | UNSUPPORTED.
2. plan: which Ontology entries are used; which assumptions are made beyond it.
3. check(plan): entities from Entities? metrics per definition? a Mapping for every field?
   If the check fails — fix the plan or change the action.
4. compile: build the query strictly from Mappings.
5. Output: action, plan, check result, Ontology entries applied, result.
</Procedure>

<UpdateRule>
When I send a confirmed correction {correction}:
1. Find the ONE Ontology entry it applies to.
2. Propose a targeted edit.
3. Write corr_checks: questions whose answer MUST change.
4. Write ret_checks: questions whose answer MUST stay the same.
5. Run both groups following the <Procedure> rules.
   Accept the edit only if both groups pass. Otherwise roll it back and explain why.
</UpdateRule>

What to fill in: in {entities} — your database schema and allowed field values. In {metric} — definitions from your policies. In {correction} — your confirmed fixes, e.g., "revenue must exclude tax".

🚀 Quick start — paste into the chat:

Here's a template for an ontology-guided analyst. Adapt it to my task: [your task and data]. 
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask about your tables and fields, metric definitions and where they come from, and which questions can't be answered from this data. That matters because the method rests on the three parts of the dictionary: entities, metrics, table mappings. Without metrics and data boundaries, the agent slides back into guessing.

⚠️ Requires an engineering build: in the original everything is automated (building the dictionary from the database, documents and examples, the confirmation pass, the warm-up loop, rolling back edits). In a chat you can reproduce the idea, but not the automation. You'll have to maintain the dictionary by hand or write a script.

⚠️ Results come from data the authors prepared themselves: they assembled the databases, documents and example tasks on their own, from open data. Comparisons with other methods used the same model, so on your model and your data the effect may differ.

⚠️ The dictionary goes stale and can be wrong: if a definition in it is incorrect, the agent will confidently compute the wrong thing. A "confirmed" status is no substitute for human review.

⚠️ Not for simple queries: if your database has no hidden business rules, the dictionary won't buy you anything — it just adds upkeep.

⚠️ Not all results are covered here: the paper text we had cuts off at the first experiments section. Results on dictionary building, on analysis and on updates, as well as the description of the real-world deployment, aren't covered in this breakdown.

The researchers took two open databases from different fields: World Development Indicators (country-level economics) and EPA air quality data. For each one they built an "enterprise" environment: a database, business documents and example tasks. That produced 606 queries (282 and 324) and 44 confirmed corrections with 174 post-edit queries. An analysis of a retail dataset showed that queries require 46 business concepts that aren't in the database schema. That's exactly the gap the dictionary closes.

They compared against four methods: DIN-SQL, ReFoRCE, MAC-SQL (multi-stage SQL generation) and Vanna (a tool-using agent). In every case they plugged in the same model AgentOnto used, so the difference comes from the method, not the model's horsepower. A query counted as correct only if the action was chosen correctly (execute, clarify, decline) and, where applicable, the result was right.

Bottom line: AgentOnto hit 87.5% accuracy on average (79.1% and 96.0% across the two domains), versus 55.0% for the best competitor. The gains were 32.6 and 30.3 percentage points. The practical insight: writing down business meaning explicitly pays off more than cleverer SQL generation. The two checks on dictionary updates, "should change" and "should stay", guard against fixing one thing and breaking another.

🔧 Technique: add a "reference question set" → edits don't break what already worked

Keep a separate list of 10–20 questions with known correct answers. Before accepting any change to the definitions, ask the agent to run through that list:

Before accepting the edit: run every question from <ReferenceSet>.
If even one answer changed without my say-so — roll back the edit and show me which one.

This brings the "should stay the same" checks into a regular chat. The original builds those checks automatically.

Extrapolation: a dictionary self-check before launch. Before going live, ask: "Come up with 4 questions that probe the weak spots of this dictionary. Solve them by the rules. Show me where the dictionary is ambiguous or empty, and propose fixes." This is a stripped-down version of the warm-up on self-generated tasks: one round, no separate checker model. For reliability, run the second pass in a fresh chat.

• AgentOnto: Ontology-Guided LLM Agents for Enterprise Data Analytics — Dayan Pan, Bincheng Zuo, Xiaojian Wang, Daogang Zhang, Jingyuan Wang. Beihang University (Beijing) and Omaha AIGC (Shanghai). KDD '27.
• Systems and methods mentioned in the paper: DIN-SQL, ReFoRCE, MAC-SQL, Vanna, SiriusBI, CoeusBI, SODA, ATHENA.
• Data: World Development Indicators, U.S. EPA Air Quality System.