TL;DR
When an LLM analyzes your data (tables, reports, databases) and your prompt only explains what you want to know, the model starts guessing what your metrics mean. Instead, you can hand it a short document: what "revenue" means in your specific table, how margin is calculated, which order statuses count. The study calls this a semantic layer, a layer of meaning.
Without such a document the model answers correctly in roughly half the cases. That happens because a data schema stores no business logic, only structure. Column names like SalesAmount or FactOnlineSales don't tell you: is this all sales or just online? Are returns subtracted? Is "region" the customer's region or the store's? The model sees a bare structure and has to guess.
Adding 1–2 pages of explanations to the same data lifts accuracy by 17–23 percentage points, whether you're using Claude or GPT. And that's the key insight: the gap between models is negligible, while the gap between "with context" and "without context" is huge.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Write a semantic document → 1-2 pages of markdown with the rules of your table
(what counts as "revenue," how to treat ambiguous statuses, which columns are responsible for what)
STEP 2: Put the document + data + question into one prompt → an answer with the correct business logic
Everything happens in a single prompt, no separate requests.
Example
Task: You're on the team of an Amazon seller. You want Claude to analyze the weekly sales report. The table has orders, keep_rate, gross_sales, fees, returns. Without context Claude starts guessing: is "gross sales" revenue or GMV? Is "keep rate" a percentage or an absolute number? Which orders should the date filter apply to?
Prompt:
## Business context (semantic layer)
**Metrics and their definitions:**
- "Orders": number of placed orders regardless of status
- "Keep rate": orders delivered and kept by the customer divided by all delivered orders; expressed as a percentage; excludes orders with status "in transit to customer"
- "Gross sales": total of all orders at selling price, before returns and fees
- "Net revenue": gross sales minus returns minus the Amazon referral fee (15%)
- "Return": any order with status "refused on delivery" or "returned to warehouse"
**Table conventions:**
- The date in the `order_date` column is the order date, not the delivery date
- If `promo_code` is empty, the order had no promotion
- The `region` column is the customer's region, not the warehouse's
**Rule for "the latest period":**
- If I ask "for the week," take the last 7 days counting back from the maximum date in the table, not from today
---
## Data
[paste the table or CSV]
## Question
Which 5 SKUs generated the most net revenue over the last 2 weeks? Show gross sales, returns and final net revenue for each.
Result: The model applies your definitions literally: it calculates net revenue using the formula from the document, takes the correct period from the maximum date, and doesn't confuse gross sales with revenue. Without the document there's a ~50% chance it would guess wrong, for example treating gross sales as revenue or counting dates from "today."
Why It Works
LLMs are good at applying explicitly written rules. They're bad at guessing rules from data structure. Those are different tasks, and the semantic layer turns "guess what this column means" into "take the definition from the document."
A data schema tells you what exists. A semantic layer tells you what it means and how to use it. The gap between the two is the source of most data-analysis errors. According to the study, more than 80% of errors when LLMs work with data are exactly this type: wrong column, wrong aggregation, wrong interpretation of a term.
The key lever: you're not making the model smarter, you're changing the task itself. "Guess the business logic from a bare schema" → "apply explicitly written business logic." These are fundamentally different instructions.
Control levers: - Level of detail in the document → more detail = less guessing. Start with the 5-7 least obvious metrics - Disambiguation rules → if a term can mean two different things, write down both readings and the rule for choosing - Data conventions → string booleans ('Yes'/'No' instead of 1/0), "latest period" dates, sentinel values: spell them out - Which table to use for what → if you have several data sources, state the priority
Prompt Template
## Semantic layer
**Metric definitions:**
- "{metric_1}": {what it means, how to calculate it, what to include/exclude}
- "{metric_2}": {what it means, how to calculate it}
**Table conventions:**
- Column "{column}" means {what}, not {what it could be confused with}
- If field {field} is empty, {how to interpret it}
- The period "last {period}" is counted from {starting point}
**Rules for resolving ambiguity:**
- If the question says "{term}", use {which source/metric exactly}
- If no dates are given, use {default period}
---
## Data
{paste the table, CSV or a description of the structure}
## Question
{your question}
Placeholders:
- {metric} — the key metrics in your field: revenue, conversion, LTV, keep rate
- {column} — the column names in your table
- {period} — typical time-based asks: "week," "quarter," "last month"
- {term} — words with a double meaning in your context
🚀 Quick start — paste into the chat:
Help me put together a semantic layer for my data.
Ask me questions one at a time so I can fill in the template.
My task: [describe your task: what the data is and what you want to do with it]
[paste the template above]
The LLM will ask for your metric names, what they mean, and whether any terms are ambiguous, because without that it doesn't know where to get a disambiguation rule. It will take the template's structure and adapt it to your field.
Limitations
⚠️ Accuracy ceiling: Even with detailed context the models answered correctly in ~68% of cases, so a third of the answers were still wrong. The more complex the query (several aggregations + conditions + joins), the higher the chance of error.
⚠️ You have to write the document: A semantic layer doesn't create itself; someone has to sit down and pin down the business logic. It's one-time work, but it takes time.
⚠️ Context is advice, not law: Unlike a technical system (code, a database), an LLM in a chat receives your document as a recommendation, not a hard rule. Sometimes the model still drifts from it, especially in long prompts.
⚠️ The study is about SQL, not everything: The experiment was run on tasks that generate SQL queries against databases. The principle works for analyzing tables in a chat, but the paper has no data on other kinds of analytical tasks.
How They Tested It
The idea was simple: take the same 100 questions about one dataset (Contoso retail sales: 25 tables, data from 2007–2009), three models (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4), and test each one twice. Once with the bare database schema, and once with the same schema plus 4 kilobytes of markdown explaining the business logic. Everything else was identical: same questions, same tool, same judge for scoring.
The most surprising result: the models didn't differ from each other. Without the document, all three scored 45–50% correct answers. With the document, all three hit 68%. The difference between models is statistically insignificant; the difference between conditions is huge (p ≤ 0.0015). For practitioners the message is direct: don't waste time picking a provider, spend it on the quality of your context.
The questions were deliberately hard: from simple "how many orders in January" to expert-level "calculate cohort LTV accounting for returns and the dataset's notional current date." The improvement from the document held steady across all difficulty levels, not just the easy or hard ones.
A telling detail: in a parallel Bayer study on medical data (60 questions against a pharmacovigilance database), GPT-4 without a context document scored 8.3% accuracy, and 78.3% with one. A 70-percentage-point gap. The effect isn't unique to retail; it's structural.
Original Text from the Paper
A semantic layer is an authoritative description of a dataset's business concepts —
the measures, dimensions, entities, and relationships that together constitute the
language in which analytical questions are asked. It is the artifact in which a data
team records decisions like "revenue means SalesAmount from the online-sales fact,
excluding canceled orders," or "region means the customer's country-continent
hierarchy, not the store's," or "inventory means the latest daily snapshot, not a
sum over time."
A well-constructed semantic layer for a given dataset typically specifies:
• Fact table selection rules — Which fact table is authoritative for which question class
• Measures and their formulas — How metrics are computed with the exact columns and aggregations
• Dimensional hierarchies and join paths — How entities relate and which keys to join on
• Data conventions and quirks — How the data actually encodes things: string-booleans,
sentinel values, snapshot-vs-cumulative semantics, currency conventions
• Disambiguation rules — How to resolve ambiguous phrasing: whether "sales" defaults
to online or all-channel, whether "region" refers to customer or store geography,
whether "last quarter" is relative to today or to the dataset's latest date
Context: The authors describe exactly what goes into a semantic layer. This is their working definition from Section 3.1, which became the basis of the 4KB document used in the experiment.
Adaptations and Extrapolations
💡 Adaptation: A semantic layer for analyzing any document, not just data
The principle reaches beyond SQL. If you ask Claude to analyze a contract, financial statements or a technical document, give it a "glossary" for your field up front.
Task: Analyzing a supply contract with non-standard terms
Prompt add-on before the document:
## Context for the analysis Terminology of this contract: - "Supplier" in our context = Miller Supply LLC; we act as the "Customer" - "Penalties" in our practice apply only after the 2nd violation; the 1st is a warning - "Acceptance" is considered complete once both parties sign, not upon delivery - We resolve disputes in the courts of Cook County, Illinois, even if the contract names another venue Analyze the contract with these rules in mind.
🔧 Technique: Document the "red flags" for your data domain
Instead of (or in addition to) metric definitions, give the model a list of typical traps in your data.
## Known traps in this data - The is_active column contains the strings 'Y'/'N', not booleans: don't filter with == True - The orders table has duplicate rows for orders with multiple statuses: count DISTINCT order_id - Dates in created_at are UTC+0, our sales are in UTC-5: shift by 5 hours when grouping by day - Orders with promo code NULL and with promo code 'NONE' are different situations (NULL = no promo, 'NONE' = promo rejected)
This removes the most common class of "silent errors," where the LLM returns a syntactically correct but wrong result.
🔧 Technique: A cumulative semantic layer you update as you go
Start with a minimal document, look at where the model goes wrong, and add rules.
- Ask a question with a basic semantic layer
- Get the answer
- Ask: "Explain step by step how you calculated this metric"
- Find where it diverges from your business logic
- Add that rule to the document
After 5–7 iterations you'll have a document that covers most of the typical errors for your specific data.
Resources
Title: Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
Authors: Michael Rumiantsau, Ivan Fokeev — Cube (cube.dev)
Code and data: https://github.com/cubedevinc/semantic-layer-benchmark
Dataset: Cleaned Contoso Retail Dataset — https://www.kaggle.com/datasets/bhanuthakurr/cleaned-contoso-retail-dataset
Benchmarks mentioned: BIRD (Li et al., 2023), Spider 2.0 (Lei et al., 2025), BEAVER (Chen et al., 2024), NL2SQL-Bugs (Liu et al., 2025), dbt Labs benchmark (Ganz & Perigaud, 2026), Bayer/JAMIA Open (Zargari Marandi et al., 2024)