TL;DR
When an LLM analyzes your data (tables, reports, databases) and your prompt only explains what you want to know, the model starts guessing what your metrics mean. Instead, you can hand it a short document: what "revenue" means in your specific table, how margin is calculated, which order statuses count. The study calls this a semantic layer, a layer of meaning.
In the study (models writing SQL against a retail database), without such a document the models answered correctly in roughly half the cases (45–51%). That happens because a data schema stores no business logic, only structure. Column names like SalesAmount or FactOnlineSales don't tell you: is this all sales or just online? Are returns subtracted? Is "region" the customer's region or the store's? The model sees a bare structure and has to guess.
Adding a 4 KB explanation document (about 9,000 characters, roughly 2,200 tokens) to the same schema lifted accuracy by 17–23 percentage points, to about 68%, whether the model was Claude or GPT. And that's the key insight: on this benchmark the gap between models was statistically negligible, while the gap between "with context" and "without context" was large.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Write a semantic document → a few pages of markdown with the rules of your table (the paper's was ~4 KB)
(what counts as "revenue," how to treat ambiguous statuses, which columns are responsible for what)
STEP 2: Put the document + data (or schema) + question into one prompt → an answer that follows your business logic
Everything happens in a single prompt, no separate requests.
Example
Our illustration, not an example from the paper (the paper used SQL questions over the Contoso retail database).
Task: You're on the team of an Amazon seller. You want Claude to analyze the weekly sales report. The table has orders, keep_rate, gross_sales, fees, returns. Without context Claude starts guessing: is "gross sales" revenue or GMV? Is "keep rate" a percentage or an absolute number? Which orders should the date filter apply to?
Prompt:
## Business context (semantic layer)
**Metrics and their definitions:**
- "Orders": number of placed orders regardless of status
- "Keep rate": orders delivered and kept by the customer divided by all delivered orders; expressed as a percentage; excludes orders with status "in transit to customer"
- "Gross sales": total of all orders at selling price, before returns and fees
- "Net revenue": gross sales minus returns minus the Amazon referral fee (15%)
- "Return": any order with status "refused on delivery" or "returned to warehouse"
**Table conventions:**
- The date in the `order_date` column is the order date, not the delivery date
- If `promo_code` is empty, the order had no promotion
- The `region` column is the customer's region, not the warehouse's
**Rule for "the latest period":**
- If I ask "for the week," take the last 7 days counting back from the maximum date in the table, not from today
---
## Data
[paste the table or CSV]
## Question
Which 5 SKUs generated the most net revenue over the last 2 weeks? Show gross sales, returns and final net revenue for each.
Result: The model applies your definitions literally: it calculates net revenue using the formula from the document, takes the correct period from the maximum date, and doesn't confuse gross sales with revenue. Without the document it may guess wrong, for example treating gross sales as revenue or counting dates from "today" (in the paper's SQL benchmark, about half the questions failed without the document, and anchoring "recent" to today was one of the typical failures).
Why It Works
LLMs are good at applying explicitly written rules. They're bad at guessing rules from data structure. Those are different tasks, and the semantic layer turns "guess what this column means" into "take the definition from the document."
A data schema tells you what exists. A semantic layer tells you what it means and how to use it. The gap between the two is the source of most data-analysis errors. The paper cites earlier work (Shen et al., 2025, on the Spider and BIRD text-to-SQL benchmarks) finding that schema errors cause more than 80% of execution failures: wrong table or column, wrong join, wrong interpretation of a term. In its own runs, the typical wins from the document were wrong fact table, summing daily snapshots, missing metric formulas, hidden data conventions and anchoring dates to today.
The key lever: you're not making the model smarter, you're changing the task itself. "Guess the business logic from a bare schema" → "apply explicitly written business logic." These are fundamentally different instructions.
Control levers: - Level of detail in the document → more detail = less guessing. Start with the 5-7 least obvious metrics - Disambiguation rules → if a term can mean two different things, write down both readings and the rule for choosing - Data conventions → string booleans ('Yes'/'No' instead of 1/0), "latest period" dates, sentinel values: spell them out - Which table to use for what → if you have several data sources, state the priority
Prompt Template
## Semantic layer
**Metric definitions:**
- "{metric_1}": {what it means, how to calculate it, what to include/exclude}
- "{metric_2}": {what it means, how to calculate it}
**Table conventions:**
- Column "{column}" means {what}, not {what it could be confused with}
- If field {field} is empty, {how to interpret it}
- The period "last {period}" is counted from {starting point}
**Rules for resolving ambiguity:**
- If the question says "{term}", use {which source/metric exactly}
- If no dates are given, use {default period}
---
## Data
{paste the table, CSV or a description of the structure}
## Question
{your question}
Placeholders:
- {metric} — the key metrics in your field: revenue, conversion, LTV, keep rate
- {column} — the column names in your table
- {period} — typical time-based asks: "week," "quarter," "last month"
- {term} — words with a double meaning in your context
🚀 Quick start — paste into the chat:
Help me put together a semantic layer for my data.
Ask me questions one at a time so I can fill in the template.
My task: [describe your task: what the data is and what you want to do with it]
[paste the template above]
The LLM will ask for your metric names, what they mean, and whether any terms are ambiguous, because without that it doesn't know where to get a disambiguation rule. It will take the template's structure and adapt it to your field.
Limitations
⚠️ Accuracy ceiling: Even with detailed context the models answered correctly in ~68% of cases, so about a third of the answers were still wrong. The remaining failures clustered on percentiles, correlations, standard deviations, threshold rankings and complex multi-step SQL, plus a few genuinely ambiguous questions.
⚠️ You have to write the document: A semantic layer doesn't create itself; someone has to sit down and pin down the business logic. It's one-time work, but it takes time.
⚠️ Context is advice, not law: Unlike a technical system (code, a database), an LLM in a chat receives your document as a recommendation, not a hard rule. The model can still ignore it.
⚠️ The study is about SQL, not everything: The experiment was run on tasks that generate SQL queries against databases. The principle likely carries over to analyzing tables in a chat, but the paper has no data on that or on other kinds of analytical tasks. It also tested only single-shot answers at medium reasoning effort, and the document was written by an analyst who knew the dataset.
How They Tested It
The idea was simple: take the same 100 questions (99 scored) about one dataset (Contoso retail sales: 25 tables, data from 2007–2009), three models (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4), and test each one twice. Once with the bare database schema, and once with the same schema plus 4 kilobytes of markdown explaining the business logic. Everything else was identical: same questions, same tool, same judge for scoring.
The most surprising result: the models didn't differ from each other. Without the document, all three scored 45.5–50.5% correct answers. With the document, all three landed at 67.7–68.7%. The difference between models is statistically insignificant; the difference between conditions is large (p ≤ 0.0015). For practitioners the message is direct, with the authors' caveat that this is one 99-question benchmark: spend less time picking a provider and more on the quality of your context.
The questions were spread evenly over five difficulty levels, 20 each: from simple single-table aggregations to expert questions with ambiguous business terms, recursive queries and cohort/LTV filters. The paper does not break down the improvement by difficulty level.
A telling detail: the authors cite an earlier Bayer study (JAMIA Open, 2024) on medical data (60 questions against a pharmacovigilance database), where GPT-4 without a context document scored 8.3% accuracy, and 78.3% with one. A 70-percentage-point gap. The authors argue from such results that the effect isn't unique to retail but structural.
Original Text from the Paper
A semantic layer is an authoritative description of a dataset's business concepts —
the measures, dimensions, entities, and relationships that together constitute the
language in which analytical questions are asked. It is the artifact in which a data
team records decisions like "revenue means SalesAmount from the online-sales fact,
excluding canceled orders," or "region means the customer's country-continent
hierarchy, not the store's," or "inventory means the latest daily snapshot, not a
sum over time."
A well-constructed semantic layer for a given dataset typically specifies:
• Fact table selection rules — Which fact table is authoritative for which question class
• Measures and their formulas — How metrics are computed with the exact columns and aggregations
• Dimensional hierarchies and join paths — How entities relate and which keys to join on
• Data conventions and quirks — How the data actually encodes things: string-booleans,
sentinel values, snapshot-vs-cumulative semantics, currency conventions
• Disambiguation rules — How to resolve ambiguous phrasing: whether "sales" defaults
to online or all-channel, whether "region" refers to customer or store geography,
whether "last quarter" is relative to today or to the dataset's latest date
Context: The authors describe exactly what goes into a semantic layer. This is their working definition from Section 3 and 3.1 (abridged: some examples inside the list items are omitted). The 4KB document used in the experiment covered these same categories.
Adaptations and Extrapolations
💡 Adaptation: A semantic layer for analyzing other documents, not just data
Our extrapolation (the paper tested only SQL over a database): the principle may reach beyond SQL. If you ask Claude to analyze a contract, financial statements or a technical document, give it a "glossary" for your field up front.
Task: Analyzing a supply contract with non-standard terms
Prompt add-on before the document:
## Context for the analysis Terminology of this contract: - "Supplier" in our context = Miller Supply LLC; we act as the "Customer" - "Penalties" in our practice apply only after the 2nd violation; the 1st is a warning - "Acceptance" is considered complete once both parties sign, not upon delivery - We resolve disputes in the courts of Cook County, Illinois, even if the contract names another venue Analyze the contract with these rules in mind.
🔧 Technique: Document the "red flags" for your data domain
Instead of (or in addition to) metric definitions, give the model a list of typical traps in your data.
## Known traps in this data - The is_active column contains the strings 'Y'/'N', not booleans: don't filter with == True - The orders table has duplicate rows for orders with multiple statuses: count DISTINCT order_id - Dates in created_at are UTC+0, our sales are in UTC-5: shift by 5 hours when grouping by day - Orders with promo code NULL and with promo code 'NONE' are different situations (NULL = no promo, 'NONE' = promo rejected)
This targets one class of "silent errors" the paper saw often (string booleans, sentinel values, snapshot tables), where the LLM returns a syntactically correct but wrong result.
🔧 Technique: A cumulative semantic layer you update as you go
Start with a minimal document, look at where the model goes wrong, and add rules.
- Ask a question with a basic semantic layer
- Get the answer
- Ask: "Explain step by step how you calculated this metric"
- Find where it diverges from your business logic
- Add that rule to the document
The authors note that the remaining failures in their benchmark could be addressed by extending the document with named measures and more disambiguation rules; how many rounds it takes depends on your data.
Resources
Title: Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
Authors: Michael Rumiantsau, Ivan Fokeev — Cube (cube.dev)
Code and data: https://github.com/cubedevinc/semantic-layer-benchmark
Dataset: Cleaned Contoso Retail Dataset — https://www.kaggle.com/datasets/bhanuthakurr/cleaned-contoso-dataset
Benchmarks mentioned: BIRD (Li et al., 2023), Spider 2.0 (Lei et al., 2025), BEAVER (Chen et al., 2024), NL2SQL-Bugs (Liu et al., 2025), dbt Labs benchmark (Ganz & Perigaud, 2026), Bayer/JAMIA Open (Zargari Marandi et al., 2024)