ChartMind: Teaching an Analyst Agent to Decide for Itself When to Look at a Chart and When to Look at the Numbers

STEP 1: Run each task several times in different modes
        (free / text only / charts only, with a ban on
        repeating chart types) → a set of trajectories

STEP 2: A judge scores the final outcome and labels each key step
        (helpful / useless / harmful) → labeled trajectories

STEP 3: A builder compares trajectories and writes rules
        into documents of two families:
        • "analysis" — how to choose and verify operations
        • "representation" — when text, when a chart, when both

STEP 4: Validation gate — an edit stays only if quality on
        held-out tasks strictly improved → a frozen package

STEP 5: The working agent first reads the document descriptions,
        then opens the relevant ones → solves the task using the rules

Steps 1–4 are the rule "factory"; step 5 is where the rules get used. If you don't write code, step 5 is available right away (an instructions file), and steps 1–4 can be repeated by hand; see the adaptations below.

Task: An Amazon seller exported a CSV with a year of sales: 40 SKUs, daily data, returns, price, stock levels. They need to understand why revenue dipped in October and, separately, compute the exact total of returns for one SKU. The work is done by an agent with command-line access, such as Claude Code, which can open images saved to disk.

Prompt (in the agent's instructions file):

<ExperienceAccess>
Before analyzing the export sales_2025.csv, read experience/INDEX.md.
It lists the documents with descriptions. Open only the ones that fit
the current step of the analysis.
</ExperienceAccess>

<RepresentationRules>
- Time series, trends, seasonality, outliers, anomalies →
  build a chart, save it as PNG, look at it, then describe
  what you see in words.
- Exact value, sum, finding a row, checking a condition
  ("only SKUs with returns > 5%") → print the numbers to the console.
- If a chart reveals a suspicious stretch → confirm it with numbers
  from the table before drawing a conclusion.
- Before each intermediate result, write one line:
  I choose text / chart / both — and why.
</RepresentationRules>

Task: explain the revenue dip in October 2025 and compute
the exact total of returns for SKU AMZ-48213 in dollars.

Result: The agent will first label its format choice for each step. For the October dip it will plot revenue by day and by SKU and describe where the drop shows up and which SKUs caused it. It will double-check the suspicious stretch against numbers from the table. The returns total for the SKU gets printed as a number, no chart. The final output: an explanation of the dip backed by both the chart and the numbers, plus the exact returns total.

The weakness. By default a model follows its habitual path, and for most models that path is textual: print the table, calculate, answer. Even when a chart is needed (a trend, an anomaly), the agent doesn't build one. It's not that it can't read images. It just doesn't choose that path. Some models barely draw, others draw a lot. Answer quality doesn't depend on drawing frequency.

The strength. Models follow explicit rules well, especially when those rules sit in a document and get read at the right moment. Both a chart and a table carry enough information for a good answer. The format choice can be captured as a rule: "if the task is about a trend, chart; if it's about an exact number, text."

How the method exploits this. ChartMind moves the format choice out of "model habit" and into an explicit document. Representation rules and analysis rules work together. The first raise the quality of visual evidence, the second raise the accuracy of conclusions. The documents live as separate files with short descriptions, so the agent reads only what it needs and the context doesn't bloat.

Control knobs: - The list of chart triggers → extend it for your domain (funnels, cohorts, seasonality) so the agent doesn't skip visual tasks. - The line "I choose text / chart / both — and why" → makes the choice visible, so you can catch where the agent ignores the rule. Drop it to save tokens. - The requirement to confirm a chart with numbers → boosts accuracy. Loosen it for quick exploration. - The number of documents → split by topic with short descriptions. Then the agent opens 1–2 files instead of the whole rulebook.

The method stores experience as a set of documents: an index plus separate files with the fields name, description, category. The paper doesn't publish the text of the rules themselves, so the content below is a starter draft. It's based on the patterns from the study (charts for time, patterns and anomalies; text for exact values and checks). Fill in the rest with your own rules.

1. Agent instructions (CLAUDE.md / system prompt):

<ExperienceAccess>
You have a folder experience/ with experience documents.
1. Before starting a {task_type} task, read experience/INDEX.md —
   it holds the names and descriptions of the documents.
2. Open only the documents relevant to the current step of the analysis.
3. At each intermediate step, state in one line:
   result format: text / chart / both — and which rule applies.
4. Do not edit the experience/ folder — it is read-only.
</ExperienceAccess>

2. experience/INDEX.md:

# Experience index
- analysis_{number}_{topic}.md — {what_it_covers}  [category: analysis]
- repr_{number}_{topic}.md — {what_it_covers}  [category: representation]

3. Representation document (experience/repr_01_when_chart.md):

---
name: repr_when_chart
description: When to show an intermediate result as a chart and when as numbers
category: representation
---
Chart (save a PNG, look at it, describe it in words) if the step is about:
- {change_over_time}
- {finding_a_pattern}
- {anomalies_and_outliers}

Text (print the numbers) if the step is about:
- {exact_value_or_sum}
- {finding_a_specific_row}
- {checking_a_condition}

Both: the chart found a suspicious spot → confirm it with numbers.

How to use the result next: {rule_after_chart}

4. Analysis document (experience/analysis_01_verify.md):

---
name: analysis_verify
description: How to choose and verify analytical operations
category: analysis
---
- Before concluding, cross-check {key_metric} two different ways.
- If a breakdown gives an unexpected result, check {common_data_trap}.
- {other_rules_from_your_experience}

What to fill in: {task_type} is your kind of analysis (sales, funnel, finance). The other placeholders are your rules, ideally drawn from real past runs (see the adaptation below).

🚀 Quick start — paste into a chat:

Here is a template for a set of experience documents for an analyst agent (based on the ChartMind method).
Adapt it to my task: {describe what data you analyze and what questions you ask}.
Ask me questions to fill in the fields.

[paste the template above]

The LLM will ask what data you have, which questions come up most often and where the agent usually slips. It needs that to replace generic rules with chart and text triggers tailored to your domain. The structural pattern (index, metadata, two rule families) it takes from the template.

⚠️ The effect is modest: on overall metrics the gain is a few points, and on exact-answer tasks with a strong base model it's even smaller. The only noticeable jump is in the quality of visual evidence. Don't expect "we added a rule about charts" to double your accuracy.

⚠️ The paper's rules aren't published: the available text doesn't include the experience documents themselves. The template above is a starter draft based on the patterns, not a copy of the original. Test your own rules on your own data.

⚠️ Charts hurt where precision matters: for looking up a value, a sum or checking conditions, text is better. A rule like "always draw a chart" does harm.

⚠️ You need an environment with images: the agent must be able to save an image and then "see" it. In a plain chat without code execution you can't reproduce this.

⚠️ The full loop is engineering: the automated "rule factory" with a judge and validation gates is code. You can repeat a simplified version by hand, but with less rigor.

⚠️ Evaluation is automatic and some slices are small: all answers were graded by an LLM judge. Some question types are represented by just a handful of tasks, such as forecasting and conditional reasoning. The single rule package was built on one model.

First the authors checked whether agents need charts at all. They took 200 tasks on enterprise tables from InsightBench and ran each in four modes with the same agent: no tools, text only, charts only and free choice. The charts-only mode turned out no worse than text-only. So a chart is enough to answer the question, and the agent can read what it drew itself.

Next they gave the agents free choice. It turned out the models behave completely differently. One combined text and charts in about half of the tasks; two others almost always stuck to text. Drawing frequency didn't correlate with quality. A breakdown by question type showed complementarity: charts win on temporal change, patterns and anomalies; text wins on exact reading and conditional checks. The free mode didn't inherit the chart advantage where charts are stronger.

ChartMind grew out of this. To collect rules you need trajectories with charts, and the model doesn't produce them on its own. So each task was forced to be solved several times: freely, text only, and in three chart variants where each next one couldn't repeat the chart types of the previous one. A judge scored the outcome and labeled steps as helpful, useless or harmful. A "builder" compared trajectories and wrote rules, and the validation gates let an edit through only if quality on held-out tasks strictly went up.

They tested on three external benchmarks (reports, long-horizon research, exact-answer tasks) and four models. Results improved everywhere, and models that never drew charts picked up the habit. Ablations showed that every part matters: - without the forced-chart runs, the visual part barely grows; - without the validation gates, the gain is noticeably smaller; - analysis rules alone improve accuracy; - representation rules alone improve the visual part; - together the rules give the best result.

The practical takeaway: rules about "what to compute" and "how to show it" are best kept together, but in separate documents.

💡 Context adaptation: a manual ChartMind loop in chats. Without code you can repeat a simplified version: 1. Take 5–10 typical questions about your data. 2. Have the agent solve each one twice: once "numbers only," once "charts only." 3. Ask another model or a fresh chat to act as the judge:

Here are two solutions to the same task and a reference answer. Score each from 0 to 1.
Label the key steps: helpful / useless / harmful.
Separately, say whether the chosen format (text or chart) helped or got in the way.
Then derive 3–5 rules: when tasks like these need text and when they need a chart.

4. Write the rules into a repr_*.md document and test them on new questions that weren't in the first sample. If quality didn't improve, reject the rule (that's your stand-in for the validation gate).

🔧 Technique: add "confirm the chart with numbers" to the instructions → fewer false conclusions. Charts are great at showing where something looks off, but bad at exact values. Requiring a number check on the suspicious spot lowers the risk of concluding from a picture.

🔧 Technique: drop the line "I choose text / chart / both — and why" → save tokens. Keep it during debugging while you check whether the agent follows the rules. Remove it in production.

Extrapolation: "experience" as skills. The structure from the paper (an index, a document description, on-demand loading) matches how skills and instruction files work in modern agents. Split the rules for your analyst agent into separate skills by topic. The agent reads the descriptions and opens the full text only when needed. That saves context, and the format-choice rules end up right next to the calculation rules.

• Paper: Charts as Epistemic Actions for Experience-Driven Agentic Data Analysis
• Authors: Xinzhe Wang, Xudong Weng, Zang Li (Tencent), Shaolei Zhang (Renmin University of China)
• Benchmarks used: ProbeBench (200 tasks based on InsightBench), DAComp-DA, DDR-Bench, DSBench
• Foundational concept: ReAct (Yao et al., 2023)