TL;DR
Models lose up to 30% accuracy when a problem comes with an image. The cause isn't a lack of subject knowledge — it's an inability to first parse the image and then think about the problem. Structured Multimodal Dialogue is a method that splits those two steps apart: first the model describes the visual content, then it solves the problem.
Main finding: on text-only problems, models answer correctly 96% of the time. Add an image and accuracy drops to 64–89% depending on the model. The authors call this the Multimodal Interference Effect: visual processing "overloads" the model, and it starts confusing even the basic meaning of the problem — even though its subject knowledge hasn't gone anywhere. It's like a smart person who knows physics inside out but is reading the problem sheet upside down.
The method fixes this with a three-step dialogue. Step 1: ask the model to describe the image. Step 2: correct only factual perception errors (don't hint at the solution). Step 3: ask it to solve the problem. This sequence fixed 82% of errors. Visual errors — 100% of them.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
All three steps happen in one conversation (separate messages in the same chat)
STEP 1 → First message:
Attach the image + ask for a description:
"Describe what you see in this image/graph"
→ You get: a text description of the visual
STEP 2 → Your reply:
Check the description. If something is wrong, correct
only the visual facts (numbers, directions, labels).
Don't hint at the solution.
→ You get: corrected perception
STEP 3 → Next message:
"Now solve the problem: {problem}"
or: "Read the question again and solve the problem"
→ You get: a solution based on correct perception
Example in Action
Task: You're looking at a screenshot of a sales funnel from HubSpot and want to find where the biggest drop-offs are. The numbers on the screenshot are tiny, the conversion rates are jumbled, and the model is clearly misreading something on the first try.
Prompt (Step 1):
[Attach the funnel screenshot]
Look carefully at this image.
Describe exactly what it shows:
which stages, which numbers, which conversion rates between them.
List everything you see — in order, from top to bottom.
Your response (Step 2):
You read the description and spot a mistake — say, the model mixed up the "Lead" and "Qualified Lead" stages or misread 1,200 as 1,20.
You write: "Not quite: the 'Qualification' stage has 3,400 leads, not 340. And that stage comes after 'Lead,' not before it."
Prompt (Step 3):
Now, based on this data:
find the stage with the biggest customer loss in percentage terms
and suggest 2-3 hypotheses for why conversion dropped right there.
Result: The model will deliver an analysis based on the correct data — it will find the funnel's bottleneck, name the specific loss figure and propose hypotheses. Without steps 1-2, it would be analyzing a distorted picture and drawing wrong conclusions.
Why It Works
The LLM's weakness: the model processes text and image at the same time, "in one pass." That's an overload. The visual part needs its own attention — reading axes, parsing labels, remembering numbers. That "takes up space" and throws the model off when it frames the problem. As a result, errors appear not because the model doesn't know physics or marketing, but because it misread the source data.
The LLM's strength: when the data is given as text, the model reasons excellently. That's exactly why accuracy on text problems is 96%. The knowledge is there. The problem is reading, not thinking.
How the method uses this: the method splits the two processes. First — only perceiving the image (step 1). Then — a human corrects the perception (step 2). And only then — working with the data (step 3). In essence, you turn the image into verified text, and from there the model works in its strong zone.
Prompt control levers:
- Level of detail in step 1 → ask it to "name all numbers and labels" for complex charts, or to "describe the overall structure" for diagrams
- Scope of correction in step 2 → fix only facts, give no hints about the solution (this is essential — otherwise you're solving the problem yourself)
- Wording of step 3 → if the description was long, add "reread the question" before "solve" so the model doesn't lose sight of the goal
- Multiple images → run each one separately through steps 1-2, then launch step 3 with all the descriptions
Prompt Template
Step 1 — Description:
[Attach the image]
Describe in detail what you see in this image.
{Task-specific note: "List all numbers and labels" /
"Describe the structure and connections" / "Name all elements from top to bottom"}
Step 2 — Your correction (if needed):
Correction: {fix only the visual fact — a number, a direction, an order}.
{Other details} are correct.
Step 3 — The problem:
Now, based on this data, {problem}.
What to fill in:
- {Task-specific note} — depends on the image type: charts → numbers and axes; diagrams → connections and order; tables → headers and values
- {fix only the visual fact} — don't explain the problem, only correct the perception
- {problem} — your question, unchanged from the very beginning
🚀 Quick start — paste into the chat:
Here is a three-step method for working with images (Structured Multimodal Dialogue).
Adapt it to my task: {describe your task}.
Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what kind of image it is and what the end goal is — because that determines what to ask it to describe in step 1 and how to phrase step 3.
Limitations
⚠️ Context errors are hard to fix: If the model misunderstood the gist of the problem from the start (rather than just a visual fact), the method helps only 44% of the time. Describing the image doesn't always cure flawed reasoning about the setup.
⚠️ Requires human checking: Step 2 is you. You need to know what's correct in the image yourself. If you're not sure either, the method won't help.
⚠️ Not for drawing tasks: If you need the model to create or edit an image, this doesn't apply. Analysis of existing images only.
⚠️ Small sample: 44 problems isn't many. Fixing 100% of visual errors is impressive, but the lower bound of the confidence interval is 81%. Larger studies are needed for confidence.
How They Studied It
The team took 100 text problems and 44 problems with images from OpenStax — an open university physics textbook. Three models (Claude, Gemini, ChatGPT) solved the problems under identical conditions. First pure text, then problems with graphs, diagrams and tables. Correctness was checked by a separate multi-model LLM system.
An interesting detail: on text problems the models were practically indistinguishable — 96% for all three. But as soon as images appeared, the gap became statistically significant. That means a model's "smarts" with text don't predict its ability to handle visuals.
The errors were sorted into four types, and the team found something important: computational errors appeared only in text problems, while visual errors appeared only in multimodal ones. This suggests an image doesn't just add difficulty — it creates a different kind of error. So different correction strategies are needed.
Then they applied the three-step dialogue to all the wrong answers. They fixed 28 of 34 errors. Visual ones — 100%. Contextual ones — 44%. The difference turned out to be the key insight: if the model sees wrong, that's fixable. If it misunderstands the meaning of the problem, that's harder.
Original from the Paper
The intervention followed a three-step dialogue:
(i) "Describe what you observe in the figure/graph,"
(ii) researcher correction of any misidentified visual elements
[corrections were restricted to observable visual properties
in order to align the model's perception with the source image.
Researcher input was limited to clarifying the direction of a force vector,
or the numerical value of a graph's intercept
without providing any hints regarding the underlying physics principles or formulas.]
(iii) "Solve the problem"
(or "Read the question again and solve the problem"
when the description was lengthy)
Context: This is the exact intervention protocol from the Methods section. Note the remark on step (ii): correct only visual facts, no hints about the physics or the solution method — otherwise you're solving the problem yourself.
Adaptations and Extrapolations
💡 Adaptation for medicine and health
The same protocol works for analyzing scans, lab results and medical charts.
[Attach the scan/result]
Describe what you see in this image:
all visible structures, markings, numbers, anomalies.
Be as specific as possible.
→ You check for obvious perception errors.
Now explain to me in plain words
what these values mean for my health.
⚠️ Not for diagnosis. For understanding and preparing questions for your doctor.
💡 Adaptation for design reviews and UX
[Attach the UI screenshot]
Describe the structure of this screen:
what elements there are, where they're placed,
what the text says, what the hierarchy is.
→ If something is wrong, you correct it.
Now find 3 UX problems
and suggest how to fix them.
🔧 Technique: remove the human from step 2 → automatic mode
If you can't check the description yourself (say, you're working fast), replace step 2 with a model self-check:
[Step 1 — description]
Now reread your description and check:
did you read everything correctly?
Is there anything in the image you're unsure about?
[Step 3 — the problem]
The effect is weaker (no external check), but better than going straight to "solve the problem." Fine for low-stakes tasks where accuracy isn't critical.
Resources
Paper title: A Dialogue-Based Framework for Correcting Multimodal Errors in AI-Assisted STEM Education
Authors: Akshay Syal, Lawrence Swaminathan Xavier Prince, Evin Gultepe, Nik Bear Brown, Srinivas Sridhar
Textbook base: OpenStax Physics — openstax.org
Affiliation: Northeastern University (based on Nik Bear Brown's author profile)