TL;DR
The Forecast Critic is a technique for evaluating time-series forecasts where the LLM looks at a chart (history + forecast) and decides whether the forecast looks reasonable. You upload a chart image to ChatGPT/Claude, add a prompt, and the model analyzes trends, periodicity and spikes and tells you "reasonable" or "suspicious."
LLMs are good at catching obvious blunders: the forecast heads down while the history goes up; a sudden jump for no reason; a forecast that's too flat for volatile data. But they struggle to see compressed/stretched periodicity, when the forecast keeps the pattern but changes its frequency (for example, weekly cycles turn into 5-day ones). The best model (Claude 3.7 Sonnet) scores F1 = 0.88; a human scores 0.97. Forecasts flagged as "unrealistic" really are 10-30% more wrong on accuracy metrics.
The method works in a single request: you upload the image, add context (for example, "there was a sale on March 15, and another one is coming on March 22"), and the model takes it into account. It can detect missing promo spikes (when there should be a jump but there isn't) and false ones (there's a jump but no reason for it).
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
INPUT: Chart image (history + forecast) + text context
↓
The LLM analyzes:
- Trend: does the forecast continue or break the history's direction?
- Volatility: does the forecast keep the amplitude of the swings?
- Periodicity: do the patterns repeat at the same frequency?
- Spikes: are there unexplained jumps or missed events?
↓
OUTPUT: "Reasonable" or "Unreasonable" + an explanation
It all happens in one prompt. No multiple requests or coding required.
Example in Action
Task: You've launched targeted ads for a new collection in your online clothing store. An analyst gave you a sales forecast for the next 30 days that promises growth, but the chart looks off: the history shows spikes for Black Friday and Valentine's Day, yet the forecast has none, even though Mother's Day and Memorial Day weekend are coming up. You want to check whether the analyst botched the model.
Prompt:
Assess whether this sales forecast looks reasonable.
On the chart:
- Blue line: sales history for the last 90 days (units per day)
- Red line: forecast for the next 30 days
Context:
- November 24 was Black Friday (spike in the history)
- February 14 had a Valentine's Day promo (spike in the history)
- May 8-10 (Mother's Day weekend) and May 23-25 (Memorial Day weekend) fall within the forecast period
Answer:
1. Reasonable or Unreasonable?
2. What looks concerning or, on the contrary, logical?
3. If unreasonable, what mistake did the model make?
[attach the chart image]
Result:
The model will analyze the chart and say something like: "Unreasonable. The forecast ignores the May holidays: the history clearly shows sales spikes during promotions (Black Friday, Valentine's Day), but the forecast line stays flat on May 8-10 and May 23-25. That's unrealistic: holidays should drive sales up, as they did before." Or, if the forecast is sound: "Reasonable. The forecast continues the upward trend of the history, keeps the weekly periodicity (weekend dips), and shows spikes on May 8-10 and May 23-25, which makes sense given the holidays."
Why It Works
The LLM's weakness: Models are bad at handling raw time-series numbers; they don't "feel" abstract sequences of digits. Ask an LLM to evaluate the series [120, 125, 130, 128, 133, 40, 145] and it may miss the anomaly (40) because there's no visual context.
The LLM's strength: Models are excellent with visual patterns and spatial relationships. Multimodal LLMs (GPT-4V, Claude) were trained on millions of charts, diagrams and schematics; they "see" trends, slopes and spikes as visual objects, not abstract numbers. To an LLM, a chart is like a photo of a cat: it recognizes shape patterns rather than computing pixels.
How the method uses this: The Forecast Critic turns time-series analysis into a computer vision task. Instead of "evaluate the sequence [1, 2, 3, 10, 4, 5]" → "look at the chart and tell me whether the red line looks like a logical continuation of the blue one." The LLM sees: the blue line rises at a 30° angle → the red one suddenly drops → visually illogical → unreasonable. It's the same mechanism that kicks in with "look at this infographic and explain what's wrong", just applied to forecasts.
Extra strength: The text context (promotions, holidays) overlays the visual pattern. The LLM connects "the text says 'sale on March 15'" + "the chart has a peak on March 15" → logical. Or "the text says 'sale on March 15'" + "the forecast is flat on March 15" → illogical, there should be a spike. This works because multimodal LLMs are trained to link visual and text signals: the same principle that lets them answer "what's in this photo?" or "find element X on this diagram."
Control knobs:
Strictness level: Add "flag only OBVIOUS errors" → the model gets more conservative, fewer false alarms. Remove that phrase → the model gets pickier and notices more nuances.
Context: The more events you list (holidays, promotions, seasonality) → the more accurate the assessment. But if the context is wrong → the model will err (say "there should be a spike" when there shouldn't).
Explanation format: Ask to "explain it for a non-specialist" → you get plain language. Ask to "explain it technically" → you get terms like "periodicity is broken."
Prompt Template
Assess whether this forecast looks reasonable.
On the chart:
- {color_1} line: {history_description} (for example: sales history for the last 90 days)
- {color_2} line: {forecast_description} (for example: forecast for the next 30 days)
Context (if any):
{Describe key events, seasonality, promotions: whatever affects the data}
Answer:
1. Reasonable or Unreasonable?
2. What in the forecast looks logical or, on the contrary, concerning?
3. If unreasonable, what mistake did the model make?
[attach the chart image]
What to fill in:
{color_1},{color_2}: the line colors on the chart (blue, red, green){history_description}: what the historical part shows (sales, traffic, a metric){forecast_description}: what the forecast shows (how many days ahead){context}: external factors: holidays, promotions, product launches, seasonality (summer/winter), known events (for example, "we launched Google Ads in March")
If there's no context, drop the "Context" section and the model will judge by the visual pattern alone.
Limitations
⚠️ Periodicity: LLMs are bad at spotting compressed/stretched cycles. If the forecast keeps a weekly pattern but shifts it to every 5 days instead of 7, the model may miss it. Visually it "looks like waves" → seems fine, even though the frequency is wrong.
⚠️ Missing spikes: The hardest task for an LLM is noticing the absence of an event. If the history had a peak every month and the forecast doesn't, the model catches that worse (F1 ≈ 0.75) than false spikes (F1 ≈ 0.85). The human eye is better at seeing "something's missing here."
⚠️ Newer models ≠ better: Claude 3.7 Sonnet beat Claude 4.0 Sonnet and Opus on this task. A newer/bigger model doesn't guarantee accuracy in visual forecast review.
⚠️ "Reasonableness" is subjective: The model relies on visual logic but doesn't know the business context. A forecast may look "weird" yet be correct because of hidden factors (for example, the forecasting model accounted for a supplier change that the LLM can't see).
Resources
The Forecast Critic: Leveraging Large Language Models for Poor Forecast Identification The study uses these datasets: synthetic time series (14 basis functions), the M5 Forecasting Competition (retail sales), and the Chronos model to generate forecasts.
Luke Bhan (UC San Diego), Hanyu Zhang, Andrew Gordon Wilson, Michael W. Mahoney, Chuck Arvin (Amazon).