TL;DR
Visual ICL (in-context learning, learning from examples right in the prompt) for models that work with images (VLMs) means this: before your question, you show a few "image → correct answer" pairs. The model switches from "what this kind of object usually looks like" to "what's actually visible in this image." The examples should be chosen so the answers are balanced, with atypical cases mixed in alongside typical ones.
The headline finding: when a model gets an unusual image wrong, it often can see what's needed, but habit beats its eyes. Shown a three-legged peacock, it says "2 legs," because birds usually have two. That's not a vision error but an error of "deferring to what it knows." Sixteen examples gave large models a boost of roughly +15 to +27 pp on atypical images, while accuracy on ordinary images barely changed.
The method is single-step: in one request you include 8–16 "image → answer" pairs from the same task, then ask your question. One important condition: the model has to be able to perform the operation in question. If the capability isn't there (for example, counting 4–5 objects in a complex scene), examples barely help.
Want this for your own case? Apply this study to your task: AI adapts its method to what you are working on and writes the prompt.
Method Blueprint
STEP 1: Define the task + a closed list of answers → one line of instruction
STEP 2: Pick 8–16 examples (image → answer) from the SAME task
• answers are balanced (not 12 "red" and 2 "blue")
• typical and atypical cases mixed together
STEP 3: Add the query: a new image + the same question → a one-word answer from the list
(everything in one prompt, examples come BEFORE the question)
Example in Practice
Task: You moderate product listings on a marketplace (Amazon, Etsy). You need to automatically check the listed color of a product against the photo. The model keeps "correcting" reality: on a photo of blue bananas for a bakery photo shoot, it answers "yellow," because bananas are yellow. This is the method's sweet spot: the model distinguishes colors easily — only habit gets in the way.
Prompt:
You are checking product photos for a marketplace. Identify the main color
of the item AS IT APPEARS IN THIS PHOTO. Don't rely on what color
this kind of item usually is. Allowed answers: red, blue, yellow,
green, pink, purple, black, white.
Example 1: [photo of a pink banana]. Question: what color is the item? Answer: pink
Example 2: [photo of a yellow lemon]. Question: what color is the item? Answer: yellow
Example 3: [photo of a blue "Blue Willow" mug]. Question: what color is the item? Answer: blue
Example 4: [photo of a red cucumber]. Question: what color is the item? Answer: red
Example 5: [photo of a green tomato]. Question: what color is the item? Answer: green
Example 6: [photo of a black banana]. Question: what color is the item? Answer: black
... (up to 8–16 examples, answers balanced)
Now the task: [photo of a blue banana]. Question: what color is the item?
Answer with one word from the list.
Result: The model returns one word from the list. Expect answers on atypical photos (the blue banana) to match the real color more often than with a no-examples request. On ordinary photos (the yellow lemon), quality will barely change. If errors remain in your sample, check whether your examples are too uniform.
Why It Works
The weakness. The model learned from billions of images where peacocks have two legs and bananas are yellow. When an image says otherwise, two "answer paths" compete inside it. Path one: recognize the object and answer from habit. Path two: look at the pixels and count or identify. If the request gives no hint about which path is needed, habit usually wins.
The strength. The model is good at picking up a task's rule from examples. "Image → answer" pairs show it that a visual answer is what's needed here. According to the study, the model's attention shifts from the "peacock" tokens to the "legs" tokens. The examples act as a pointer, not as new knowledge.
How the method uses this. The examples shift the odds toward the "look at the image" path. Answer balance is needed so the gain can't be explained by the model simply learning to "answer X more often."
Control knobs: - Number of examples: 1–2 give almost nothing. Gains climb up to 16, and for large models up to 32. For simple cases, 4–8 are enough, which saves tokens and money. - Answer balance: equalize the classes. Otherwise the model will start guessing by frequency. - Example mix: include both typical and atypical cases. "Weird" examples alone can push the model the other way. - Instruction wording: require "a closed list of answers" and "answer by what you see, not by habit." That narrows the space and cuts down on rambling. - Model size: on small models (around 4–8B) the effect is weaker and less selective. They pick up the answer format more than the meaning of the examples.
Prompt Template
You are checking images. Task: {what_to_identify}.
Answer ONLY based on what is actually visible in this image.
Don't rely on how this kind of object usually looks.
Allowed answers: {answer_list}.
Example 1: {image_1}. Question: {question}. Answer: {answer_1}
Example 2: {image_2}. Question: {question}. Answer: {answer_2}
...
Example {N}: {image_N}. Question: {question}. Answer: {answer_N}
Now the task: {new_image}. Question: {question}.
The answer is one value from the list.
What to fill in:
- {what_to_identify}: a single operation (color, shape, count, direction, relative position).
- {answer_list}: a closed set of options, as small as possible.
- Examples: 8–16 from the SAME task. Answers balanced, atypical cases included.
- {question}: the exact same wording in every example and in the query.
🚀 Quick start — paste into a chat:
Here's a visual few-shot template for checking images. Adapt it to
my task: [your task]. Ask me questions to fill in the fields.
[paste the template above]
The LLM will ask what exactly to identify in the image, what the allowed answers are, and which typical and atypical cases you run into. It needs this to build the balanced example set and the closed answer list the method relies on.
Limitations
⚠️ A baseline capability is required: if the model is bad at the operation itself, examples barely help. Gains dropped on hard counting (4–5 objects), where the model has to rely on fragile vision. The method selects a skill the model already has; it doesn't create one.
⚠️ Small models: in smaller models the effect is weaker and less selective. They mostly pick up the answer format. In one case, with 16 examples, accuracy on atypical images fell below the no-examples level.
⚠️ A cost on ordinary cases: on typical images accuracy generally didn't suffer, but in counting it dropped by 6–7 pp. Check quality on both kinds of cases.
⚠️ A narrow setup: the tasks were closed-ended, with one correct answer from a short list. The conclusions don't transfer directly to open-ended image descriptions.
⚠️ Lab data: the images are mostly artificial, some generated and edited. The models are open ones (Qwen, InternVL, Gemma), not GPT, Claude, or Gemini in their chat versions. Test the effect on your own photos.
⚠️ The mechanistic part is out of reach: masking the "hinting" tokens in an image requires access to the model's internals. You can't reproduce that in a chat; only the examples trick applies.
How They Tested It
The idea was simple: if a model fails on an unusual image, either it can't look properly or habit is overriding its vision. The authors used in-prompt examples as a diagnostic tool. If examples fix the error, the vision was working all along.
They built a set of five tasks: counting animal legs, animal color, road-sign shape, object direction, and spatial position. Each has "real" images (habit and fact agree) and "counterfactual" ones (habit and fact diverge, like a diamond-shaped stop sign or a three-legged peacock). Counterfactual images were made by targeted edits: only the relevant property was changed, everything else left as is. Two annotators checked the images blind.
They tested Qwen3-VL models from 2B to 235B and four other families (Qwen2.5-VL, Qwen3.5, InternVL3.5, Gemma-4) with 0, 1, 2, 4, 8, 16, and 32 examples, using 10 different example sets per task. The examples were balanced by answer. The key test: if examples only teach format, the gain will be the same on real and counterfactual images. The gain turned out to be uneven. For Qwen3-VL 235B it was +27.2 pp on counterfactual images versus +2.9 pp on real ones. For 32B: +15.1 versus −1.6. All families improved at ~30B, by +9.7 to +30.3 pp.
The authors then looked inside the models and showed that with examples, attention shifts from "habit" features to "real answer" features. Masking the "habit" tokens has a similar effect. A surprising finding that's also useful in practice: on small models and on hard counting, examples barely work. That means the method selects an existing ability rather than creating a new one.
Adaptations and Extrapolations
🔧 Technique: add an "anti-habit" instruction + examples → even less chance of a habit-driven answer
At the start of the prompt: "Rule: if the photo shows something that differs from the norm, answer based on the photo. The norm is not an argument." The paper didn't measure the effect of this phrase separately; it measured the examples. Use it as an add-on, not a replacement.
Extrapolation (not tested in the paper): in agents that look at screenshots (Claude Code, Cursor with images), you can put 3–6 "UI screenshot → what exactly is broken" pairs in the instructions file. Especially atypical ones: a button in an unusual spot, a nonstandard status color. Same hypothesis: we show the model it should look at the pixels, not at "how it usually is." Test on your own screenshots.
Resources
- Paper: In-Context Learning Can Help Vision-Language Models Overcome Training Priors
- Authors: Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci
- University: Princeton University
- Models covered: Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5, Gemma-4
- Analysis tools: logit lens, LatentLens (for the mechanistic part; readers don't need them)