Visual prompt engineering for video models
Editing a task's image to look photorealistic (visual prompt engineering, VIPE) reliably boosts video models' visual reasoning, often more than text prompting or extra test-time sampling.
It shows a cheap, automatable lever for improving video-model reasoning and reveals that abstract-looking benchmarks may be systematically underestimating models' true reasoning ability.
Robert Geirhos · Yuxuan Li · Thaddäus Wiedemer · Neha Kalibhat · Zi Wang · Mani Malek · Oyvind Tafjord · Kevin Swersky · Been Kim · Priyank Jaini — meet the researchers →
Two readings, equal authority
How to choose: The paper’s words is verbatim — use it when you need to quote, or to judge how they write. Plain language is a paraphrase written for comprehension — use it when you want the idea fast. Neither is a summary of the other; they are two doors into the same room.
“In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task (“Where does the ball land, after passing a set of obstacles?”), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page.”
The authors ask whether editing a task's input image, rather than its text prompt, can help video-generation models reason better, the way text prompt engineering helps language models. They find that turning sketch-style task images photorealistic ("visual prompt engineering," VIPE) reliably improves accuracy across several visual reasoning tasks, can beat both text-prompt tuning and extra test-time sampling, and reveals that video models have a built-in preference for realistic-looking scenes.
What this paper defines
Every definition below is the paper’s own sentence, with its locator. The plain gloss is a reading aid and is marked as one.
Visual prompt engineering (VIPE)
“Visual prompt engineering (VIPE) aims to improve downstream reasoning performance by replacing an original image 𝑥 𝑖 with a variant 𝑣𝑖∗ without altering the task’s underlying logic.”2. Visual prompt engineering (VIPE)
In plain terms: VIPE means editing a task's input image so a video model can reason about it more effectively, without changing what the task is actually asking.
Ideator
“Conditioned on an ideation prompt 𝑡ideate ∈ T and 𝑘 task samples, the ideator (say, a human or a language model) 𝐼 : (X × T ) 𝑘 × T → T describes a visual edit in natural language”2. Visual prompt engineering (VIPE), Step 1: Ideator
In plain terms: The ideator is the component (a person or a language model) that proposes, in words, what visual edit to try next.
Editor
“Let 𝐸 : X × T → X be a general image editor (in our case an image editing model; though in principle this could also be a human), conditioned on edit instructions”2. Visual prompt engineering (VIPE), Step 2: Editor
In plain terms: The editor is the image-editing model that actually carries out the proposed edit on the image.
Filter
“To select the highest quality variant, let 𝑆 : X × X → ℝ be a scoring function (e.g., a vision-language model) that evaluates a candidate’s quality and its faithfulness to the original prompt image”2. Visual prompt engineering (VIPE), Step 3: Filter (optional)
In plain terms: The filter is an optional scoring step, such as a vision-language model, that picks the best edited image out of several candidates.
Atomic Concept Edit (ACE)
“An atomic concept edit (ACE) is a simple edit that either adds, removes or replaces a single concept at a time.”4.2. Step-by-step edits: Atomic Concept Editing (ACE)
In plain terms: An ACE is one small, single-concept change made to an image or text prompt, used to explore edits systematically one step at a time.
Realism bias
“Video models clearly have a realism bias.”8. Why does visual prompt engineering help?
In plain terms: Realism bias is the pattern that video models generate more consistent, reliable outputs when the input scene looks photorealistic rather than abstract or synthetic.
What they actually did
Each step is a synthesis. Open any step to see the paper’s own sentence it was derived from, with its locator — so nothing here floats free of the source.
- Formalized VIPE as a three-stage pipeline: an ideator proposes an edit, an editor (image model) applies it, and an optional filter selects the best candidate.
Trace this step to the paper
“Visual prompt engineering (VIPE) aims to improve downstream reasoning performance by replacing an original image 𝑥 𝑖 with a variant 𝑣𝑖∗ without altering the task’s underlying logic.”2. Visual prompt engineering (VIPE)
- Converted the sketch-style VPCT ball-and-ramp dataset into a photorealistic version using an image editor with an author-written edit instruction, filtering among five candidates and manually verifying difficulty was preserved.
Trace this step to the paper
“Filtering is performed with Gemini 3.1 Pro [33], selecting the best out of 𝑚 = 5 proposal variants.”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
- Compared video model accuracy on VPCT before and after VIPE, scoring outputs with an autorater that detects which bucket the ball lands in.
Trace this step to the paper
“essentially the video model is asked to simulate the ball’s trajectory, and an autorater detects the bucket in which the ball lands.”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
- Ran an ablation with unnatural, non-photorealistic textures that still had 3D depth, to separate the effect of realism from the effect of mere 3D appearance.
Trace this step to the paper
“we performed an ablation where samples have completely unrealistic textures (=no realism), while still having depth and therefore “looking 3D”.”3. ... Ablation: unnatural textures
- Automated VIPE with freeform, open-loop ideation: a VLM proposes an image-edit instruction, an image editor applies it, and a VLM filter picks the best of several attempts, tested across six visual reasoning tasks with Veo 3.1.
Trace this step to the paper
“we instantiate the VIPE ideator with Gemini 3.1 Pro [33], the image editor as Nano Banana 2 (Gemini 3.1 Flash Image), and the filtering model again as Gemini 3.1 Pro”4.1. Freeform ideation by a VLM
- Automated VIPE with a second, more structured method (Atomic Concept Editing, ACE): a tree search of single-concept edits guided by autorater feedback, run on three of the six tasks.
Trace this step to the paper
“An atomic concept edit (ACE) is a simple edit that either adds, removes or replaces a single concept at a time.”4.2. Step-by-step edits: Atomic Concept Editing (ACE)
- Compared VIPE to test-time scaling via self-consistency (majority voting) on VPCT, and tested combining both.
Trace this step to the paper
“self-consistency scales just as well on top of the engineered visual prompts as on the original prompts, yielding a compounded 68.0% (+27.7 percentage points) with 20 test-time samples.”5. Can visual prompt engineering be used for test-time scaling?
- Directly compared visual prompt engineering against text prompt engineering on four tasks by adapting the same ideator to propose alternative text prompts instead of image edits.
Trace this step to the paper
“We adapt the prompt engineering ideator from Sec. 4.1 to propose alternative text prompts instead of image edits, keeping all source images from a baseline task fixed.”6. Is language or visual prompt engineering more effective for video models?
- Tested whether VIPE also helps when a native image-generation model (rather than a video model) is the reasoner, on VPCT, across image-output and text-output configurations.
Trace this step to the paper
“it is clear that VIPE does not systematically improve image model reasoning performance.”7. Does visual prompt engineering help native image generation models, too?
- Investigated the mechanism behind VIPE's benefit by step-wise increasing realism on a synthetic dataset and having humans rate scene consistency at each step.
Trace this step to the paper
“Human ratings of scene consistency improved from 0% (!) in the synthetic setting to 59% in the fully realistic setting.”8. Why does visual prompt engineering help?
Exactly what was run, and how
| Model | Developer | Temp | Effort / reasoning | Deployment | Other settings |
|---|---|---|---|---|---|
| Wan2.2 (TI2V and I2V checkpoints) | — | not reported | not reported | unstated | Optional prompt rewriter (Qwen2.5-Plus) was tested but disabled for reported numbers because it lowered VPCT performance. |
| Veo 3.1 | Google DeepMind | not reported | not reported | API | Vertex AI API; 720p output; 8-second video duration; cost USD 0.40/second (USD 3.20/video). Used as the reasoning video model for nearly all experiments. |
| Omni Flash | Google DeepMind | not reported | not reported | API | Run via the Vertex AI API alongside Gemini and Veo (per App. B). |
| Gemini 3.1 Pro | Google DeepMind | not reported | not reported | API | Used throughout as ideator, filter, and VLM autorater; also evaluated directly as a text-only VLM reasoner on VPCT (96% accuracy). |
| Nano Banana Pro (Gemini 3 Pro Image) | Google DeepMind | not reported | not reported | API | Used as the image editor for the main VPCT VIPE dataset and as an image-output reasoner in Sec. 7; image editing costs ~USD 0.005 per 720p input image and ~USD 0.067 per 720p output image (for Nano Banana 2). |
| Nano Banana 2 | Google DeepMind | not reported | not reported | API | The paper names the underlying Gemini version inconsistently: Sec. 4.1 identifies it as Gemini 3.1 Flash Image, while Sec. 7 and Table 1 identify the same 'Nano Banana 2' as Gemini 2.5 Flash Image. |
Source for Wan2.2 (TI2V and I2V checkpoints) settings
“TI2V is a 5B checkpoint that can perform both T2V (not used here) and I2V; while the I2V checkpoint is a MoE model with 14B active parameters [34]”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
Source for Veo 3.1 settings
“We run inference with Gemini, Veo and Omni models via the Vertex AI API. Images and videos are generated at 720p. Videos are generated with a duration of 8s.”B. Inference details
Source for Omni Flash settings
“Finally, Omni Flash [36] improves from 56.3% to 67.5% via VIPE.”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
Source for Gemini 3.1 Pro settings
“An updated version, Gemini 3.1 Pro, even scores 96% on this dataset in our own experiments.”C. VPCT: evaluation details
Source for Nano Banana Pro (Gemini 3 Pro Image) settings
“For VPCT, we instantiate the editor 𝐸 with Nano Banana Pro [32]”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
Source for Nano Banana 2 settings
“the image editor as Nano Banana 2 (Gemini 3.1 Flash Image)”4.1. Freeform ideation by a VLM
What they reported — and what they left out
The paper states which model plays which role (ideator, editor, filter, reasoner, autorater), gives some checkpoint sizes and per-call cost/API details, but reports no temperature, sampling, or reasoning-effort settings for any model, and inconsistently names the Gemini version behind 'Nano Banana 2' (Gemini 3.1 Flash Image in Sec. 4.1 vs. Gemini 2.5 Flash Image in Sec. 7/Table 1).
The numbers they report
Visual prompt engineering substantially improves Veo 3.1's physics-reasoning accuracy on VPCT.
41.3% to 59.3%
See it in the paper
“For Veo 3.1 [35], which already starts with higher baseline performance, the effect of visual prompt engineering is even more pronounced, taking it from 41.3% to 59.3%.”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
Omni Flash also improves on VPCT with VIPE.
56.3% to 67.5%
See it in the paper
“Finally, Omni Flash [36] improves from 56.3% to 67.5% via VIPE.”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
An ablation with unnatural (non-realistic) but still 3D-looking textures shows little benefit, indicating realism rather than 3D depth drives the improvement.
See it in the paper
“most models do not show a significant difference between the baseline and unnatural texture condition, suggesting that 3D alone is insufficient and it is indeed visual realism that drives the performance improvement.”3. ... Ablation: unnatural textures
Automated step-by-step (ACE) visual prompt engineering achieved a complete error reduction on the Sort 3 Numbers task.
100% reduction rate
See it in the paper
“ACE’s more structured edits yield more optimized per-sample variants, however, which lead to a 100% reduction rate on Sort 3 Numbers.”4.2. Step-by-step edits: Atomic Concept Editing (ACE)
Self-consistency (majority voting) test-time scaling alone improves Veo 3.1's VPCT accuracy, but by less than a single VIPE-engineered prompt.
41.3% to 50.0% (+8.7 percentage points) with 20 samples
See it in the paper
“self-consistency scales just as well on top of the engineered visual prompts as on the original prompts, yielding a compounded 68.0% (+27.7 percentage points) with 20 test-time samples.”5. Can visual prompt engineering be used for test-time scaling?
Combining VIPE with self-consistency compounds the accuracy gains further.
68.0% (+27.7 percentage points) with 20 test-time samples
See it in the paper
“self-consistency scales just as well on top of the engineered visual prompts as on the original prompts, yielding a compounded 68.0% (+27.7 percentage points) with 20 test-time samples.”5. Can visual prompt engineering be used for test-time scaling?
A VIPE variant is much cheaper to produce than an additional generated video.
1/8 the cost of a video
See it in the paper
“VIPE variant currently costs only 1/8 of a video,”5. Can visual prompt engineering be used for test-time scaling?
On Sort 3 Numbers, both the best text variant and the best image variant produced roughly a 20-fold accuracy improvement over baseline.
86% (best text) and 76% (best image) vs. 4% baseline
See it in the paper
“the best text variant and the best image variant each achieved roughly a 20-fold improvement (86% and 76%, vs. 4% without prompt engineering).”6. Is language or visual prompt engineering more effective for video models?
On VPCT, one particular image edit among 20 freeform proposals notably raised Veo 3.1 accuracy.
57% to 73%
See it in the paper
“a particular image edit among 𝑛 = 20 proposals as notable exception, raising Veo 3.1 accuracy from 57% to 73%.”6. Is language or visual prompt engineering more effective for video models?
For a native image-generation model (Nano Banana Pro) used as a reasoner, VIPE gives only a modest benefit.
50% (VIPE) vs. 45% (sketch) Pass@1, +5 percentage points
See it in the paper
“For NB Pro in image mode, VIPE yields only a modest +5 pp improvement in Pass@1 over raw sketches (50% vs. 45%).”G. Image model comparison
For a different native image-generation model (Nano Banana 2 / NB Flash) in image-output mode, VIPE actually hurts performance.
37% (sketch) to 21% (VIPE) Pass@1, -16 percentage points
See it in the paper
“for NB Flash in image mode, VIPE hurts; Pass@1 drops from 37% (sketch) to 21% (VIPE), a −16 pp degradation.”G. Image model comparison
For the same underlying model (NB Flash), switching from generating an image answer to generating a text answer substantially improves accuracy, showing the image-generation pathway itself is a bottleneck.
21% to 50% Pass@1 on VIPE inputs, +29 percentage points
See it in the paper
“On VIPE inputs, switching from image to text mode improves Pass@1 from 21% to 50% (+29 pp).”G. Image model comparison
A text-only VLM reasoner (Gemini 3.1 Pro) substantially outperforms an image-output reasoner (NB Pro) on the same VPCT sketch task.
96% (text-only VLM) vs. 50% (image-output), 46 percentage-point gap
See it in the paper
“Gemini 3.1 Pro (text-only VLM) achieves 96% accuracy on VPCT sketches (Tab. 2), while NB Pro (image mode) achieves only 50%, a 46 pp gap.”G. Image model comparison
A control condition with upside-down buckets shows vision-language models' physics understanding on VPCT is shallow.
accuracy drops to 3-4%
See it in the paper
“The result is striking: when buckets are turned upside-down, model accuracy drops down to a mere 3–4% (see Tab. 2 for accuracies).”C. ... Have VLMs solved physics reasoning on VPCT?
Human-rated scene consistency rises steadily as a synthetic dataset is edited step-by-step toward full photorealism.
0% to 59%
See it in the paper
“Human ratings of scene consistency improved from 0% (!) in the synthetic setting to 59% in the fully realistic setting.”8. Why does visual prompt engineering help?
What they assert, beside what they showed
Left is the claim in the paper’s own words. Right is the data offered for it. Where the two do not fully meet, a gold band names the distance.
Visual prompt engineering improves video reasoning performance across tasks.
“We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks.”
“For Veo 3.1 [35], which already starts with higher baseline performance, the effect of visual prompt engineering is even more pronounced, taking it from 41.3% to 59.3%.”
3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?Visual prompt engineering can be even more effective than text prompt engineering or test-time scaling.
“In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling.”
“a single test-time sample on the engineered visual prompts from Sec. 3 already achieves 59.3% (+18 percentage points).”
5. Can visual prompt engineering be used for test-time scaling?Video models systematically prefer photorealistic context over abstract inputs, so sketch-based benchmarks can underestimate model capability.
“We reveal that video models systematically prefer photorealistic context over abstract inputs: sketch-based benchmarks can significantly underestimate model capabilities.”
“Human ratings of scene consistency improved from 0% (!) in the synthetic setting to 59% in the fully realistic setting.”
8. Why does visual prompt engineering help?VIPE does not systematically help native image-generation models the way it helps video models.
“it is clear that VIPE does not systematically improve image model reasoning performance.”
“for NB Flash in image mode, VIPE hurts; Pass@1 drops from 37% (sketch) to 21% (VIPE), a −16 pp degradation.”
G. Image model comparisonVisual prompt engineering does not need human intuition; it can be successfully automated, whether by freeform VLM ideation or structured step-by-step edits.
“Visual prompt engineering does not need to rely on human intuition: it can be successfully automated in different ways, from freeform ideation with a VLM to structured, step-by-step concept edits.”
“Overall, we observe systematic improvements over the “no visual prompt engineering” baseline, for a similar final performance to freeform VIPE.”
4.2. Step-by-step edits: Atomic Concept Editing (ACE)It is realism specifically, not merely 3D-looking depth, that drives video models' improved reasoning under VIPE.
“most models do not show a significant difference between the baseline and unnatural texture condition, suggesting that 3D alone is insufficient and it is indeed visual realism that drives the performance improvement.”
“Top row: original sketch-like dataset samples (realism ✗ 3D ✗). Middle row: visual prompt engineering samples (realism ✓ 3D ✓). Bottom row: unnatural textures ablation (realism ✗ 3D ✓).”
Figure 2 captionVisual prompt engineering is a more cost-effective test-time-scaling strategy than simply generating more videos.
“Visual prompt engineering is a more cost-effective test-time scaling strategy than simply generating more videos; both approaches can be combined for best results.”
“VIPE variant currently costs only 1/8 of a video,”
5. Can visual prompt engineering be used for test-time scaling?How they frame it, and what they want next
Their framing
The authors present VIPE as the natural visual analogue of well-established text prompt engineering: a simple, cheap, and automatable lever that reliably helps video models. They lean on their realism-bias finding to argue that many existing abstract-looking benchmarks understate video models' true reasoning competence, while also candidly noting that the same realism bias is itself an undesirable brittleness reminiscent of early language models' prompt sensitivity.
Register: The paper is assertive about its central empirical claim (VIPE reliably improves reasoning) but explicitly hedges on mechanism, on generality to image-generation models, and on the future trajectory of the effect, and frames its own 'realism bias' explanation as a hypothesis the results support rather than a settled fact.
Where they hedge
“While far from perfect, their abilities are rapidly improving.”1. Introduction
“We expect that as image editing models become ever more faithful, filtering / verification steps will no longer be required.”3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?
“The biggest limitation, at the moment, is that the quality of a prompt engineering variant is highly dependent on the quality of the image editing model.”9. Discussion, Limitations
“Only time will tell, but if there’s anything to be learned from the history of language model development, it’s the lesson that a model can only ever be as good as its prompt.”9. Discussion, Outlook
What they say it means
- Benchmarks for video-model reasoning that use abstract or sketch-style visuals may be underestimating models' true reasoning competence.
the paper’s words
“reasoning performance on datasets that look abstract / synthetic (which is the case for most of these datasets) is often only a lower bound on the model’s true, underlying reasoning ability.”9. Discussion
- Visual instructions should be treated as an optimizable input space for video models, analogous to how text prompts are optimized for language models.
the paper’s words
“The success of visual prompt engineering points toward a shifting paradigm where visual instructions are treated as an optimizable space rather than fixed inputs, just like text prompts are carefully crafted and optimized to elicit better language model performance.”9. Discussion, Outlook
- The realism bias that makes VIPE work may itself be an undesirable shortcut or dataset bias that models absorbed from their training data, rather than a purely benign property.
the paper’s words
“realism bias might be characterized as an unwanted shortcut [43, 44] or dataset bias [45] that models pick up from their training data.”9. Discussion
What they call for next
- Stop evaluating video models on abstract, impoverished task versions whenever a more realistic version can be produced via visual prompt engineering.
the paper’s words
“we recommend to stop evaluating video models on abstract tasks when there’s the option to visually prompt-engineer a more realistic version of the same task.”9. Discussion
Limitations they state
“The biggest limitation, at the moment, is that the quality of a prompt engineering variant is highly dependent on the quality of the image editing model.”9. Discussion, Limitations
“We mitigated this risk through a filtering / quality control step where e.g. a VLM reviews (and possibly rejects) image edits.”9. Discussion, Limitations
“Another limitation is the potential cost overhead of visual prompt engineering.”9. Discussion, Limitations
Moves worth stealing
Ends each experimental section with a boxed, numbered 'Takeaway' sentence that distills the section's finding for skimmers.
“Takeaway 1 Similar to how text-based prompt engineering can improve language model performance on language tasks, visual prompt engineering can systematically improve video model performance on visual reasoning tasks.”
States the boundary conditions of its own preferred mechanism candidly, including where it predicts (and finds) the effect should NOT hold.
“But for image generation models that are already covering both representation spaces better, VIPE provides little benefit.”
Treats a null/negative result for its own method as diagnostic evidence supporting the paper's explanatory theory, rather than downplaying it.
“it is clear that VIPE does not systematically improve image model reasoning performance.”
Where else this leads
Same people
- Visual General Intelligence: A White Paper Google DeepMind
shares Robert Geirhos - ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation Google DeepMind
shares Zi Wang
Same territory
- Introducing the Conceptual Reasoning Index Anthropic
benchmarks - TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic
benchmarks - Project Pilot: Can AI control a drone? Anthropic
benchmarks
Published alongside it
The nearest publications in time, across all three labs.
- Discovering cryptographic weaknesses with Claude Anthropic
2026-07-28 - Scientific computing in the age of agentic AI OpenAI
2026-07-28 - How enabling two settings tripled our scores on the ARC-AGI-3 benchmark OpenAI
2026-07-29 - Ten advances in mathematics and theoretical computer science OpenAI
2026-08-01
What this page was built from
Extracted from the full arXiv preprint text including all appendices (App. A-H); the plain-text extraction interleaves some figure captions, chart axis labels, and two-column layout fragments non-linearly around Figs. 2-6, 9, and 17-19, so a few chart-only bar values could not be reliably attributed to specific conditions and were omitted rather than guessed.