TL;DR: Just as text-based prompt engineering systematically improves language model performance, visual prompt engineering (VIPE) serves as a simple, compute-efficient approach to elicit better visual reasoning performance from video models by transforming the input task image (e.g., making abstract sketches photorealistic).

Abstract

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models.

Task Examples

Below, we show examples for each task and split. We show a solution attempt on the original image (which often fails), a solution attempt on the most successful VIPE variant for the task and split, and other VIPE variants obtained by freeform VIPE or atomic concept edit (ACE) VIPE.

Physics VPCT

Predict which bucket the ball will land in after rolling down the ramps.

Text Prompt: "The ball moves down in a physically plausible way, sliding down the obstacles, and ends up in one of the three containers at the bottom. [...]"

Conjunctive Search

Locate the target object defined by a conjunction of features (e.g., color and shape).

Text Prompt: "The two blue balls begin to glow."

Connect the Dots

Connect same-colored dots.

Text Prompt: "Connect each pair of same-colored circles with a line."

Sort 3 Numbers

Sort three numbers in ascending order.

Text Prompt: "The numbers pop and disappear one at a time, in numeric order, starting from the smallest one."

Maze Solving

Find a path from start to finish in various maze sizes.

Text Prompt: "Create a 2D animation based on the provided image of a maze. The red circle slides smoothly along the white path, stopping perfectly on the green circle. [...]"

RushHour

Slide blocks to allow the red car to exit the grid.

Text Prompt: "[...] Plan the minimal sequence of moves needed to free the red car and allow it to exit the parking lot. Output: A video demonstrating the full solution to the puzzle, one move at a time."

Citation

@article{geirhos2026visual,
  title={Visual prompt engineering for video models},
  author={Geirhos, Robert and Li, Yuxuan and Wiedemer, Thadd{\"a}us and Kalibhat, Neha and Wang, Zi and Malek, Mani and Tafjord, {\O}yvind and Swersky, Kevin and Kim, Been and Jaini, Priyank},
  journal={arXiv preprint arXiv:2607.25537},
  year={2026}
}