BirdDreams: A Driving World Model
NVIDIA's BirdDreams is a driving world model that jointly predicts perspective video and bird's-eye view scenes from a single camera image.
Deep dive
- Joint generation enables both views to inform each other, improving spatial consistency.
- BEV provides a geometric interface for ego control, allowing the model to follow specific meter-based trajectories.
- The model outperforms perspective-only baselines in generating reactive traffic behaviors, such as vehicles adjusting lanes to avoid aggressive maneuvers.
Decoder
- BEV (Bird's-Eye View): A top-down representation of a scene, commonly used in autonomous driving to provide a scale-accurate map of road and vehicle positions.
- Ego Motion: The movement of the vehicle itself (the camera car) through the environment.
- Autoregressive: A method where the model predicts future states based on a sequence of preceding inputs.
Original article
BirdDreamsA driving world model with precise ego control and traffic that can react.
From one camera image, BirdDreams jointly predicts driving video and a bird’s-eye view (BEV) of the scene. With an optional ego trajectory in meters, it follows that path while predicting how surrounding traffic moves.
More plausible surrounding-vehicle motion
We compare against an OmniDreams-based baseline trained to predict RGB without surrounding-vehicle-state inputs. For this comparison, we use HD-map-conditioned variants with the same starting image, ego trajectory, and road map.
Which video has more realistic vehicle behavior?
- 57% BirdDreams
- 24% Baseline
- 19% About the same
25 participants · 375 pairwise judgments. The OmniDreams-based and BirdDreams variants use HD maps but no surrounding-vehicle-state inputs, with the same starting scene and ego maneuver.
When we specify the ego’s path
We set the ego trajectory to make the ego cut in aggressively or drive through a red light. BirdDreams predicts how surrounding vehicles react.
When the model predicts ego motion too
Without an input ego trajectory, BirdDreams predicts how both the ego and surrounding vehicles move. Here, both the ego and a crossing car brake to avoid a collision.
Precisely following ego trajectories specified in meters
Following small steering changes
Both models receive the same ego trajectory, taken from the recorded drive on the left. Compare the camera car’s position relative to the lane markings as it steers slightly right.
With ego-action input
BirdDreams follows ego-action inputs more closely than the baseline. ADE: 35% lower error in following the input trajectory.
Without ego-action input
Without action inputs, BirdDreams generates trajectories closer to recorded driving than the baseline. minADE: 21% lower trajectory error.
Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as accurate control of the ego trajectory. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV depicts roads and vehicles as simple shapes at a fixed metric scale, providing a direct view of their positions and motion. Perspective video, in turn, retains visual cues that the BEV omits. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition for generation. Trained with BEV targets rendered from scene annotations, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs at inference. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without trajectory conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.
Acknowledgements
We are deeply grateful to Sanja Fidler, Zan Gojcic, Xuanchi Ren, Kevin Xie, Yifan Lu, and Zian Wang for their valuable discussions, insightful feedback, and generous support throughout this work. We also thank Yuxuan Zhang for developing the HD-map-conditioned variant of OmniDreams.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi