Introducing GWM Worlds 2

Interactive worlds generated in real time: continuous 720p video at 24 fps and audio at 48,000 Hz, responding to your inputs as you explore. Extending GWM Worlds, showcased in December last year, GWM Worlds 2 supports generated audio and rich subject and scene control.

In December, we introduced GWM Worlds, a world model for real-time environment simulation. GWM Worlds extended our research efforts around real-time video generation, and excelled at maintaining spatial consistency across long sequences of movement.

Today, we're sharing GWM Worlds 2, the next iteration of our research to create interactive environments. Built on top of our foundational audio-video generation model, GWM Worlds 2 turns high-fidelity video and audio generation into real-time interactive simulation. You define the environment, subjects, visual style, physical rules and ambience. Once inside, you steer the world with text actions addressed to any subject or to the scene itself, alongside continuous camera motion. A text action can be a sword slash, a spoken reply, a room flooding with water, a dust storm rolling in or anything else you can put into words.

The world continues from each new input instead of following a fixed clip or script, so sessions have no preset length. This creates a foundation for interactive entertainment, virtual characters, robotics and embodied-agent simulation and generative design and interfaces.

WorldPrompt

Rich control over a world requires an input representation that generalizes across use cases. We base ours on the insight that a world can be split into two kinds of state: what persists and what changes over time. We formalize this as WorldPrompt, a format with two complementary layers.

• Persistent world context:
  • A genesis prompt that includes a scene description covering the environment, its layout, materials, lighting and ambient sound; the subjects that can participate in events, with their attributes; and the laws, reusable conventions that govern behavior, from gravity and collision to character abilities and the camera perspective.
  • A first frame to ground the generation visually.
• A timestamped event stream: Actions that describe movement, gestures, object interactions, speech and sound. Each action is a free-form text prompt with start and end timestamps, addressed to a subject or to the scene itself. Multiple actions can overlap. Camera input that represents a per-frame stream of viewpoint translation and rotation.

Finetuning the Model on WorldPrompt

We take our foundational audio-video generation model and finetune it on the WorldPrompt format. Below are some results showing the model follows the WorldPrompt structure correctly.

Real-Time Generation

The finetuned model is still too slow, and cannot yet generate autoregressively because it is still bidirectional. Therefore, we post-train the bidirectional model into a real-time autoregressive model, GWM Worlds 2. Unlike the bidirectional model, GWM Worlds 2 is also not restricted to a fixed duration and can generate indefinitely.

The same WorldPrompt structure applies here as well, but now the world responds as you play: each clip below was generated live at 24 fps, with a user steering the world through text actions addressed to subjects and the scene, plus continuous camera motion. Recurring actions can be bound to keys for fast play.

The model supports both first-person and third-person navigation like walking, driving and riding through a world. The camera and the subject can be controlled independently.

Subjects and the scene can be directed to take arbitrary actions: movement like running, climbing and leaping; object interactions like switching a lamp on or changing its color; and scene events like a theater stage erupting in fire.

Characters can hold conversations and control the voice, tone and language of the speech, e.g. NPC–player dialogue or a vlogger talking to the camera. Lip movement and delivery are generated to match.

Model Overview

GWM Worlds 2 is an autoregressive diffusion video and audio model generating 720p video at 24 fps and audio at 48,000 Hz. GWM Worlds 2 conditions each AR step on three kinds of context: a global context (the genesis prompt and first frame), the current frame's inputs (camera input and the text actions that span the current frame) and the past generated frames, cached in a sliding window. The video and audio decoders are causal and run with a cache for faster decoding.

Using the Model

The model itself takes the WorldPrompt, with its persistent and event-based context, and outputs video and audio. There are various ways we could imagine using such a model:

• Ahead of time. The user (possibly with the assistance of an LLM) authors the timestamped event stream once at the start, and the model generates the whole video and audio with it. Uses: filmmaking, advertising.
• Turn-based. The generation runs until the user has to make a decision. The user then chooses an action, which an LLM could put into the timestamped event stream format. The model proceeds to generate until the next decision point. Uses: visual novels, interactive film.
• Real-time. The model continuously generates video and audio, and the user can take actions that immediately affect the generation stream. This is the most challenging scenario, as it requires text to be available with low latency, reacting immediately to the world. Making users type actions would be too slow, and even VLMs cannot react within tens of milliseconds. Uses: games, interactive experiences.

Demo

For some videos in the real-time generation section we used the ahead-of-time method to author rich actions beforehand without having to come up with them on the fly. For most of them we used our demo, which implements the real-time method.

Continuing Play from a Video

Instead of starting from an image, we can also prefill the generation with an existing video. Here is an example where we prefill with a generated eight second video and then continue playing from it. Note that the model keeps the environment and audio consistent with the input video.

Agentic Control

We can let agents control the actions of the different subjects. Here is an example where we let an agent control both the character (to move and attack) and the environment (to direct its lighting). Thus, GWM Worlds 2 can be used as a simulated environment for evaluations of agents, and for training them in diverse environments.

Multiplayer

We can also let different users control different subjects. This can be used to implement multiplayer experiences by letting each user control their own character, or by having one user control the world.

World Authoring

In our demo, sessions are started from presets that contain the first frame, the genesis prompt and the possible actions for the different subjects, bound to different keys. Instead of manually typing out these prompts and coming up with actions, we allow users to create their own presets quickly by generating everything with the assistance of an LLM.

Limitations

GWM Worlds 2 is a research preview, and real-time generation still trades fidelity for speed. Difficult camera inputs such as very quick rotations can cause details, textures and geometry to degrade. Long-term memory is also imperfect, and the model doesn't support image references beyond the first frame or prefilled video and audio.

While free-form text makes the control surface general, fully leveraging it can require an external real-time harness that tracks the state of the world and generates actions on the fly, for example the NPC dialogue in an NPC–player interaction.

Real-time video generation is still in its earliest stages, and the constraints outlined in this post will be solved with continued research.