InteractiveWorld

A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Overview

InteractiveWorld simulates the interaction that matters for wearable AI assistants: the assistant sees through a person's eyes and talks, the person decides and acts, and a stateful 3D world responds with the next first-person view.

The repository has two layers. InteractiveWorld is the reusable simulator: open a scene and drive the human with short commands such as [Grab] <cup> (1). InteractiveBench builds 120 frozen multi-turn tasks on top of it, where one model plays the assistant and another plays the human.

Repository structure

InteractiveWorld/
├── simulator/              InteractiveWorld simulator
│   ├── api.py              Python API: Simulator, Action, Observation, Score
│   ├── physical_world/     Isaac Sim scenes, human motion, manipulation, rendering, BEHAVIOR baking
│   ├── environment/        Persistent environment: reset, step, observe, native score
│   ├── interaction/        Human/Assistant turn protocol and prompts
│   ├── human_model/        SMPL+H skinning and the bundled grasp atlas
│   └── world_catalog/      44 admitted scenes and the BEHAVIOR asset manifest
├── benchmark/              InteractiveBench
│   ├── tasks/              120 frozen TaskPacks (6 categories × 20) and their manifest
│   ├── runner/             One-command evaluation: model endpoints, multi-GPU, reports, videos
│   ├── evaluation/         Release rubric, evidence and scoring
│   └── oracle_videos/      Integrity manifest of the 120 oracle videos
├── launch/                 Command-line entry points (prepare_assets, simulator, interactiveworld, ...)
├── examples/               Minimal Python programs
├── docs/                   Detailed guides
└── tests/                  Release tests

InteractiveWorld

1. Preparation

InteractiveWorld runs on Linux with an NVIDIA GPU (we test on A100 80 GB; one simulator worker uses about 4 GB of GPU memory) and needs about 100 GB of disk. You download four things; one script then prepares everything else.

Step 1 — Isaac Sim 5.1, OmniGibson and the BEHAVIOR-1K dataset. The official installer creates a conda environment named behavior and downloads the dataset to BEHAVIOR-1K/datasets.

Step 2 — InteractiveWorld.

Step 3 — Human model files. Register on the MANO and AMASS websites and download mano_v1_2.zip and the ACCAD SMPL-X N archive (ACCAD.tar.bz2) from the table above. Both are licensed for non-commercial research and cannot be redistributed.

Step 4 — Prepare the assets. This bakes the 6,831 BEHAVIOR objects used by the release, copies the OmniGibson materials, converts the SMPL+H model, extracts the walk clip, verifies every frozen checksum and writes an environment file.

2. Your first actions

Open a scene and give the human a sequence of commands. --video saves what the human sees (requires ffmpeg), and the JSON on stdout reports the outcome of every action.

A command is [Verb] followed by object keys, each with an instance number. Run the same command without --action to list the keys of a scene (initial_observation.objects). The human works out where to stand, how to walk there and how to grip the object.

3. Simulator basics

Discovery commands start instantly and do not launch Isaac Sim. The same simulator is available from Python. import simulator is lightweight; Simulator.launch starts Isaac Sim in a separate process, so your application keeps its own Python and CUDA environment.

InteractiveBench

InteractiveBench freezes 120 multi-turn tasks, 20 in each of six categories: Guidance, Planning, Accessibility, Decision, Prevention and Finding. The model under test is the Assistant: it sees only the human's first-person view and the dialogue. A second, fixed model is the Human: it follows the Assistant's guidance by issuing simulator actions. A task passes when every goal holds in the final world state.

1. Evaluate a model

One command selects the two models, the tasks and the GPUs. The Assistant and the Human can each be a local open-source model behind an OpenAI-compatible server (vLLM, Ollama) or a hosted API (openai, anthropic, gemini). API keys are read from the environment variable you name, never from the command line.

2. Add your own task

A task is one TaskPack JSON file: the scene and the objects to place (world), the final state that counts as success (goal), what the human did before asking (story.prologue), how the human reacts to guidance (runtime.human_policy) and an oracle solution (runtime.oracle_commands). The easiest start is to derive a new task from a bundled one.

Citation

@misc{wang2026interactiveworld,
  title         = {InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants},
  author        = {Wang, Qinsi and Wen, Wei and Zhou, Chong and Suri, Saksham and Liu, Zechun and Wu, Lemeng and Li, Hai Helen and Chen, Yiran and Krishnamoorthi, Raghuraman and Qian, Qi},
  year          = {2026},
  eprint        = {2610.33966},
  archivePrefix = {arXiv},
  primaryClass  = {cs.HC},
  url           = {https://arxiv.org/abs/2610.33966}
}

License

InteractiveWorld is released under the CC BY-NC 4.0 license. The BEHAVIOR-1K dataset, the SMPL+H/MANO body model and the AMASS motion data are distributed by their owners under their own licenses; obtain and use them according to those terms.