RecreationWorld

Scalable and Verifiable Environments for Hybrid Computer-Use Agents

Overview

RecreationWorld is a five-platform framework for studying and improving hybrid computer-use agents that autonomously interleave GUI exploration, implementation with coding tools, and visual verification of their own running artifacts. By framing recreation around a running reference as an executable oracle, it turns open-source applications into scalable, verifiable training experience that transfers beyond recreation, while RecreationBench provides 250 held-out tasks with reference-grounded programmatic and visual evaluation.

Recreation workflow

Each task gives the agent a high-level request, interactive access to a running reference, and both GUI-control and software-development tools. The agent decides when to explore the reference, implement source code, build and launch its candidate, inspect the result, and revise it—forming a recurring explore–implement–verify loop rather than a fixed sequence of stages.

The final candidate is evaluated by a frozen suite of reference-validated programmatic and visual assertions. Scoring depends on observable behavior rather than source-level similarity, so implementations remain free to use different languages, frameworks, and architectures.

RecreationBench Results

Scores are macro-averaged within each platform and then equally weighted across platforms. Average is the unweighted mean of Prog and VLM; estimated costs assume 90% cache reads.

Model	Prog (%)	VLM (%)	Average (%)	Prog ≥90% (% apps)	Prog =100% (% apps)	Estimated cost (USD/task)
GPT-6 Astra	58.19	57.92	58.06	17.60	2.80	115.80
Claude Opus 5	45.99	42.34	44.16	5.53	0.80	117.17
GPT-5.6 Sol	40.63	43.49	42.06	5.20	0.40	25.46
Grok 4.6	39.02	34.45	36.73	1.60	0.00	12.58
Qwen3.8-Max-0902	35.53	34.07	34.80	2.00	0.00	38.76
Kimi K3	32.07	30.74	31.41	2.00	0.00	63.10
Claude Opus 4.8	32.40	29.81	31.10	1.60	0.40	69.16
GLM-5.3	26.30	22.46	24.38	2.00	0.00	70.98
Gemini 3.7 Flash	24.91	17.34	21.12	2.40	0.00	—
Qwen3.7-Plus	9.18	9.12	9.15	0.00	0.00	1.23

Quickstart

Install uv, then run from the repository root:

uv sync
uv run rb run --help

A scored run also needs a matching frozen task bundle, a prepared execution environment, and model and judge endpoints. Choose a platform for setup and batch runs:

Platform	Tasks	Evaluation interface
Ubuntu	50	AT-SPI
macOS	50	AXUIElement
Windows	50	UI Automation
Android	50	UiAutomator
Web	50	Browser assertions

To verify the checkout, run these offline checks; they do not require benchmark data or credentials:

uv run python scripts/release/smoke_providers.py
uv run python scripts/release/smoke_runtime.py

Citation

@misc{qwen2026recreationworld,
      title={RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents},
      author={Shuai Bai and Jiayong Deng and Sicheng Fan and Yikun Fu and Chang Gao and Xuhao Hu and Mianqiu Huang and Yizhen Jiang and Yuheng Jing and Dehui Kong and Keliang Li and Ning Li and Wanli Li and Dayiheng Liu and Dunjie Lu and Changwei Luo and Que Shen and Zheyuan Wang and Zijian Wang and Jie Wu and Gao Wu and Zhihui Xie and Rui Xie and Haiyang Xu and An Yang and Jiakang Yuan and Yanming Zhang and Jiajun Zhang and Xi Zhang and Zhenru Zhang and Zhuo Zhen and Mingkang Zhu and Bowen Zhou},
      year={2026},
      eprint={2609.22000},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.22000},
}