4DSynth Controllable Procedural World Synthesis for Dynamic Embodied Simulation

Language, a blueprint mask, or a single photograph becomes an editable dynamic 3D environment: a procedural scene with planned actors and a validated camera, exported to OpenUSD for physics simulation and embodied evaluation.

Paper · arXiv PDF Code BibTeX

System overview: a director instruction on the left is compiled into a furnished living room with two animated characters and a planned camera path; panels on the right show three camera views over time, the four input conditions, three scene types, and human and camera trajectory variants — **Figure 1.** A director instruction specifies the environment, the actors and their actions, and the camera intent. 4DSynth builds an editable scene by native procedural generation, text or mask conditioning, or real-to-sim reconstruction from one photograph, then plans actor and camera motion against the final geometry. Right: camera views over time, the four input conditions, scene types, and trajectory control. Swipe sideways to pan the figure.

Abstract

4DSynth is a procedural authoring system that converts language descriptions, blueprint masks, or a single RGB image into editable dynamic 3D environments. Four scene-realization routes, native indoor and outdoor generation, layout-conditioned synthesis, and single-image real-to-sim, share a Stage representation of final geometry, walkability, and scene content. Against this common geometry the system measures animated character assets, plans collision-aware actor tracks, validates camera motion, and exports an OpenUSD stage for physics simulation, so scene structure, motion, and physical annotations stay explicit and editable rather than purely visual. Downstream, 4DSynth-Nav turns 122 generated physics-enabled scenes into 333 validated tasks with animated human obstacles, on which Qwen3-VL-30B-A3B-Thinking reaches 13.2% success and 22.4% subtask progress, exposing failures in blind-start exploration, multi-stage goal pursuit, and collision avoidance.

Pipeline diagram. User conditions (text prompt, optional actor identities, mask, image) enter a two-tier execution pipeline: tier 1 scene construction with Parse, Scene, and Finalize stages; tier 2 4D realization with Animation, Nav, and Camera stages; published artifacts are rendered video, physics export, and navigation tasks; a persistent WorldState records every intermediate contract — **Figure 2.** One connected pipeline behind every route. Tier 1 builds the scene contract and finalizes the Stage; tier 2 plans and validates motion against it; a persistent WorldState records every intermediate contract. Swipe sideways to pan the figure.

Results

Four ways in, one editable 4D world

Each panel shows the input to one route and its generated result. All four routes share the same editable scene representation, on which actors and cameras are planned over time.

Still render inside the generated home: a lavender living room with a grey sofa, a wall mirror, a side table with a lamp, and a green shuttered window — **Native indoor**text → editable multi-room home

A mannequin character walks across a mossy meadow with rocks and a shoreline in the distance — **Native outdoor**text → animated natural environment

Cutaway render of the compiled multi-room apartment with furnished bedrooms, kitchen, bathrooms, living room, and balcony — **Text / mask**blueprint → controllable multi-room scene · explore it below

Annotated input mask for a multi-room apartment: room boundaries, doors, windows, and oriented object footprints — **Text / mask**blueprint → controllable multi-room scene · explore it below

The photographed living room compiled as procedural assets with aligned materials, rendered from the recovered camera — **Real-to-sim**photo → aligned procedural scene

Input photograph of a living room with two grey sofas, orange cushions, a round black coffee table, and blinded windows — **Real-to-sim**photo → aligned procedural scene

Synchronized 4D state · text/mask route actor + camera over time

A mannequin character walks across the generated living room while a planned camera tracks it

video unavailable · download mp4

walkable furniture actor root camera

t 0.00 s · actor and camera synchronized

4D, not static 3D. The rendered sequence and its scene state share one timeline: One person walks through the living room. Use a wide tracking camera that follows the person and keeps the room layout visible. The map visualizes walkable space, scene objects, actor motion, and camera motion. Drag the map or filmstrip to inspect any moment.

Overhead rendering of the indoor scene with the measured actor-root trajectory in orange, the solved camera trajectory dashed in blue, and a purple marker for the fixed-root dancing actor — **Figure 7.** Overhead renderings with the measured actor-root trajectory (solid orange) and the solved camera trajectory (dashed blue), each paired with four frames sampled at the numbered times; the purple indoor marker is the fixed-root dancing actor.

Indoor frame t0: two characters in the living room, one walking toward the camera — **Figure 7.** Overhead renderings with the measured actor-root trajectory (solid orange) and the solved camera trajectory (dashed blue), each paired with four frames sampled at the numbered times; the purple indoor marker is the fixed-root dancing actor.

Executable world

Explicit geometry you can drive, exported to OpenUSD

Explore a generated multi-room scene directly in the browser while an animated pedestrian follows its planned route. Alongside it, the pipeline’s OpenUSD exports are shown in NVIDIA Isaac Sim with geometry, materials, character motion, and physical scene structure preserved.

Manual control · animated 4D browser preview ready

Cutaway render of the compiled multi-room apartment, shown until the interactive scene is loaded

Living Room distance 0.00 m

W A S D or arrow keys · animated pedestrian

Control a Carter robot through the generated apartment while an animated pedestrian follows a collision-aware route. Movement respects the exported scene geometry, and the live map shows the rooms, both agents, and the route you have travelled.

An animated person and NVIDIA Carter robot in the compiled real-to-sim living room rendered in Isaac Sim — **Animated PhysX replay.** Carter navigates a real-to-sim room while a USD-Skel character remains animated with physical collision geometry in Isaac Sim. The scene is compiled from this one photograph.

The input photograph of the beige living room that was compiled into this scene — **Animated PhysX replay.** Carter navigates a real-to-sim room while a USD-Skel character remains animated with physical collision geometry in Isaac Sim. The scene is compiled from this one photograph.

The exported native-indoor home rendered in Isaac Sim: a cutaway view of the living room with a purple accent wall, bookshelves, sofas, and two animated characters — **Native-indoor export in Isaac Sim.** The generated multi-room home is opened from its OpenUSD export with scene appearance, physical structure, and animated characters preserved.

4DSynth-Nav

World generation becomes task generation

Navigation and pick-and-place tasks are instantiated automatically from 122 physics-enabled indoor scenes generated by the pipeline and rendered in Isaac Sim 4.5, each with animated characters as dynamic obstacles that do not react to the agent; 333 of 345 candidates pass every physical gate. Tiers: L2, one target starting out of view; L3, two phases with the first target visible; L4, both.

122: generated physics-enabled scenes
333: validated tasks across 113 scenes
13.2%: success, Qwen3-VL-30B-A3B-Thinking
33.3%: success, Gemini 3.1 Pro

Success rate (SR) by tier; overall SR, subtask progress (SP), final goal distance (GD), and recorded contacts per episode on all 333 tasks.

Model	L2 SR	L3 SR	L4 SR	All SR	All SP	GD (m)	Coll.
Qwen3-VL-30B-A3B-Thinking	16.8	18.2	4.5	13.2	22.4	4.27	39.2
Gemini 3.1 Pro	57.5	23.6	18.2	33.3	40.7	2.68	19.0

Blind-start exploration (L2) produces the largest gap. Partial progress without completion and frequent contacts reveal persistent weaknesses in multi-stage behavior and collision avoidance.

Top-down plot of a failed L4 episode: the agent path loops around the room from the start marker with move-forward, turn-left, and turn-right segments, ending 1.95 m from the book stack goal after 140 steps and 47 collisions; the large plant goal is never reached — **Figure 8.** A failed L4 episode, “Pick up the book stack and bring it to the large plant”, with two runners as obstacles: repeated local turns rather than progress toward either goal.

Paper

4DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation

arXiv:2608.26947, 2026.

arXiv PDF Code

Scene generation builds on Infinigen and Infinigen Indoors; single-image perception uses Depth Anything 3, SAM 3, and SAM 3D; simulation and benchmark rendering use NVIDIA Isaac Sim 4.5. Source code is released under the BSD 3-Clause License; third-party dependencies and user-supplied runtime assets remain governed by their own terms.

@misc{4dsynth2026,
  title         = {4DSynth: Controllable Procedural World Synthesis
                   for Dynamic Embodied Simulation},
  year          = {2026},
  eprint        = {2608.26947},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.26947}
}