ReGen: Generative Robot Simulation via Inverse Design
Authors: Peter (Phat) Nguyen, Johnson (Tsun-Hsuan) Wang, Zhang-Wei Hong, Erfan Aasi, Andrew Silva, Guy Rosman, Sertac Karaman, Daniela Rus
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we start in Section 3.2 with qualitative analysis that demonstrates examples and capabilities of the inverse design approach; in Section 3.3, we compare diversity of the generated simulation against existing benchmarks or simulations; in Section 3.4, we showcase Re Gen can be used for effective corner case generation; in Section 3.5, we demonstrate that the complexity of our generated environments allows us to create vision-language-action datasets that pose greater challenges to VLMs compared to existing datasets. |
| Researcher Affiliation | Collaboration | Phat Nguyen1, Tsun-Hsuan Wang1, Zhang-Wei Hong1, Erfan Aasi1, Andrew Silva2, Guy Rosman2, Sertac Karaman1, Daniela Rus1 1Massachusetts Institute of Technology 2Toyota Research Institute |
| Pseudocode | Yes | Algorithm 1 Graph Expansion Input: vbehavior Vevent Output: G G procedure POTENTIALLY CONNECT TO NODE(G,v, ...) vcandidate = node proposal(v, . . . ) G, edge added = edge construction(v, vcandidate, G) return G, vcandidate, edge added end procedure Initialize: G = {} G; G vbehavior while input degree(v G) < 1 do vevent {v | input degree(v G) < 1} G, v event, = potentially connect to node(G, vevent, . . . ) break whenever by the user. end while for {vevent | v G, v Vevent} do G, ventity, edge added = potentially connect to node(G, vevent, . . . ) if edge added then for All possible property types do G, vproperty, = potentially connect to node(G, ventity, . . . ) end for end if end for |
| Open Source Code | Yes | Please check our website here: https://regen-sim.github.io/. |
| Open Datasets | Yes | For autonomous driving, we utilize the CARLA simulator (Dosovitskiy et al., 2017). For manipulation tasks, we use the Py Bullet simulator (Coumans & Bai, 2016 2021) and selected 10 example reward functions from Robo Gen (Wang et al., 2024d). We compare our method against CARLA Leaderboard 2.0. scenarios, which cover traffic scenarios based on the NHTSA typology (National Highway Traffic Safety Administration, 2007). Chat Scene (Zhang et al., 2024) employs few-shot prompting of large language models to generate diverse safety-critical scenarios, while Drive LM (Sima et al., 2024) provides graph annotations from the Nu Scenes dataset (Caesar et al., 2020). We compare to Behavior-100 (Srivastava et al., 2021), RLBench (James et al., 2019), Gen Sim (Wang et al., 2024b), and Robo Gen (Wang et al., 2024d). |
| Dataset Splits | Yes | From each dataset, we randomly select 30 simulation traces, together with the ground-truth desired ego action for each one. |
| Hardware Specification | No | The paper does not explicitly mention specific hardware details such as GPU models, CPU models, or memory specifications used for running its experiments. It mentions using CARLA, Py Bullet, and Google's CP-SAT solver, but not the underlying hardware. |
| Software Dependencies | Yes | Both steps employs the gpt-4o-2024-08-06 model with parameters temp=0 and top-p=0. For autonomous driving, we utilize the CARLA simulator (Dosovitskiy et al., 2017). For manipulation tasks, we use the Py Bullet simulator (Coumans & Bai, 2016 2021). Parameter values (e.g., start locations, speeds) are determined using Google s CP-SAT solver (Perron & Furnon, 2019). We use Google s the CP-SAT solver to determine parameter values (e.g. object location), and employ Soft Actor Critic (SAC) (Haarnoja et al., 2018) for policy training. |
| Experiment Setup | Yes | Both steps employs the gpt-4o-2024-08-06 model with parameters temp=0 and top-p=0. Each behavior is defined in natural language and mapped to a predefined route, encoded in an XML file that defines the start location, target waypoints, and speed. The ego-vehicle follows this route generated by an A search algorithm, with longitudinal and lateral PID controllers for speed and steering control. To simulate other agents, we use motion primitives such as stationary(location) and drivingforward(start, speed). These agents also use A for path planning and PID controllers for low-level control. We use Google s the CP-SAT solver to determine parameter values (e.g. object location), and employ Soft Actor Critic (SAC) (Haarnoja et al., 2018) for policy training. |