Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
Authors: Junwei Zhou, Xueting Li, Lu Qi, Ming-Hsuan Yang
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion. Video and results can be viewed on our website: https://colezwhy.github.io/ layoutyour3d/ We evaluate our method with a proposed validation set, extensive experiments, and ablations to showcase the ability of Layout-Your-3D in rapid and accurate 3D generation. In addition, we conduct comparisons with current state-of-the-art text-to-3D methods, demonstrating the potential of Layout-Your-3D in compositional 3D generation, as well as customized design and editing. |
| Researcher Affiliation | Collaboration | Junwei Zhou1 Xueting Li2 Lu Qi3,4, Ming-Hsuan Yang5,6 1Huazhong University of Science and Technology 2NVIDIA 3Wuhan University 4Insta360 Research 5UC Merced 6Yonsei University |
| Pseudocode | No | The paper describes its methodology in Section 3, titled "Layout-Your-3D", with subsections like "Coarse 3D Generation Stage" and "Disentangled Refinement Stage". While it provides detailed steps and mathematical formulations, it does not include any explicitly labeled pseudocode or algorithm blocks in a structured format. |
| Open Source Code | Yes | Video and results can be viewed on our website: https://colezwhy.github.io/ layoutyour3d/ We implement Layout-Your-3D based on threestudio (Guo et al., 2023) for a more integrated and readable system. |
| Open Datasets | No | To better validate Layout-Your-3D s ability in compositional 3D generation, we construct a validation set naming Compo20 for evaluation. The Compo20 consists of 20 compositional text prompts, each containing two or more instances with specific interactions. For each text prompt, we have a user to manually provide a 2D layout and also automatically generate a 2D layout using LLM-grounded Diffusion (Lian et al., 2023). Table 9: Detailed prompt list of our proposed Comp20 validation set, including overall text prompt YB and 2D layout set B. We provide both the user-given 2D layout set and the LLM-generated one. |
| Dataset Splits | No | To better validate Layout-Your-3D s ability in compositional 3D generation, we construct a validation set naming Compo20 for evaluation. The Compo20 consists of 20 compositional text prompts, each containing two or more instances with specific interactions. For each text prompt, we have a user to manually provide a 2D layout and also automatically generate a 2D layout using LLM-grounded Diffusion (Lian et al., 2023). |
| Hardware Specification | Yes | We conduct all of our experiments based on NVIDIA RTX6000 GPUs. |
| Software Dependencies | No | We implement Layout-Your-3D based on threestudio (Guo et al., 2023) for a more integrated and readable system. |
| Experiment Setup | Yes | In the Coarse 3D Generation Stage, we render the 3D instances at 10-degree intervals when estimating the coarse rotation ri. For the Disentangled Refinement Stage, in the collision-aware layout refinement step, we use the original hyper-parameter settings for the SSDS loss Lssds and optimize the layout for 400 iterations, with λf and λc set to 10.0 and 0.2, respectively. In the instance-wise refinement step, we utilize Deep Floyd (Shonenko et al., 2023) guidance with a total of 1500 iterations following the short refinement strategy (see Fig. 5). We set the timestep range to [0.10, 0.50] from step 0 to 800, and [0.02, 0.75] from step 800 to 1500. The parameters λs and λtv are set to 1.0 and 0.2, respectively. The extended instance-wise refinement strategy and additional implementation details are provided in Appendix B. |