SOO-Bench: Benchmarks for Evaluating the Stability of Offline Black-Box Optimization
Authors: Hong Qian, Yiyi Zhu, Xiang Shu, Shuo Liu, Yaolin Wen, Xin An, Huakang LU, Aimin Zhou, Ke Tang, Yang Yu
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, baseline and state-of-the-art algorithms are tested and analyzed on SOO-Bench. Hopefully, SOO-Bench is expected to serve as a catalyst for the rapid developments of more novel and stable offline optimization methods. |
| Researcher Affiliation | Collaboration | Hong Qian, Yiyi Zhu, Xiang Shu, Shuo Liu, Yaolin Wen, Xin An, Huakang Lu, Aimin Zhou East China Normal University, China EMAIL, EMAIL Ke Tang Southern University of Science and Technology, China EMAIL Yang Yu Nanjing University, China Polixir Technologies, China EMAIL |
| Pseudocode | Yes | Algorithm 1 Offline Bayesian Optimization Trabucco et al. (2022) ... Algorithm 2 Covariance Matrix Adaptation Evolution Strategy (CMA-ES) Hansen (2006) ... Algorithm 3 Accumulative Risk Controlled Offline Optimization (ARCOO) Lu et al. (2023) ... Algorithm 4 Tri-mentoring for Offline Model-based Optimization Chen et al. (2023) ... Algorithm 5 Autofocused Model-based Optimization (Autofocusing Cb AS) Brookes et al. (2019) ... Algorithm 6 Tri-Training Data-Driven Evolutionary Algorithm (TTDDEA) Huang et al. (2021) ... Algorithm 7 Offline Data-Driven Optimization at Scale: A Cooperative Coevolutionary Approach (CCDDEA) Gong et al. (2023) ... Algorithm 8 Constrained Accumulative Risk Controlled Offline Optimization (CARCOO) ... Algorithm 9 Data-Driven Evolutionary Optimization with Penalty Function (DDEA-PF) Huang & Wang (2021b) ... Algorithm 10 Constrained Conservative Objective Models for Offline Optimization (CCOMs) |
| Open Source Code | Yes | The code is available at https: //github.com/zhuyiyi-123/SOO-Bench. |
| Open Datasets | Yes | GTOPX: Space Mission Optimization (Schlueter et al., 2021). ... The license for this dataset is GNU General Public License. ... CEC Task: Industrial and Design Optimization (Kumar et al., 2020). ... The license for this dataset is CC-BY 4.0 License. ... PROTEIN: DNA Sequence Optimization (Trabucco et al., 2022). ... You can access the data and code by visiting the URL https: //github.com/brandontrabucco/design-bench. ... The license for this dataset is MIT License. |
| Dataset Splits | Yes | In this paper, we select the middle 50% of the data (i.e., m% n% = 50%) to construct a simulated dataset as a reasonable baseline without leveraging any prior knowledge. Furthermore, when constructing the constraint dataset, we set the ratio of satisfying constraints to not satisfying constraints as 2 : 3. |
| Hardware Specification | Yes | The computing resources required for the research described in this paper are relatively modest, requiring only a single Nvidia Ge Force RTX 3090 GPU. |
| Software Dependencies | No | The paper mentions software like Bo Torch and various algorithm implementations from GitHub repositories but does not provide specific version numbers for these software dependencies. For example, 'We employ the quasi-expected Improvement (q EI) acquisition function within the Bo Torch framework (Balandat et al., 2019)' lacks a Bo Torch version. |
| Experiment Setup | Yes | We conduct a total of T = 150 optimization steps. ... CMA-ES: ... with the parameter σ set to 0.5. ... Autofocusing Cb AS: ... we have set this threshold to 0.9. ... ARCOO: ... initial momentum set at 0.2, ... sampling by 64 steps of Langevin dynamics. ... Tri-mentoring: ... number of neighborhood samples to 10. To maintain consistency, we standardized the number of optimization steps to 100 across all experiments. |