High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy
Authors: Lam Ngo, Huong Ha, Jeffrey Chan, Vu Nguyen, Hongyu Zhang
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our proposed method on various benchmark synthetic and real-world problems. The results demonstrate that our method outperforms existing state-of-the-art techniques. |
| Researcher Affiliation | Collaboration | Lam Ngo EMAIL RMIT University, Australia Huong Ha EMAIL RMIT University, Australia Jeffrey Chan EMAIL RMIT University, Australia Vu Nguyen EMAIL Amazon, Australia Hongyu Zhang EMAIL Chongqing University, China |
| Pseudocode | Yes | We illustrate the CMA-based meta-algorithm in Fig. 1 and the pseudocode in Algorithm 1. First, an initial search distribution N(m(0), (σ(0)) 2C(0)) is set (line 6), and a local region S(0) is computed based on this search distribution and the chosen BO optimizer bo_opt (line 9). |
| Open Source Code | Yes | The implementation of our method is available at https://github.com/Lam Ngo1/cma-meta-algorithm. |
| Open Datasets | Yes | We conduct experiments on eight synthetic and three real-world benchmark problems to evaluate all methods. For synthetic problems, we use Levy-100D, Alpine-100D, Rastrigin-100D, Ellipsoid-100D, Schaffer2-100D, Branin2-500D, and two modified versions, Shifted-Levy-100D and Shifted-Alpine-100D. For real-world problems, we use Half-cheetah-102D, Lasso DNA-180D and Rover-100D. These are the benchmark BO problems that used in related works including Wang et al. (2016; 2018); Eriksson et al. (2019); Nguyen et al. (2020); Wang et al. (2020); Eriksson & Poloczek (2021); Papenmeier et al. (2022); Song et al. (2022); Nguyen et al. (2022); Ziomek & Ammar (2023). For Half-cheetah-102D, we use the same implementation as described in Song et al. (2022), which parameterizes the Half-cheetah-v4 Mujoco environment from the Gym package3 into a 102D reinforcement learning (RL) problem. For Lasso DNA-180D, we use the implementation from the Python Lasso Bench library (Šehić et al., 2022) as in Papenmeier et al. (2022). For Rover-100D, we use the implementation provided by Wang et al. (2018). |
| Dataset Splits | Yes | Sample n0 initial data points D0 Latin hypercube All the methods are initialized with 20 initial data points and are run for 10 repeats with different random seeds. All experimental results are averaged over these 10 independent runs. |
| Hardware Specification | No | The first and second authors (L.N. & H.H.) would like to thank the Google Cloud Research Credits Program for the computing resources on this project. |
| Software Dependencies | No | All the developed CMA-based BO methods (CMA-BO, CMA-Tu RBO, CMA-BAx US) are implemented using GPy Torch (Gardner et al., 2018) as with Tu RBO and BAx US. All the Python-based methods are run with the same Python package versions. |
| Experiment Setup | Yes | We use Matérn 5/2 ARD kernels for the GPs in all methods. The input domains of all problems are scaled to have equal domain lengths in all dimensions as in Loshchilov & Hutter (2016). The output observations are normalized following a Normal distribution y N(0, 1). For the hyperparameters of the CMA strategy in all CMA-based BO and ES methods, we set them using the suggested values in Hansen (2016). Specifically, the population size λ is set to be 4 + 3 + ln d . The initial mean vector m(0) is selected by minimizing 20 initial data points following a Latin hypercube sampling (Jones, 2001). The covariance matrix C(0) is initialized with an identity matrix Id, and the initial step size σ(0) is set to 0.3(u l) where u, l denote the upper and lower bounds of the search domain X, i.e., X = [l, u]d. To ensure fair comparison between the CMA-based BO methods and the corresponding BO optimizers, we set the hyperparameters of the CMA-based BO methods to be the same as those of the corresponding BO optimizers. Specifically, for BO and CMA-BO, the hyperparameter settings of the GP and the TS acquisition function of these methods are the same. For Tu RBO and CMA-Tu RBO, we follow the same setting suggested by Tu RBO (Eriksson et al., 2019) to set the initial TR base side length L0, the maximum and minimum TR side lengths Lmax and Lmin, and the success and failure threshold τsucc and τfail. For BAx US and CMA-BAx US, we also follow the same setting suggested by BAx US (Papenmeier et al., 2022) to set the initial TR base side length L0, the maximum and minimum TR side lengths Lmax and Lmin, the success and failure thresholds τsucc and τfail, and the bin size b. |