Hyperparameter Optimization via Sequential Uniform Designs
Authors: Zebin Yang, Aijun Zhang
JMLR 2021 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments are conducted on both global optimization tasks and HPO applications. The numerical results show that the proposed Seq UD strategy outperforms benchmark HPO methods, and it can be therefore a promising and competitive alternative to existing Auto ML tools. ... A large amount of global optimization experiments and HPO experiments are presented in Section 5 and Section 6, respectively. |
| Researcher Affiliation | Academia | Zebin Yang EMAIL Aijun Zhang EMAIL Department of Statistics and Actuarial Science University of Hong Kong Pokfulam Road, Hong Kong |
| Pseudocode | Yes | A formal description of the Aug UD construction is given in Algorithm 1, which requires input of multiple pre-specified parameters. ... A summary of the above procedures is provided in Algorithm 2. |
| Open Source Code | Yes | Two open-source Python packages are developed, including the Py Uni DOE package that employs efficient C++ code for generating uniform designs, and the Seq UD package that implements the proposed Seq UD method for HPO tasks. ... 1. https://github.com/Self Explain ML/Py Uni DOE ... 3. https://github.com/Self Explain ML/Seq UD |
| Open Datasets | Yes | We consider 20 regression and 20 binary classification data sets obtained from the UCI machine learning repository and Open ML platform, in which we select the data with moderate features and sample sizes. |
| Dataset Splits | Yes | For each data, we split 50% of the data samples for training, and the remaining 50% is used for testing. Five-fold crossvalidation (CV) performance in the training set is employed as the optimization target. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU or CPU models, processor types, or memory amounts used for running the experiments. It only refers to 'computing resources' in a general sense. |
| Software Dependencies | No | The paper mentions several software components like 'scikit-learn', 'xgboost', 'py DOE', 'sobol seq', 'Hyperopt', 'Spearmint', and 'SMAC3'. However, it does not specify any version numbers for these software dependencies, which is required for reproducibility. |
| Experiment Setup | Yes | In Seq UD, we set the number of runs and levels per stage as n = q = 15 when s ≤ 5; otherwise, we use n = q = 25 for higher-dimensional tasks. ... For SVM, The popular Gaussian kernel is fixed, and we tune the two continuous hyperparameters, i.e., the kernel parameter and penalty parameter. They are optimized in the base-2 log scale within [2^-16, 2^6] and [2^-6, 2^16], respectively. The number of components in the Nystroem transformer is fixed to 500. The initial learning rate is set to 0.01, and we use the adaptive optimizer to adjust the learning rate during optimization. ... For XGBoost, Eight important hyperparameters in XGBoost are introduced, including booster (categorical; gbtree or gblinear ), maximum tree depth (integer-valued; within the range [1, 8]), number of estimators (integer-valued; within the range [100, 500]), ratio of features in each tree (continuous; within the range [0.5, 1]), learning rate (continuous; the base-10 log scale within the range [10^-5, 10^0]), minimum loss reduction (continuous; the base-10 log scale within the range [10^-5, 10^0]), ℓ1-regularization (continuous; the base-10 log scale within the range [10^-5, 10^0]) and ℓ2-regularization (continuous; the base-10 log scale within the range [10^-5, 10^0]). |