qNBO: quasi-Newton Meets Bilevel Optimization
Authors: Sheng Fang, Yongjin Liu, Wei Yao, Chengming Yu, Jin Zhang
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Numerical experiments demonstrate the comparable or superior performance of the proposed algorithms in real-world learning tasks, including hyperparameter optimization, data hypercleaning, and few-shot meta-learning. In this section, we conduct numerical experiments to evaluate the performance of the q NBO algorithms in solving bilevel optimization problems. We first validate the theoretical convergence through experiments on a toy example, followed by an assessment of efficiency by comparing q NBO with its closest competitor, SHINE (Ramzi et al., 2022), as well as other bilevel optimization (BLO) algorithms |
| Researcher Affiliation | Academia | 1Public Health School, Fujian Medical University, 2School of Mathematics and Statistics, Fuzhou University, 3Center for Applied Mathematics of Fujian Province, 4Department of Mathematics, SUSTech, 5National Center for Applied Mathematics Shenzhen, SUSTech, 6School of Science, BUPT, 7CETC Key Laboratory of Smart City Modeling Simulation and Intelligent Technology, The Smart City Research Institute of CETC. Email domains like fzu.edu.cn, sustech.edu.cn, and bupt.cn also indicate academic affiliations. |
| Pseudocode | Yes | The details of q NBO are presented in Algorithm 1. Algorithm 1 q NBO : quasi-Newton Meets Bilevel Optimization. Algorithm 2 A(x, y0): gradient descent steps + q N steps for the LL problem. Algorithm 3 B(x, y, H0, d, Q). Algorithm 4 Cb(d, H0, {si, gi}t 1 i=0): Two-loop recursion for computing r = Htd when Ht is the inverse of the BFGS matrix. Algorithm 5 Cs(d, H0, {si, gi}t 1 i=0): Computing r = Htd when Ht is the inverse of an SR1 matrix. |
| Open Source Code | No | The specific values of tol and other parameters can be found in the code provided in the supplementary materials. https://github.com/sowmaster/esjacobians. SHINE-OPA: The maximum number of outer iterations is K = 30, maximum number of inner iterations is T = 1000. The initial matrix H0 = I, for more details see SHINE code. https://github.com/zaccharieramzi/hoag/tree/shine. (The first quote refers to supplementary materials for parameters, not a general code release statement, and the provided link is for PZOBO's experiment code, not qNBO's. The second quote refers to SHINE's code, not qNBO's.) |
| Open Datasets | Yes | We perform hyperparameter optimization for l2-regularized logistic regression on the 20News (Lang, 1995) and Real-sim (Chang & Lin) datasets... The 20news dataset (Lang, 1995)... the Real-sim dataset (Chang & Lin) contains... URL https: //www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. Accessed: 2021-0506. This subsection focuses on data hyper-cleaning for the MNIST (Deng, 2012) and Fashion MNIST (Xiao et al., 2017)... The mini Image Net dataset (Vinyals et al., 2016), derived from Image Net (Russakovsky et al., 2015)... The FC100 dataset (Oreshkin et al., 2018), generated from Krizhevsky & Hinton (2009)... the Omniglot dataset (Lake etor Lake et al., 2015)... |
| Dataset Splits | Yes | The 20news dataset (Lang, 1995) comprises a total of 18,846 samples with 130,107 features. It is divided into three subsets: the training set Dtrain has 16961 samples, the validation set Dval has 943 samples, and the test set Dtest has 942 samples. Similarly, the Real-sim dataset (Chang & Lin) contains 72,309 samples, each with 20,958 features. This dataset is also split into three parts: the training set Dtrain has 65078 samples, the validation set Dval has 3616 samples, and the test set Dtest has 3615 samples. The data is divided into four parts: training set, validation sets 1 and 2, and the test set. The training set comprises 50000 samples, while the validation and test sets contain 5000 and 10000 samples, respectively. Following Arnold et al. (2020), we partition the classes into 64 classes for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing. |
| Hardware Specification | Yes | All experiments are conducted on a server equipped with two NVIDIA A40 GPUs, an Intel(R) Xeon(R) Gold 6326 CPU, and 256 GB of RAM. |
| Software Dependencies | No | No explicit software dependencies with version numbers are provided in the paper. |
| Experiment Setup | Yes | The initial point for all algorithms is (x0, y0) = (2e, 2e) where e denotes the vector of all ones. BOME: The maximum number of outer iterations is K = 5000, the number of inner iterations is T = 100, the inner step size is α = 0.1, the outer step size is ξ = 0.1... (followed by similar detailed parameters for other algorithms throughout Appendix C). |