Boosting Data-Driven Mirror Descent with Randomization, Equivariance, and Acceleration
Authors: Hong Ye Tan, Subhadip Mukherjee, Junqi Tang, Carola-Bibiane Schönlieb
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide theoretical convergence guarantees for our schemes under standard assumptions and demonstrate their effectiveness in various computational imaging and machine learning applications such as image inpainting, and the training of support vector machines and deep neural networks. In this section, we demonstrate the performance of the proposed algorithms on image reconstruction and various training tasks. |
| Researcher Affiliation | Academia | Hong Ye Tan EMAIL Department of Applied Mathematics and Theoretical Physics University of Cambridge, United Kingdom Subhadip Mukherjee EMAIL Department of Electronics and Electrical Communication Engineering Indian Institute of Technology, Kharagpur, India Junqi Tang EMAIL School of Mathematics University of Birmingham, United Kingdom Carola-Bibiane Schönlieb EMAIL Department of Applied Mathematics and Theoretical Physics University of Cambridge, United Kingdom |
| Pseudocode | Yes | Algorithm 1 Accelerated mirror descent (AMD) with approximate mirror updates; Algorithm 2 Stochastic mirror descent (SMD) (Nemirovski et al., 2009); Algorithm 3 Accelerated stochastic mirror descent (ASMD) (Xu et al., 2018, Alg. 2); Algorithm 4 Learned AMD (LAMD); Algorithm 5 Learned SMD (LSMD); Algorithm 6 Learned ASMD (LASMD); Algorithm 7 Learned AMD with minibatched gradients (LAMD*); Algorithm 8 Reptile (Nichol et al., 2018, Alg. 1) |
| Open Source Code | No | The paper mentions 'All implementations were done in Py Torch' and refers to a third-party library 'Deep Inversion Validation Library (DIVal)' with a GitHub link, but it does not provide concrete access to the authors' own source code for the methodologies described in this paper. |
| Open Datasets | Yes | We consider image denoising on an ellipse data set, with images of size 128 128, where we apply LAMD, LSMD and LASMD with pre-trained mirror maps given in Tan et al. (2023a). ... The training data used for the pre-trained mirror map are noisy ellipse phantoms in X-ray CT, generated using the Deep Inversion Validation Library (DIVal) (Wang & Zhou, 2006; Leuschner et al., 2019). The phantoms were generated in the same manner as in Tan et al. (2023a). We additionally consider the image inpainting setting in Tan et al. (2023b). The images to reconstruct are STL-10 images. We consider training a support vector machine (SVM) over a dataset with many samples. Similarly to Tan et al. (2023b), the task is to classify the digits 4 and 9 from the MNIST data set. We utilize the Omniglot dataset, consisting of 20 instances each of 1623 characters (Lake et al., 2011). |
| Dataset Splits | Yes | Using the pre-trained neural network, we train SVMs using the training fold of MNIST using the 4 and 9 classes, which consists of 11791 images. For testing the generalization accuracy of these methods, we use the 4 and 9 classes in the testing fold, which consists of 1991 images. The characters are split into training and testing sets of 1200 and 423 characters respectively as in Finn et al. (2017). |
| Hardware Specification | Yes | All implementations were done in Py Torch, and training was done on Quadro RTX 6000 GPUs with 24GB of memory (Paszke et al., 2019). |
| Software Dependencies | No | The paper states 'All implementations were done in Py Torch' but does not provide a specific version number for PyTorch or other software dependencies. |
| Experiment Setup | Yes | A grid search was used to find the baseline optimizer step-sizes, chosen to minimize the loss at iteration 100. The maximum unrolled iterations is set to N = 10, and the LMD step-sizes are also learned for these iterations (Tan et al., 2023b). SGD with learning rate 1 10 1, and Adam with learning rate 1 10 2, where other parameters are kept as default. The MNIST batch-size is taken to be 500, resulting in unrolling N = 360 steps. |