Dependency-aware Maximum Likelihood Estimation for Active Learning
Authors: Beyza Kalkanli, Tales Imbiriba, Stratis Ioannidis, Deniz Erdogmus, Jennifer Dy
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate the impact of including the dependency term during model parameter estimation for the active learning process on the model s prediction performance by testing it on datasets consisting of images, texts, or features; to have diverse experiments, we used datasets with different sizes and complexities. We use eight classification datasets: Iris (Fisher, 1988), Mnist (Deng, 2012), Fashion Mnist (FMnist) (Xiao et al., 2017), SVHN (Netzer et al., 2011), Reuters-21578 (Reuters) (Lewis, 1997), Emnist-Letters (Emnist) (Cohen et al., 2017), Cifar-10 (Krizhevsky et al., 2009), and Tiny Image Net (Le & Yang, 2015). [...] Table 2 demonstrates the test accuracy results of using DMLE vs. IMLE after 100 collected samples across seven datasets, employing entropy as the acquisition function, with four different sample selection schemes [...]. |
| Researcher Affiliation | Academia | Beyza Kalkanlı EMAIL Department of Electrical and Computer Engineering Northeastern University; Tales Imbiriba EMAIL Department of Computer Science University of Massachusetts Boston; Stratis Ioannidis EMAIL Department of Electrical and Computer Engineering Northeastern University; Deniz Erdoğmuş EMAIL Department of Electrical and Computer Engineering Northeastern University; Jennifer Dy EMAIL Department of Electrical and Computer Engineering Northeastern University. |
| Pseudocode | Yes | Algorithm 1 Dependency-aware Maximum Likelihood Estimation for Active Learning |
| Open Source Code | No | The paper does not provide an explicit statement about open-sourcing its own code, a specific repository link for the methodology, or clear indication of code in supplementary materials. It only mentions using Keras for implementation, which is a third-party tool. |
| Open Datasets | Yes | We use eight classification datasets: Iris (Fisher, 1988), Mnist (Deng, 2012), Fashion Mnist (FMnist) (Xiao et al., 2017), SVHN (Netzer et al., 2011), Reuters-21578 (Reuters) (Lewis, 1997), Emnist-Letters (Emnist) (Cohen et al., 2017), Cifar-10 (Krizhevsky et al., 2009), and Tiny Image Net (Le & Yang, 2015). [...] All datasets we use are publicly available. Our experiments include Mnist and SVHN datasets both distributed under the GNU General Public License. Reuters-21578 collection resides with Reuters Ltd. where Reuters Ltd. and Carnegie Group, Inc. allows free distribution of the dataset for research purposes only. Additionally, we utilize Emnist, Fashion-Mnist, Cifar-10, and Iris all with MIT License. We download Mnist, Reuters-21578, Fashion-Mnist, Cifar-10, and Iris from Keras datasets repository; Emnist from Tensorflow datasets repository, and SVHN from http://ufldl.stanford.edu/housenumbers/. |
| Dataset Splits | Yes | We report the test accuracy on separate test sets for each dataset where the test sets for Mnist and FMnist consist of 384 samples while Iris has 30 samples, SVHN has 510 samples, Reuters has 450 samples, Emnist has 336 samples, Cifar-10 has 386 samples, and Tiny Image Net has 296 samples. Additionally, we have separate validation sets of similar size to the test sets of each dataset. |
| Hardware Specification | Yes | We utilized two different computing resources during the experiments. For the experiments with smaller models and datasets like Iris, we used an Intel(R) Core(TM) i9-10900KF processor paired with RTX 3090 GPU. For larger datasets and more complex models, which constitute the rest, we used an internal cluster with an Nvidia Tesla K80 GPU. |
| Software Dependencies | No | We use Keras for the implementation of the neural networks (Chollet et al., 2015). For all models, we use Adam optimizer (Kingma & Ba, 2017) with a learning rate of 0.001. The paper mentions Keras and Adam optimizer but does not provide specific version numbers for Keras or other key software components. |
| Experiment Setup | Yes | Aiming a fair comparison, at each active learning cycle, for all combinations, we use the same number of epochs, the same hyperparameter combinations, and the same acquisition functions. We present results with β = 1 for the stochastic batch selection as practiced and suggested by Kirsch et al. (2021) which can be important for the model and acquisition performance. Different values for β might be explored further for better results. For all models, we use Adam optimizer (Kingma & Ba, 2017) with a learning rate of 0.001. |