Reliable Active Learning via Influence Functions
Authors: Meng Xia, Ricardo Henao
TMLR 2023 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results show that the proposed reliable active learning via influence functions (RALIF) can consistently outperform the random selection baseline as well as other existing and state-of-the art active learning approaches. Our experiments aim to demonstrate the reliability of RALIF on various selection scenarios. Below, we first introduce the baseline algorithms with which we compare. Then, we describe the different AL selection scenarios that we consider in our experiments, along with the model training details. Finally, we will present the results of our experiments, highlighting the performance of the proposed RALIF framework. |
| Researcher Affiliation | Academia | Meng Xia EMAIL Department of Electrical & Computer Engineering University of Duke Ricardo Henao EMAIL Department of Electrical & Computer Engineering University of Duke Biological, Environmental Sciences and Engineering Division King Abdullah University of Science and Technology (KAUST) |
| Pseudocode | Yes | Algorithm 1 RALIF: Reliable active learning via influence functions with truncated-IL selection. Input: L0: Initial labeled dataset, U0: initial unlabeled dataset, M: model, C: AL cycles, B: query set, P: candidate set, ϕMLc 1( ): convolutional encoder, and τ: truncation hyperparameter. 1: for c = 1, . . . , C do 2: Train model MLc 1 with labels set Lc 1 3: |P| = 0.1|Uc 1| |
| Open Source Code | Yes | Source code is available at https://github.com/mx41-m/Active-Learning.git |
| Open Datasets | Yes | In our experiments, we use three datasets: Cifar10 (Krizhevsky et al., 2009), Cifar100 (Krizhevsky et al., 2009), and i Naturalist (14 super classes) (Van Horn et al., 2018), representing simple, medium complexity, and imbalanced datasets, respectively. |
| Dataset Splits | Yes | We set different initial labeled dataset sizes for each selection scenario based on model capacity and the complexity of the dataset. For the CLIP model, which has shown excellent performance with a smaller training dataset size compared to other models, we set the initial labeled dataset size to 100 samples. This choice is made to ensure that the initial accuracy is not excessively high, allowing us to observe the effectiveness of the AL algorithms. For all other models, we set the initial labeled dataset size to 1000 samples for Cifar10 and 6000 samples for Cifar100 and i Naturalist. |
| Hardware Specification | No | The paper does not explicitly describe the hardware used to run its experiments. It mentions using 'deep neural networks' and 'DL architectures' but no specific GPU models, CPU models, or other hardware specifications. |
| Software Dependencies | No | The paper mentions 'stochastic gradient descent' and 'Cosine Annealing LR' and refers to a 'pytorch-cifar' repository, but does not provide specific version numbers for these software components or libraries. |
| Experiment Setup | Yes | We trained our models using stochastic gradient descent (Sutskever et al., 2013) with a weight decay of 10^-4 and a momentum of 0.9. The initial learning rate, learning rate scheduler, and training epochs were set based on the model capacity. For pretrained ResNet18 and CLIP, we employed Cosine Annealing LR (Loshchilov & Hutter, 2016) as the learning rate scheduler and set the initial learning rate to 0.01. The pretrained ResNet18 and CLIP models were trained for 200 and 100 epochs, respectively, the latter accounting for the excellent performance of the CLIP model. For ResNet18 (scratch), following the suggestions in He et al. (2016), we set the initial learning rate to 0.1 and decrease it by a factor of 10 at the 160-th epoch, for a total of 200 training epochs. |