Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training
Authors: Utku Ozbulak, Hyun Jung Lee, Beril Boga, Esla Timothy Anzaku, Ho-min Park, Arnout Van Messem, Wesley De Neve, Joris Vankerschaver
TMLR 2023 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this survey, we review a plethora of research efforts conducted on image-oriented SSL, providing a historic view and paying attention to best practices as well as useful software packages. While doing so, we discuss pretext tasks for image-based SSL, as well as techniques that are commonly used in image-based SSL. Lastly, to aid researchers who aim at contributing to image-focused SSL, we outline a number of promising research directions. ... For the majority of the discriminative SSL frameworks covered in this survey, we provide a comparison of model size to linear probing accuracy on Image Net in Figure 6. ... From Table 24 to Table 30, we provide benchmarks on Image Net-1K. In Table 31, we provide benchmarks on COCO. |
| Researcher Affiliation | Collaboration | Utku Ozbulak1,2 EMAIL Hyun Jung Lee1,2 EMAIL Beril Boga3 EMAIL Esla Timothy Anzaku1,2 EMAIL Homin Park1,2 EMAIL Arnout Van Messem4 EMAIL Wesley De Neve1,2 EMAIL Joris Vankerschaver1,2 EMAIL 1Ghent University, Belgium 2Ghent University Global Campus, South Korea 3BSH Hausgeräte GmbH, Germany 4University of Liège, Belgium |
| Pseudocode | No | The paper describes various SSL frameworks and their methodologies in text, accompanied by illustrative figures (e.g., Figure 2, 3, 4, 5) but does not include any explicitly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | No | The paper states that 'many SSL frameworks come with implementations and trained models that are publicly available' and provides tables (17-19) detailing the availability of official implementations for the surveyed frameworks. It also lists 'useful SSL repositories' (Table 20). However, there is no explicit statement that the source code for the methodology described in this particular survey paper (i.e., its own analysis or contributions) is provided or made publicly available. |
| Open Datasets | Yes | For the majority of the discriminative SSL frameworks covered in this survey, we provide a comparison of model size to linear probing accuracy on Image Net in Figure 6. ... From Table 24 to Table 30, we provide benchmarks on Image Net-1K. In Table 31, we provide benchmarks on COCO. |
| Dataset Splits | Yes | In the SSL literature, three types of evaluations are commonly used: (i) fine-tuning the entire model, (ii) linear evaluation, also known as linear probing or linear protocol, and (iii) K-nearest neighbors (KNN) evaluation using extracted features. ... For the majority of the discriminative SSL frameworks covered in this survey, we provide a comparison of model size to linear probing accuracy on Image Net in Figure 6. ... From Table 24 to Table 30, we provide benchmarks on Image Net-1K. In Table 31, we provide benchmarks on COCO. |
| Hardware Specification | No | The paper discusses the computational demands of Self-Supervised Learning models in general, citing examples like 'the training of Mo Co-v3 with a vision transformer backbone requires approximately 625 TPU days' from another work (Chen et al., 2021). However, it does not explicitly state the specific hardware details (e.g., GPU/CPU models, memory amounts) used for its own analysis or to generate its benchmarking results and figures. |
| Software Dependencies | No | The paper lists various Github repositories related to Self-Supervised Learning (Table 20) and discusses the availability of implementations for the surveyed frameworks. However, it does not explicitly provide specific software dependencies (e.g., library names with version numbers like Python 3.8, PyTorch 1.9) that were used to conduct its own analysis, compile benchmarks, or generate figures. |
| Experiment Setup | No | The paper describes general evaluation methods for Self-Supervised Learning models, such as 'linear evaluation' and 'fine-tuning,' mentioning that 'the linear layer that maps the features to classes is trained separately' and 'the entire model is (re)trained on the training dataset'. However, it does not provide specific hyperparameters (e.g., learning rates, batch sizes, optimizers, epochs) or detailed training configurations for these evaluations as performed by the authors of this paper to generate their benchmark results or figures. |