Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
Set-to-Sequence Methods in Machine Learning: A Review
Authors: Mateusz Jurewicz, Leon Derczynski
JAIR 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This paper provides a comprehensive introduction to the field as well as an overview of important machine learning methods tackling both of these key challenges, with a detailed qualitative comparison of selected model architectures. The listed set encoding methods can be applied to a wide spectrum of tasks. As a result, their performance has been tested on a variety of datasets, often in subtly different settings, which prevents direct comparison. Both Pointer Networks and the Read-Process-and-Write (RPW) method have been tested on the simple task of sorting a set of five floating point numbers between 0 and 1 (Vinyals et al., 2016). The Pointer Network achieved an accuracy of 0.90 compared to 0.94 reached by the RPW. |
| Researcher Affiliation | Collaboration | Mateusz Jurewicz EMAIL, EMAIL Department of Computer Science IT University of Copenhagen, 2300 Copenhagen, Denmark Tjek A/S, 1408 Copenhagen, Denmark. Leon Derczynski EMAIL Department of Computer Science IT University of Copenhagen, 2300 Copenhagen, Denmark. |
| Pseudocode | Yes | The attention mechanism in the decoder block is as follows: zi j = v Ttanh(W1ei + W2dj) for i (1, . . . , n) (2) aj = softmax(zj) for j (1, . . . , n) (3) P(yi|y1, . . . , yi 1, x) = aj for j (1, . . . , n) (4)...The Process Block evolves the permutation invariant representation of the input set by repeating the following steps t times: qt = LSTM(q t 1) (5) zi t = f(mi, qt) (6) ai t = exp(zi t) P j exp(zj t ) (7) i ai t mi (8) q t = qt, rt (9)...Transformer Attention(Q, K, V ) = softmax(QK )V (11)... |
| Open Source Code | No | The paper does not provide any specific links to source code repositories, nor does it state that the code for the methodologies discussed in this review is openly available. It is a review paper summarizing existing work. |
| Open Datasets | Yes | However, in the case of the Set Transformer experiments were performed on sets of handwritten characters from the Omniglot dataset (Lake et al., 2019), in the case of Deep Sets on the MNIST8m hand-written digits (Loosli et al., 2007) and in the case of Janossy Pooling on simple integer sets. Particularly the Model Net40 dataset (Wu et al., 2015) has been used to test four of the mentioned models. Both the Set Transformer and Deep Sets methods have been tested on the task of set anomaly detection, specifically by way of the Celeb A dataset (Liu et al., 2015). The FSPool technique s performance has been compared to the Janossy Pooling method through a visual question answering task, employing the CLEVR (Johnson et al., 2017) dataset. Finally, the performance of Deep Sets, Set Transformer and Rep Set has been compared on the task of graph classification through the 5 datasets proposed by Kersting et al. (2016). |
| Dataset Splits | No | The paper discusses various datasets and their performance metrics from other studies but does not provide specific train/test/validation split percentages, sample counts, or detailed splitting methodologies within its own text. It summarizes results from other research without detailing the exact data partitioning used in those external experiments. |
| Hardware Specification | No | The paper is a review of existing methods and does not describe any specific experimental hardware used for its own work or for the implementation of the reviewed methods. No GPU models, CPU types, or other hardware specifications are mentioned. |
| Software Dependencies | No | The paper is a review and does not detail the software dependencies (e.g., library names with version numbers) used for implementing or evaluating the discussed methodologies. It focuses on the theoretical and architectural aspects of machine learning models. |
| Experiment Setup | No | As a review paper, this document synthesizes information from other research but does not present its own experimental setup details, including hyperparameter values, training configurations, or system-level settings. These details would typically be found in original research papers that conduct experiments. |