Probabilistic preference learning with the Mallows rank model
Authors: Valeria Vitelli, Øystein Sørensen, Marta Crispino, Arnoldo Frigessi, Elja Arjas
JMLR 2017 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We test our approach using several experimental and benchmark data sets. We report results from extensive simulation experiments carried out in several different parameter settings, to investigate if our algorithm provides correct posterior inferences. The experiments considered in this section illustrate the use of our approach in various situations corresponding to different data structures. |
| Researcher Affiliation | Academia | Valeria Vitelli EMAIL Oslo Centre for Biostatistics and Epidemiology, Department of Biostatistics, University of Oslo, P.O.Box 1122 Blindern, NO-0317, Oslo, Norway Øystein Sørensen EMAIL Oslo Centre for Biostatistics and Epidemiology, Department of Biostatistics, University of Oslo, P.O.Box 1122 Blindern, NO-0317, Oslo, Norway Marta Crispino EMAIL Department of Decision Sciences, Bocconi University, via R ontgen 1, 20100, Milan, Italy Arnoldo Frigessi EMAIL Oslo Centre for Biostatistics and Epidemiology, University of Oslo and Oslo University Hospital, P.O.Box 1122 Blindern, NO-0317, Oslo, Norway Elja Arjas EMAIL Oslo Centre for Biostatistics and Epidemiology, Department of Biostatistics, University of Oslo, P.O.Box 1122 Blindern, NO-0317, Oslo, Norway |
| Pseudocode | Yes | The above described MCMC algorithm is summarized as Algorithm 1 of Appendix B. The MCMC algorithm described above and used in the case of partial rankings is given in Algorithm 3 of Appendix B. The pseudo-code of the clustering algorithm is sketched in Algorithm 2 of Appendix B. The MCMC algorithm for clustering based on partial rankings or pairwise preferences is sketched in Algorithm 4 of Appendix B. Algorithm 5: MCMC Sampler for full rankings. Algorithm 6: MCMC Sampler for full rankings with clusters. |
| Open Source Code | No | All methods presented have been implemented in C++, and run efficiently on a desktop computer, with the exception of the Movielens experiment, which needed to be run on a cluster. The paper mentions implementation in C++ but does not provide specific access details, such as a repository link or a statement about code availability in supplementary materials, for their own developed methodology. |
| Open Datasets | Yes | Here we consider pair comparison data (Section 4.2)... Sushi Data... using the benchmark data set of sushi preferences collected across Japan (Kamishima, 2003)... Movielens Data... The Movielens data set1 contains movie ratings from 6040 users. 1. www.grouplens.org/datasets/. |
| Dataset Splits | Yes | We generated the data with N = 200, n = 15, C = 3, α1, ..., αC = 4, ψ1, ..., ψC = 50, obtaining the true Rj,true for every assessor. Then, we assigned to each assessor j a different number, Tj Trunc Poiss(λT , Tmax), of pair comparisons... Before converting ratings to preferences, we discarded for each user j one of the rated movies at random. Then, we randomly selected one of the other movies rated by the same user, and used it to create a pairwise preference involving the discarded movie. This preference was then not used for inference. |
| Hardware Specification | No | The computations shown here were performed on a desktop computer, and the off-line computation with K = 106 samples for n = 10 took less than 15 minutes... K = 106 samples for n = 100 were obtained on a 64-cores computing cluster in 12 minutes. Computing times for the simulations, performed on a laptop computer... All methods presented have been implemented in C++, and run efficiently on a desktop computer, with the exception of the Movielens experiment, which needed to be run on a cluster. No specific hardware details like GPU/CPU models or memory amounts are provided for the desktop, laptop, or cluster environments. |
| Software Dependencies | No | All methods presented have been implemented in C++, and run efficiently on a desktop computer... The R packages sets (Meyer and Hornik, 2009) and relations (Meyer and Hornik, 2014) efficiently compute the transitive closure. While programming language (C++) and R packages are mentioned, specific version numbers for C++ compilers, libraries, or the R packages used in the experiments are not provided. |
| Experiment Setup | Yes | Some model parameters are kept fixed in the various cases: αjump = 10, σα = 0.15, and L = n/5 (for the tuning of the two latter parameters, see the simulation study in the Supplementary Material, Section 1). In the analysis, we run Algorithm 4 of Appendix B on these data, using the exact partition function, for 105 iterations (of which 104 were for burn-in). We set L = 40, σα = 0.95, λ = 0.05 and αjump = 1... We set L = 2, σα = 0.1, λ = 0.1 and αjump = 100. We set L = 1, σα = 0.1, λ = 0.1 and αjump = 100. We set: L = 20, σα = 0.05, αjump = 10 and λ = 0.1, after some tuning. |