On the Detection of Reviewer-Author Collusion Rings From Paper Bidding
Authors: Steven Jecmen, Nihar B Shah, Fei Fang, Leman Akoglu
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct an empirical analysis of two realistic conference bidding datasets and evaluate existing algorithms for fraud detection in other applications. We find that collusion rings can achieve considerable success at manipulating the paper assignment while remaining hidden from detection: for example, in one dataset, undetected colluders are able to achieve assignment to up to 30% of the papers authored by other colluders. |
| Researcher Affiliation | Academia | Steven Jecmen EMAIL Carnegie Mellon University Nihar B. Shah EMAIL Carnegie Mellon University Fei Fang EMAIL Carnegie Mellon University Leman Akoglu EMAIL Carnegie Mellon University |
| Pseudocode | No | The paper describes various algorithms like DSD, OQC-Greedy, OQC-Local, Tell Tail, Fraudar, and OQC-Specialized, and refers to their implementation or underlying principles (e.g., "we implement the LP-based exact algorithm from Charikar (2000)"). However, it does not provide any structured pseudocode blocks or algorithm figures for these methods within the paper. |
| Open Source Code | Yes | The code and data we use in our analysis are available online at https://github.com/sjecmen/peerreview-collusion-detection. |
| Open Datasets | Yes | The first dataset, which we refer to as AAMAS , contains a subset of the real bidding from the 2021 International Conference on Autonomous Agents and Multiagent Systems, an AI conference. This dataset is publicly available from Pref Lib (Mattei & Walsh, 2013)... Our second dataset, which we refer to as S2ORC , is the semi-synthetic dataset constructed and made publicly available by Wu et al. (2021). |
| Dataset Splits | No | The paper describes methods for injecting colluders and conducting trials for different parameter settings (e.g., "We repeat this procedure for 50 trials for each setting."). It does not, however, specify standard training, validation, or test dataset splits for the underlying AAMAS or S2ORC datasets. |
| Hardware Specification | Yes | All of our experiments in these sections were conducted on a server with 515 GB RAM and 112 CPUs. |
| Software Dependencies | No | The paper mentions various algorithms and methods (e.g., "LP-based exact algorithm from Charikar (2000)", "TPMS algorithm (Charlin & Zemel, 2013)") but does not provide specific version numbers for any software libraries, programming languages, or tools used in their implementation or experimentation. |
| Experiment Setup | Yes | For each setting of (k, γ), we choose a subset of k reviewers uniformly at random from among those reviewers that authored at least one paper. We then add edges uniformly at random between colluding reviewers until the subgraph has edge density at least γ. This modified graph is then passed as input to each detection algorithm. We repeat this procedure for 50 trials for each setting. |