Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences
Authors: Nikos Dimitriadis, Pascal Frossard, François Fleuret
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments show that Pa Lo RA outperforms state-of-the-art MTL and PFL baselines across various datasets, scales to large networks, reducing the memory overhead 23.8 31.7 times compared with competing PFL baselines in scene understanding benchmarks. |
| Researcher Affiliation | Collaboration | Nikolaos Dimitriadis EPFL Pascal Frossard EPFL Franc ois Fleuret University of Geneva, Meta FAIR |
| Pseudocode | Yes | Algorithm 1: Pa Lo RA |
| Open Source Code | Yes | Contact: EMAIL. Code available at github.com/nik-dim/palora. |
| Open Datasets | Yes | We evaluate Pa Lo RA on a variety of tasks and datasets, ranging from multi-label classification to complex scene understanding benchmarks in City Scapes and NYUv2. ...First, we test the effectiveness of Pa Lo RA on Multi MNIST, a digit classification dataset based on MNIST... ...We consider UTKFace (Zhang et al., 2017), a dataset with images and three tasks... ...Finally, we explore SARCOS (Vijayakumar & Schaal, 2000), a robotic dataset... ...City Scapes (Cordts et al., 2016) contains high-resolution urban street images... ...we consider the NYUv2 dataset (Silberman et al., 2012) for the tasks of semantic segmentation, depth estimation, and surface normal prediction... |
| Dataset Splits | Yes | NYUv2 Similar to previous MTL works, we consider the NYUv2 dataset (Silberman et al., 2012) for the tasks of semantic segmentation, depth estimation, and surface normal prediction, and report the results in Table 2. We reserve 95 images for validation and use a setup similar to City Scapes but train for 200 epochs. |
| Hardware Specification | Yes | All experiments are conducted with Py Torch(Paszke et al., 2019) in Tesla V100-SXM2-32GB GPUs. |
| Software Dependencies | No | All experiments are conducted with Py Torch(Paszke et al., 2019) in Tesla V100-SXM2-32GB GPUs. Our source code extends the codebases of previous works (Dimitriadis et al., 2023; Navon et al., 2022; Liu et al., 2019). |
| Experiment Setup | Yes | City Scapes (Cordts et al., 2016) contains high-resolution urban street images and we focus on the tasks of semantic segmentation and depth regression. We train a Seg Net (Badrinarayanan et al., 2017) for 100 epochs, using Adam optimizer (Kingma & Ba, 2015) with learning rate 10 4 that is halved after 75 epochs. The results are presented in Table 1 for rank r = 4 and M = 5. |