Efficient Large Language Models: A Survey
Authors: Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, Mi Zhang
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. |
| Researcher Affiliation | Collaboration | The Ohio State University Imperial College London Michigan State University University of Michigan Amazon AWS AI Google Research Microsoft Research Asia Boson AI |
| Pseudocode | No | The paper is a survey and describes various techniques and architectures conceptually. It does not include any structured pseudocode or algorithm blocks for its own methodology. |
| Open Source Code | No | We have also created a Git Hub repository where we organize the papers featured in this survey at https://github.com/AIo T-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. (The GitHub repository organizes papers, not the source code for the survey's methodology itself.) |
| Open Datasets | No | This paper is a survey of existing research and does not present new experimental results or datasets used in its own methodology. While it references datasets used in other works, it does not provide access information for a dataset used by the authors for their own research presented here. |
| Dataset Splits | No | This paper is a survey of existing research and does not present new experimental results or perform data splitting in the context of its own methodology. |
| Hardware Specification | No | This paper is a survey and reviews hardware specifications mentioned in other papers (e.g., Figures 1 and 2 mention GPU types used for LLaMA models by other researchers). It does not provide specific hardware details for conducting the research presented in this survey. |
| Software Dependencies | No | This paper is a survey and reviews various LLM frameworks and techniques, some of which involve specific software. However, it does not provide specific software dependencies with version numbers for the methodology used to conduct this survey itself. |
| Experiment Setup | No | This paper is a survey of existing research and does not describe an experimental setup with hyperparameters or system-level training settings for its own methodology. |