Encryption-Friendly LLM Architecture
Authors: Donghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim, Hyunsik Chae, Ernest Ryu, Jung Hee Cheon
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results of our modified BERT model on encrypted data using the CKKS scheme demonstrate its ability to securely process natural language data. Our findings show promise for offering privacy-preserving LLM services in areas where data protection is crucial. Our code is available on GitHub1. ... Section 5: EXPERIMENTAL RESULTS |
| Researcher Affiliation | Collaboration | 1Seoul National University, Department of Mathematical Sciences EMAIL 2Crypto Lab Inc, EMAIL 3UCLA, Department of Mathematics EMAIL |
| Pseudocode | Yes | Algorithm 1 Adam W-HE ... Algorithm 2 Split & repeat row-wise (Figure 5b) ... Algorithm 3 Repeat column-wise ... Algorithm 4 Collect into the first column ... Algorithm 5 Lo RA CCMMs |
| Open Source Code | Yes | Our code is available on Git Hub1. 1https://github.com/Donghwan-Rho/Encryption-friendly_LLM_Architecture |
| Open Datasets | Yes | We evaluate our model on the GLUE benchmark (Wang et al., 2018). ... We fine-tune using the cross-entropy loss for tasks including Co LA (Warstadt et al., 2019), MRPC (Dolan & Brockett, 2005), RTE (Giampiccolo et al., 2007), QNLI (Wang et al., 2018), and SST-2 (Socher et al., 2013), and MSE loss for STSB (Cer et al., 2017). |
| Dataset Splits | No | The paper mentions using the GLUE benchmark and specific tasks within it, which typically have standard splits. However, it does not explicitly state the dataset split percentages, sample counts, or refer to specific predefined splits with citations for reproducibility in its text. |
| Hardware Specification | Yes | Our implementation is based on the C++ HEaa N library (Crypto Lab, 2022). All of our experiments used 8 Nvidia Ge Force RTX 4090 24GB GPUs. |
| Software Dependencies | No | Our implementation is based on the C++ HEaa N library (Crypto Lab, 2022). ... Remez algorithm is computed using the Sollya tool (Chevillard et al., 2010). The paper mentions the HEaaN library and Sollya tool but does not provide specific version numbers for these or other software dependencies, which are necessary for reproducible descriptions. |
| Experiment Setup | Yes | We set the number of transformer layers as 2 for the practical computation time. We apply Lo RA only to the query, value, and key layers as applying Lo RA to other layers (e.g., FFN) did not give a noticeable performance gain in our experiments. Lo RA rank is 2 for all Lo RA layers. ... Table 11: Hyperparameters used for HE experiments. Epsilon means ε of Adam W-HE in section 5.1. Warmup steps, Number of cycles are used in transformers (Wolf et al., 2020) cosine scheduler, and betas are used in Adam W-HE. (Followed by a table with specific values for Learning Rate, Epsilon, Warmup Steps, Number of Cycles, Betas for each task). |