Sequential Query Encoding for Complex Query Answering on Knowledge Graphs
Authors: Jiaxin Bai, Tianshi Zheng, Yangqiu Song
TMLR 2023 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Despite its simplicity, SQE demonstrates better faithfulness and inference capability than state-of-the-art neural query encoders over FB15k (Bollacker et al., 2008; Bordes et al., 2013), FB15k-237 (Toutanova & Chen, 2015), and NELL (Carlson et al., 2010), under an extended benchmark, which includes twenty-nine types of logical queries (Wang et al., 2021). Moreover, we further evaluate the compositional generalization (Fodor & Lepore, 2002) of the SQE to see whether it can also generalize to the out-of-distribution query types that are unobserved during the training process. Again, the SQE method with an LSTM backbone demonstrates comparable inference capability to state-of-the-art neural query encoders on the out-of-distribution queries. ... In this section, we first explain the evaluation settings and metrics for query encoding. Then we discuss the knowledge graphs and the benchmark datasets for evaluation. Then we briefly present the neural query encoding baseline methods that we directly compared with. After this, we disclose the implementation details for SQE. Finally, we discuss the experiment results and conduct further analysis on SQE. |
| Researcher Affiliation | Academia | Jiaxin Bai EMAIL Department of Computer Science and Engineering Hong Kong University of Science and Technology Tianshi Zheng EMAIL Department of Computer Science and Engineering Hong Kong University of Science and Technology Yangqiu Song EMAIL Department of Computer Science and Engineering Hong Kong University of Science and Technology |
| Pseudocode | Yes | In SQE, instead of parameterizing the operators and executing the computational graph, we use a search-based algorithm to linearize the graph into a sequence of tokens. After this, SQE uses a sequence encoder, like LSTM (Hochreiter & Schmidhuber, 1997) and Transformer (Vaswani et al., 2017), to encode this sequence of tokens. Its output... A directed acyclic computational graph is first linearized to a sequence of tokens. Our linearizing algorithm as shown in Algorithm 1, starts from its target node T. |
| Open Source Code | Yes | Code available: https://github.com/HKUST-KnowComp/SQE |
| Open Datasets | Yes | In our experiment, following previous work, we also use the following three knowledge graphs: FB15k (Bollacker et al., 2008; Bordes et al., 2013), FB15k-237 (Toutanova & Chen, 2015), and NELL (Carlson et al., 2010). |
| Dataset Splits | Yes | As shown in Table 8, the edges in each knowledge graph are separated into training, validation, and testing with a ratio of 8:1:1 respectively. Training graph Gtrain, validation graph Gval, and test graph Gtest are constructed by training edges, training+validation edges, and training+validation+testing edges respectively following previous setting (Ren et al., 2020). |
| Hardware Specification | Yes | All the experiments are conducted on the Nvidia Ge Force RTX 3090 graphics cards. |
| Software Dependencies | No | The paper describes the use of LSTM and Transformer models and their configurations (e.g., three layers, sixteen attention heads, hidden sizes of four hundred) but does not provide specific version numbers for software libraries or environments like Python, PyTorch, TensorFlow, etc. without which the software environment cannot be replicated. |
| Experiment Setup | Yes | The hidden sizes of both recurrent models and transformer are set to be four hundred to fairly compared with the baselines. There are also neural-symbolic query encoding methods proposed (Sun et al., 2020; Xu et al., 2022; Zhu et al., 2022). In this line of research, their query encoders refer back to the training knowledge graph to obtain symbolic information from the graph. Because of this, the query encoder is not purely learned from the training queries. As their contribution is orthogonal to our discussion on pure neural query encoders, we do not conduct direct comparisons. All the SQE and previous QE models are trained with the same batch size of 1024. |