Factor Graph Neural Networks
Authors: Zhen Zhang, Mohammed Haroon Dupty, Fan Wu, Javen Qinfeng Shi, Wee Sun Lee
JMLR 2023 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive experimental evaluation on synthetic as well as real datasets demonstrates the potential of the proposed model. 5. Experiments In this section, we evaluate our FGNN over a diverse set of tasks. First, we evaluate FGNN s performance for graphical model inference. We create synthetic PGMs in both low order and higher-order settings, and compare FGNN with other PGM inference algorithms. We also conduct experiments on low-density parity check (LDPC) decoding task (a typical PGM inference task). Next, we want to study how does FGNN perform on real-world data as against other state-of-the-art models. For this, we evaluate FGNN over three real-world problems where capturing higher-order dependencies is likely useful. We report experiments on the graph matching problem formulated as a PGM inference problem. We performed experiments on handwritten character recognition within words to demonstrate that FGNN is able to exploit sequence information. To validate the effectiveness of FGNN in capturing higher-order information from more general graph structured data, we report results on molecular datasets and compare with other state-of-the-art k-order GNNs that capture higher-order information as well. Finally, we show FGNN is well suited for modeling human motion prediction task. |
| Researcher Affiliation | Academia | Zhen Zhang EMAIL Australian Institute for Machine Learning & University of Adelaide, Australia Mohammed Haroon Dupty EMAIL National University of Singapore, Singapore Fan Wu EMAIL University of Illinois at Urbana-Champaign, USA Javen Qinfeng Shi EMAIL Australian Institute for Machine Learning & University of Adelaide, Australia Wee Sun Lee EMAIL National University of Singapore, Singapore |
| Pseudocode | Yes | Algorithm 1 The FGNN layer Input: G = (V, C, E), [f i]i V, [gc]c C, [tci](c,i) E Output: [ f i]i V, [ gc]c C 1: Variable-to-Factor: 2: gc= AGG i:(c,i) E Q(tci |ΦVF) M([gc, f i]|ΘVF) 3: Factor-to-Variable: 4: f i = AGG c:(c,i) E Q(tci |ΦFV) M([ gc, f i]|ΘFV) |
| Open Source Code | No | The paper does not provide explicit statements or links to their own open-source code for the FGNN methodology. It only mentions using code for comparison, such as "the AD3 code is from the official code repo" and "libdai". |
| Open Datasets | Yes | On problems closely related to PGM inference, we experimented with a synthetic higher-order PGM inference problem, LDPC decoding, as well as a graph matching problem. Data We construct three synthetic datasets (D1, D2 and D3) for this experiment. LDPC Decoding (MAP or Marginal Inference) ...encoding scheme 96.3.963 (Mac Kay, 2009). The Graph Matching Problem (MAP Inference) ...we use the Pascal VOC dataset (Everingham et al., 2010) with the keypoints annotated by Bourdev and Malik (2009) to evaluate the performance... 5.4 Handwritten character sequence recognition (Marginal Inference) ...handwriting recognition data set from (Taskar et al., 2004), originally collected by (Kassel, 1995). 5.5 Molecular data (Graph Regression) ...two large scale datasets on quantum chemistry, QM9 (Ruddigkeit et al., 2012; Ramakrishnan et al., 2014) and Alchemy (Chen et al., 2019) datasets... 5.6 Human Motion Prediction (Sequential Prediction) ...using the Human 3.6M (H3.6M) dataset. |
| Dataset Splits | Yes | Data We construct three synthetic datasets (D1, D2 and D3) for this experiment. In the experiment, for all learning-based methods, we generate y from randomly sampled x on the fly with SNRd B {0, 1, 2, 3, 4} and σb {0, 1, 2, 3, 4, 5}. For each learning-based method, 108 samples are generated for training. Meanwhile, for each different SNRd B and σb, 1000 samples are generated for validating the performance of the trained model. ...where 7020 annotated images are used for training and 1682 for testing. We follow a 10-fold cross-validation setup with 1 fold for training and 9 folds for testing and report averaged results. We follow the standard 80:10:10 random split for training, validation, and testing. Alchemy (Chen et al., 2019) is a recently released more challenging dataset of 119K organic molecules comprising of 7 atom-types H, C, N, O, F, S (Sulphur) and Cl (Chlorine) with up to 14 heavy atoms. These molecules are screened for being more likely to be useful for medicinal chemistry based on the functional group and complexity. Furthermore, we follow the split based on molecule size where almost all training set contains molecules with 9 or 10 heavy atoms while the validation and test set contain molecules with 11 or 12 heavy atoms. We train our model on the Human3.6M dataset using the standard training-val-test split as previous works (Mao et al., 2019; Li et al., 2018; Martinez et al., 2017)... |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU/CPU models, processor types, or memory amounts used for running the experiments. While it mentions `pytorch` which often implies GPU usage, no specific hardware models are mentioned. |
| Software Dependencies | No | The model is implemented in pytorch (Paszke et al., 2017), and trained with Adam optimizer (Kingma and Ba, 2014) with initial learning rate lr = 3 * 10^-3. We compare FGNN with three public available LDPC decoder Mac Kay-Sum (Mac Kay, 2009), Commpy-Sum (Taranalli, 2020) and Commpy-Max (Taranalli, 2020). The MPNN implementation has Edge-conv (Gilmer et al., 2017) as message function, GRU (Chung et al., 2014) as update function followed by a Set2set (Vinyals et al., 2015) function as readout for the whole graph. In our implementation, we run 3 iterations of MPNN message passing scheme on input graph followed by 3 iterations of higher-order FGNN message passing described in Algorithm 1. Learning node marginals is likely to be helpful in this case as we want good node representations to be combined in the readout function. We use max function as the aggregator in VF module and summation as aggregator in FV module. We select this combination as we found it useful while being numerically stable. We then combine the MPNN output from the third iteration and the FGNN output with concatenation followed by the set2set readout function. For FGNN module, we set the hidden vector dimension to 64. The projection dimension of VF module is set to to 512. We use Adam optimizer initialized with learning rate of 1e-3. Since we want to show improvements over MPNN, all other hyperparameters are maintained as is provided by Pytorch-geometric implementation of MPNN for a fair comparison. |
| Experiment Setup | Yes | Architecture and training details In this task, we use a factor graph neural network consisting of 8 FGNN layers (see the detail in the Appendix). The model is implemented in pytorch (Paszke et al., 2017), and trained with Adam optimizer (Kingma and Ba, 2014) with initial learning rate lr = 3 * 10^-3. After each epoch, lr is decreased by multiplying a factor of 0.98. All the models in Table 1 were trained for 50 epochs after which all of them achieve convergence. Architecture and training details In this task, we use a factor graph neural network consisting of 8 FGNN layers (the details are provided in the Appendix). The model is implemented by using pytorch (Paszke et al., 2017), and trained with Adam optimizer (Kingma and Ba, 2014) with initial learning rate lr = 1 * 10^-2. After every 10000 samples, lr is decreased by multiplying a factor of 0.98. After training over 108 samples, the training loss converges. Architecture and training details In our experiment, we use the same training protocol as in (Wang et al., 2019; Yu et al., 2019; Fey et al., 2020; Rolínek et al., 2020; Xu et al., 2021), where the input of the models are two sets of coordinates of key-points and two images. ... The above node, edge and factor features will be fed into a FGNN network composed by three blocks. ... will be trained with Adam (Kingma and Ba, 2014) optimizer with learning rate 10^-4 (10^-6 for the VGG part). Our algorithm has been trained for 200 epochs and in each epoch we random samples 16000 image pairs from the training set. Architecture and training details We use 3 standard convolution and a fully connected layer as the base feature extractor to get the zero-th order feature of 512 dimensions. We then run 3 iterations of higher-order message passing followed by a classifier. We fix the rank of all factors at 1024 and share parameters between factors with the same order. We train for 50 epochs with a learning rate of 0.001. Architecture and training details We run our message passing scheme on features initialized on MPNN network with the standard implementation provided by Pytorchgeometric (Fey and Lenssen, 2019). ... In our implementation, we run 3 iterations of MPNN message passing scheme on input graph followed by 3 iterations of higher-order FGNN message passing described in Algorithm 1. ... For FGNN module, we set the hidden vector dimension to 64. The projection dimension of VF module is set to to 512. We use Adam optimizer initialized with learning rate of 1e-3. ... All targets are normalized and are trained with the absolute error loss for 200 epochs with batch size of 64. Architecture and training details We train our model on the Human3.6M dataset using the standard training-val-test split as previous works (Mao et al., 2019; Li et al., 2018; Martinez et al., 2017), and we train and evaluate our model using the same protocol as (Mao et al., 2019) (For details, see the Appendix). |