Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data
Authors: Feng Liu, Wenkai Xu, Jie Lu, Danica J. Sutherland
NeurIPS 2021 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide both theoretical justification and empirical evidence that our proposed meta-testing schemes outperform learning kernel-based tests directly from scarce observations, and identify when such schemes will be successful. |
| Researcher Affiliation | Academia | Feng Liu Australian AI Institute, UTS EMAIL Gatsby Unit, UCL EMAIL Jie Lu Australian AI Institute, UTS EMAIL Danica J. Sutherland UBC and Amii EMAIL |
| Pseudocode | Yes | Algorithm 1 Meta Kernel Learning (Meta-KL), Algorithm 2 Testing with a Kernel Learner, Algorithm 3 Meta Multi-Kernel Learning (Meta-MKL) |
| Open Source Code | Yes | Implementation details are in Appendix B.1; the code is available at github.com/fengliu90/Meta Testing. |
| Open Datasets | Yes | We distinguish the standard datasets of CIFAR-10 and CIFAR-100 [52] from the attempted replication CIFAR-10.1 [53], similar to Liu et al. [16]. |
| Dataset Splits | Yes | Here, one divides the observed data into training and testing splits, identifies a kernel on the training data by maximizing a power criterion ˆJ, then runs an MMD test on the testing data (as illustrated in Figure 1a). [...] we set the number of training samples (Str P , Str Q ) to 50, 100, 150 per mode, and the number of testing samples (Ste P , Ste Q ) from 50 to 250. |
| Hardware Specification | No | No specific hardware details (GPU/CPU models, memory, etc.) are mentioned in the paper. |
| Software Dependencies | No | We use PyTorch [55] and Adam [56] for our implementations. |
| Experiment Setup | Yes | We use a learning rate of 1e-4 and 5 inner gradient steps for Meta-KL. We run 1000 outer iterations for the meta-training. [...] The deep neural network φ in Eq. (8) is composed of 3 fully connected layers with 1024, 512, 256 units respectively, and ReLU activation functions. |