Dirichlet stochastic weights averaging for graph neural networks

Abstract: The popularity of Graph Neural Networks (GNNs) has grown significantly because GNNs handle relational datasets such as social networks and citation networks. However, the usual relational dataset is sparse, and GNNs are easy to overfit to the dataset. To alleviate the overfitting problems, model ensemble methods are widely studied and adopted. However, model ensemble methods for GNNs are not well explored. In this study, we propose simple but effective model ensemble methods for GNNs. This is the first study that adopts stochastic weights averaging (SWA) for GNNs. Furthermore, we propose a new model ensemble method, Dirichlet stochastic weighs averaging (DSWA). DSWA adopts the running averages of the trained weights with random proportions sampled by Dirichlet distributions. DSWA provides the diverse model and its ensembles on inference time without the training time increases. We validate our models on the Cora, the Citeseer, and Pubmed datasets under usual settings and few-shot learning settings. We observe that the performance of current GNNs deteriorates when the number of specified data is limited. DSWA improves the performance of few-shot node classification tasks as well as the general node classification tasks. Graphical abstract: (Figure presented.)