Balancing Decorrelation and Label Predictability for Deep Graph Representation Learning

Deep GNNs allow us to obtain rich node representations by exploiting information from high-order neighbors. When node features are missing, deep GNNs become increasingly crucial, as they can enhance node representations by aggregating information from a wider neighborhood. However, as more GNN layers are stacked, the quality of node representations often deteriorates because of oversmoothing across nodes or overcorrelation across dimensions. Recently, decorrelation methods such as DeCorr and DeProp have been proposed, but performance degradation still happens in deep GNNs. We propose BalGNN to learn both unique (decorrelated) and useful (discriminative) hidden dimensions for deep GNNs. To this end, we suggest two loss terms to balance (1) decorrelation between hidden dimension pairs and (2) label predictability of each dimension. We also propose matrix norm to normalize the matrix of hidden representations at once instead of normalizing each column or row separately. Extensive experiments show that BalGNN consistently outperforms SOTA decorrelation methods in 39 out of 48 settings. Importantly, BalGNN exhibits high and stable accuracy in deep GNNs, unlike other methods. We also investigate the interaction of correlation and smoothness and demonstrate that we can achieve even higher accuracy by considering both. Code is available at https://github.com/na1-4an/BalGNN.