We first offer a brief introduction to graph neural networks (GNNs). We will illustrate several classical and basic GNN models and their real-world applications [1–4]. These domains include social networks [5], biological molecules [6, 7], and recommendation systems [8]. Following the basic concepts, we will introduce the scalability challenge of large-scale GNNs via both theoretical analysis and empirical examples [9, 10].
This section focuses on classic research on the scalability and efficiency of large-scale GNNs using various innovative designs, such as sampling methods [9–15], pre-computing or post-computing methods [4, 16–21], and distributed methods [22] . Targeting on different categories, we will introduce their basic methodologies, how they mitigate the scalability issue, their variants, and their limitations. This part provides the audience with the necessary knowledge about how we can work with graphs of increasing sizes.
This part will provide an overview of emerging trends and techniques in scalable GNN research. We will discuss the latest developments in lazy graph propagation [23], alternating training [24], layer-wise training [25], graph condensation [26], graph distillation [27], pre-training [28], sparsification [29] and pruning [30]. By the end of this discussion, the audience will gain a comprehensive understanding of the current state-of-the-art in scalable GNN research and their potential for solving real-world problems.
In this section, we will conduct the evaluation of introduced schemes, which involves assessing their performance in terms of accuracy, computational complexity, memory usage, and parallelism. The results of these evaluations can help the audience know the strengthless and limitations of each technique. This will not only guide the selection of appropriate techniques for scaling GNNs but also provides inspiration for researchers and practitioners to design more efficient and accurate large-scale GNNs.
As more and more large-scale graph data becomes available, scalable GNNs will become increasingly important for solving complex problems and making accurate predictions in a variety of domains. In this section, we will discuss the impact of scalable GNNs in various popular real-world applications, such as social network analysis [5], recommendation systems [8], and biological networks [6, 7]. Finally, we will wrap up by discussing some open challenges and research directions in real-world applications.
We will summarize the tutorial and provide an inspiring discussion on the future directions of large-scale GNNs.
[1] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
[2] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Combin- ing neural networks with personalized pagerank for classification on graphs. In International Conference on Learning Representations, 2019.
[3] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
[4] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International confer- ence on machine learning, pages 6861–6871. PMLR, 2019.
[5] Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370–7377, 2019.
[6] Yifei Zhang, Hanqing Zhang, and Jian Tang. Graph convolutional networks for protein structure prediction. In Advances in Neural Information Processing Systems, pages 991–1001, 2018.
[7] Hanjun Dai, Lantao Yu, Bo Dai, and Le Song. Molecular property prediction: A multilevel quantum interactions modeling perspective. In International Confer- ence on Machine Learning, pages 1083–1092. PMLR, 2018.
[8] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5):1–37, 2022.
[9] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[10] Jie Chen, Tengfei Ma, and Cao Xiao. Fastgcn: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:1801.10247, 2018.
[11] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gcn: An efficient algorithm for training deep and large graph convolu- tional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 257–266, 2019.
[12] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method. In International Conference on Learning Representations, 2020.
[13] Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. Advances in neural information processing systems, 32, 2019.
[14] Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In International Conference on Machine Learning, pages 3294–3304. PMLR, 2021.
[15] Haiyang Yu, Limei Wang, Bokun Wang, Meng Liu, Tianbao Yang, and Shuiwang Ji. Graphfm: Improving large-scale gnn training via feature momentum. In International Conference on Machine Learning, pages 25684–25701. PMLR, 2022.
[16] Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198, 7:15, 2020.
[17] Chuxiong Sun, Hongming Gu, and Jie Hu. Scalable and adaptive graph neural networks with self-label-enhanced training. arXiv preprint arXiv:2104.09376, 2021.
[18] Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer perceptron. arXiv preprint arXiv:2206.04355, 2022.
[19] Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2464–2473, 2020.
[20] Xiaojin Zhu. Semi-supervised learning with graphs. Carnegie Mellon University, 2005.
[21] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson. Combining label propagation and simple models out-performs graph neural networks. arXiv preprint arXiv:2010.13993, 2020.
[22] Vasimuddin Md, Sanchit Misra, Guixiang Ma, Ramanarayan Mohanty, Evan- gelos Georganas, Alexander Heinecke, Dhiraj Kalamkar, Nesreen K Ahmed, and Sasikanth Avancha. Distgnn: Scalable distributed training for large-scale graph neural networks. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14, 2021.
[23] Rui Xue, Haoyu Han, MohamadAli Torkamani, Jian Pei, and Xiaorui Liu. Lazygnn: Large-scale graph neural networks via lazy propagation. arXiv preprint arXiv:2302.01503, 2023.
[24] Haoyu Han, Xiaorui Liu, Torkamani Ali, Feng Shi, Victor Lee, and Jiliang Tang. Alternately optimized graph neural networks. arXiv preprint arXiv:2206.03638, 2022.
[25] Keyu Duan, Zirui Liu, Peihao Wang, Wenqing Zheng, Kaixiong Zhou, Tianlong Chen, Xia Hu, and Zhangyang Wang. A comprehensive study on large-scale graph training: Benchmarking and rethinking. arXiv preprint arXiv:2210.07494, 2022.
[26] Mengyang Liu, Shanchuan Li, Xinshi Chen, and Le Song. Graph condensation via receptive field distribution matching. arXiv preprint arXiv:2206.13697, 2022.
[27] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7074–7083, 2020.
[28] Xiaotian Han, Tong Zhao, Yozen Liu, Xia Hu, and Neil Shah. MLPInit: Embarrassingly simple GNN training acceleration with MLP initialization. In The Eleventh International Conference on Learning Representations, 2023.
[29] Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pages 1695–1706. PMLR, 2021.
[30] Li Wang, Wei Huang, Miao Zhang, Shirui Pan, Xiaojun Chang, and Steven Wei- dong Su. Pruning graph neural networks by evaluating edge properties. Knowledge-Based Systems, 256:109847, 2022.