DISTRIBUTED LEARNING

Distributed learning provides an attractive framework for scaling the learning task by sharing the computational load over multiple nodes in a network.  Despite the vast interest in distributed learning and central importance of generalization performance in data science, generalization performance of distributed approaches is not  well understood. We address this gap by focusing on the setting where the model is partitioned over a network of nodes. 

We have showed how the generalization error depends heavily on the partitioning of the model parameters among the nodes. Our results highlighted a typically overlooked relationship between the training and generalization error in distributed learning. In particular, distributed learning schemes can significantly amplify the gap between the training error and the generalization error: A distributed solution with a training error that is on the same level as that of the centralized solution is not guaranteed to have a generalization error that is as low as that of the centralized solution. These results are directly connected to double descent curves which illustrate that the relationship between the number of data points and the assumed model dimension (corresponding to the size of the partial model in a node in the distributed learning setting) can significantly affect the generalization error.

Publications:

M. Hellkvist, A. Özçelikkale, A. Ahlén, Distributed Continual Learning with CoCoA in High-dimensional Linear Regression, in IEEE Trans. on Signal Processing, vol. 72, pp. 1015-1031, 2024, doi: 10.1109/TSP.2024.3361714

M. Hellkvist, A. Özçelikkale, A. Ahlén, Linear Regression with Distributed Learning: A Generalization Error Perspective  IEEE Trans.on Signal Processing, vol. 69, pp. 5479-5495, 2021  

M. Hellkvist, A. Özçelikkale, A. Ahlén, Continual Learning with Distributed Optimization: Does CoCoA Forget Under Task Repetition? Proc. European Signal Processing Conference (EUSIPCO), 2024

M. Hellkvist, A. Özçelikkale, A. Ahlén, Generalization Error for Linear Regression under Distributed Learning, Proc. IEEE International Workshop on Signal Processing Advances in Wireless Communications, (SPAWC), 2020