Generalization-Driven Distributed Continual Learning

Signal processing and machine learning models are traditionally optimized to minimize the training error on specific datasets.

However, when these models, optimized for specific tasks or data characteristics, encounter new, yet related tasks or data, their performance declines significantly.

This inability to generalize poses a major obstacle for the reliable use of these models in dynamic real-world settings.

Hence, there is an urgent need for training methodologies and model architectures that facilitate continual learning (CL) - the ability to master new tasks while retaining previously acquired knowledge.

In this project, we address this challenge by focusing on CL under distributed learning scenarios, where multiple devices in a network collaborate.

Despite  numerous empirical successes of CL, analysis of the fundamental limits of generalization, particularly in distributed settings, is very limited. Bridging this gap is the starting point of this project. Overparameterization and the double-descent phenomenon will be key themes in our work.

We will reveal limits of generalization under distributed continual learning schemes through a rigorous theoretical framework.

Using the structured insights gained through our analysis, we will uncover the trade-offs between CL efficacy and distributed system attributes, and develop novel distributed CL methods with strong generalization properties.