CRII:OAC:United Learning: Data-Shareable Heterogeneous Distributed Learning

Many organizations and households possess heterogeneous computing resources, including personal computers, smartphones, and Internet of Things (IoT) devices, many of which often remain underutilized. By harnessing this idle computing power to train deep learning models, the need for additional equipment can be reduced, environmental impacts from manufacturing and electronic waste can be minimized, and sustainability can be promoted.

As the prevailing heterogeneous learning framework, Federated Learning does not allow data sharing among devices. However, many applications in smart homes, industrial environments, and institutions could benefit from distributed learning systems that enable data sharing across heterogeneous devices. For example, a hospital can use its existing computers to train a large language model for healthcare services without sending any sensitive health information to an external provider, while the data can be shared internally across these computers to improve model performance.

Such systems face challenges, particularly when training complex models, as devices with low computational power are often excluded because they cannot handle the complexity of these models. Efficient data sharing and utilization across heterogeneous devices is another significant hurdle. To address these challenges, United Learning unites all devices in a distributed system to train complex models efficiently while allowing data sharing among devices.

This project also contributes to designing new learning modules focused on distributed learning across devices with constrained resources. It offers students direct and practical experience to work with real devices that have limited resources. Such engaging experiences can inspire and support underrepresented minorities in pursuing careers and research in computing.

United Learning aims to integrate low-resource devices into the training of complex deep learning models, handling diverse input modalities and data distribution. The approach consists of two primary components: Knowledge Expansion and Supplementary Exemplar Augmentation. Knowledge Expansion organizes devices into learning groups based on their computational capabilities, with each group training a model that expands progressively as it moves to groups with higher resources.

The model expansion may involve adding new neural network blocks for different data modalities or increasing the depth of the model to enhance its expressive power. This iterative process ensures that models evolve toward the desired complex form. Supplementary Exemplar Augmentation addresses efficient data sharing by collecting exemplars from each device and training a Supplementary Exemplar Generator.

Supplementary Exemplar Generator generates supplementary exemplars from other devices’ data in a self-supervised manner. These exemplars are sent back to augment each device’s local dataset, improving efficiency by only exchanging representative data. Supplementary Exemplar Augmentation and Knowledge Expansion are integrated, with Supplementary Exemplar Augmentation dynamically generating exemplars during the Knowledge Expansion process, further balancing resource usage and maximizing efficiency.

Evaluation includes simulations to assess model performance, on-device prototypes to measure device performance, and real-world system deployment to examine system performance.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.