RINGS: Scalable and Resilient Networked Learning Systems

Next-generation learning systems enabling applications ranging from healthcare, energy, banking, augmented/virtual reality and car/robot navigation, will be privacy-driven, distributed and large-scale, resulting in substantially increased exposure to network congestion and failures. A typical scenario is one where a substantial number of entities (clients, organizations, communities thereof) participate in a learning task; each of these entities owns, has access to, or generates part of an overall data set which would be impractical or undesirable to gather at a central location.

While the entities benefit from a collaboration, especially in settings where the amount of local data is relatively small and thus the ability of an entity to learn a model on its own is limited, such a collaboration requires major network resources. This research project centers on developing new, as well as expanding traditional, engineering principles for the design of resilient and scalable networked learning systems.

To this end, the project investigates interrelated themes spanning the development of theoretical underpinnings, network architecture, applications and protocol design. The broader impacts offer advances in education, enhancing diversity, engaging the community and industry, and disseminating results to a wider public.

The project will initially focus on Federated Learning (FL) frameworks and, in Theme 1, study how to achieve resilience to uncertainty in FL systems experiencing intermittent client availability and time-varying network capacity. This leads to a novel approach to FL which effectively "learns how to learn" in an uncertain/resource-constrained environment.

Theme 2 addresses scalability challenges encountered in large-scale FL by relying on clustering of "exchangeable" clients. This moves from client-centric to an efficient cluster-centric system management by leveraging multicast-based estimation/tracking of cluster populations, combined with probabilistic scheduling of clients in the clusters. This offers new avenues to scalability and resiliency as well as potential privacy enhancements.

Theme 3 builds on ideas from rate-distortion theory and scalable video coding, and explores the use of scalable/layered (learned) model compression as a basis for adaptive congestion-aware FL. A key idea here is recognizing that aggressive compression leads to faster delivery, which motivates the search for a tradeoff sweet spot where FL performs more updates but with poorer (noisier) models.

This research exemplifies research synergies of ideas from information, queueing and learning theory towards achieving resilience and adaptability. The project also pursues the design and use of overlay Data Aggregation Networks which exploit the aggregative character of model updates via in-network update aggregation and associated data compression.

This can be viewed as the dual of Content Delivery Network overlays which are a core element managing the cost and performance in current network infrastructure. Theme 4 recognizes that at the base of FL applications is client participation and thus brings into focus the joint incentivization of clients and management of limited resources in uncertain environments.

Overall, the proposed research focuses on new forms of network intelligence and adaptability, aiming to address scalability through device-to-edge-to-cloud continuum.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.