CAREER: Toward Scalable Wide Area Network-enabled Geo-distributed Data Analytics

Geo-distributed data analytics (GDA) is widely used across commercial, scientific, and social domains to obtain insights from large-scale, distributed data in across multiple cloud environments. To improve performance, existing GDA systems often avoid assigning tasks to sites with low wide area network (WAN) throughput based on the assumption that WAN bandwidth is static, limited, and non-scalable.

However, these assumptions are not necessarily true and lead to sub-optimal decisions limiting performance gains in GDA systems. This project is developing a lightweight framework to enable GDA systems to exploit a dynamic and scalable WAN, improving performance with minimal cost. The project's novelties are (1) dynamically scaling WAN throughput based on GDA workloads and cost-performance goals by accurately predicting runtime WAN bandwidth and optimizing cloud configurations, and (2) precisely simulating live WAN environments within a local cluster setting.

This project's broader significance and importance impact both research and STEM education and workforce development. It will enable researchers and industry software engineers to readily benefit from the new framework with minimal effort, and the research findings will be integrated into the university's undergraduate and graduate curricula, creating new opportunities in STEM education and broadening the tech workforce.

To fully exploit WAN performance across sites while avoiding performance bottlenecks, the research objectives are (1) developing an accurate runtime WAN bandwidth prediction model, (2) determining scalable WAN bandwidth, (3) making optimal cloud configurations for scalable WAN, and (4) mimicking live WAN environments in local cluster settings. An open-source prototype implementation will be publicly available for researchers and industry software developers across various domains to (1) more easily achieve cost-performance goals in real WAN environments and (2) accurately assess their systems' efficacy in simulated WAN environments on local clusters.

We expect that the outcomes of this project to greatly improve our understanding of real and simulated WAN environments, which will significantly influence the design of future GDA systems and multi-cloud applications.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.