OAC Core: Towards Zero-Carbon Data Movement at the HPC and Cloud Data Centers with GreenDataFlow

As commercial and scientific applications generate data at increasingly high rates, the carbon footprint associated with data movement is becoming a critical concern, particularly for High-Performance Computing (HPC) and Cloud data centers. While there is substantial research focusing on power management techniques at the hardware level and lower networking stack layers during data transfers, little attention has been paid to energy-saving measures at the application layer for computing systems such as servers, HPC centers, and Cloud data centers during network data transmission.

The existing strategies in this realm are either prohibitively expensive, impractical in the short term, or sacrifice performance in pursuit of increased energy efficiency. This project develops an innovative application-layer solution, which is cost-effective, practical for immediate deployment, and importantly, does not compromise performance while boosting energy efficiency.

It possesses the ability to adaptively fine-tune several application-layer and kernel-layer transfer parameters, thereby guaranteeing efficient utilization of computing and networking resources. This, in turn, minimizes data transfer energy consumption without undermining end-to-end performance. This revolutionary approach to energy-efficient data transfers underscores the innovation and transformative potential of this project.

The models, algorithms, and tools developed within this project are poised to augment performance and reduce power consumption during end-to-end data transfers, potentially saving gigawatt hours of energy and contributing millions of dollars in savings to the US economy. Furthermore, this project seeks to permeate research insights across all tiers of education.

The well-structured research activities promise to benefit for K-12, undergraduate, and graduate students alike, fostering their academic growth and nurturing future scientists in this critical field.

This project develops novel application-layer models, algorithms, and tools for (1) prediction and tuning of the best cross-layer transfer parameter combination for energy-efficient and high-performance data transfers at the HPC and Cloud data centers; (2) a deep reinforcement learning-based approach that can adapt to the dynamically changing conditions in a wide range of network and end system configurations; (3) accurate estimation of the accompanying network device power consumption due to changing data transfer rate on the active intra- and inter-data center network links and dynamic readjustment of the transfer rate to balance the energy vs. performance ratio; and (4) a suite of service level agreement based energy-efficient transfer algorithms to the HPC administrators and Cloud service providers for dynamically adjustable performance and energy efficiency goals. The evaluation and validation of the proposed models and algorithms are performed in realistic scenarios in collaboration with the HPC Center at Texas Tech University and the Distributed Cloud Management group at IBM.

The research outcomes of this project will fill a significant gap in the data transfer energy efficiency in HPC and Cloud data centers. This project's eventual goal is to translate the research activities into robust, production-quality software libraries that will reduce the carbon footprint of data movement for a range of user communities dealing with large amounts of data.

The project will enable wider broader impacts through the development of graduate and undergraduate curricula, K-12 outreach programs, summer boot camps, the recruitment of minority groups, and broadening participation in computing.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.