Collaborative Research: Elements: Portable Library for Reinforcement Learning on Heterogeneous Cloud Cyberinfrastructure

Reinforcement Learning (RL) is a powerful Artificial Intelligence technique that enables machines to teach themselves optimal decision making by repeatedly learning from the consequences of their actions. RL has demonstrated impressive gains in building autonomous agents in a variety of scientific and engineering domains such as autonomous driving, robotics, and game playing to name a few.

However, training these agents is extremely time consuming. Moreover, existing research on reducing the execution time of RL is inaccessible to RL application developers as they require expertise in designing careful compute orchestration among different types of hardware devices (such as Graphics Processing Units (GPU), Field Programmable Gate Arrays (FPGA)) available in modern data centers.

To address this issue, the objective of this project is to develop an easy-to-use library that can enable automatic deployment of RL applications on a data center composed of GPU, FPGA, and AI accelerators. By abstracting away the complexities of deployment and orchestration of computations on data centers, the project will significantly increase the productivity of AI application developers leading to more robust AI agents with faster development cycles.

Existing RL libraries only consider homogeneous platforms that are composed of a single type of hardware device. Thus, there is a demonstrated need from researchers in the Computer & Information Science & Engineering (CISE) communities, including AI system development, RL algorithm, and domain application user communities (e.g., scientific computing and cyber-physical systems) for a library that can enable seamless deployment of RL applications on emerging heterogeneous platforms (composed of multiple types of hardware devices) while achieving high performance (for example, reduced training time).

To address this need, this project will leverage novel algorithmic, architectural, and memory optimizations across heterogeneous devices to create a performance portable library that will enable automatic system composition for high-throughput RL on heterogeneous cyber infrastructures. The library will build upon and harden the research artifacts developed in the recent NSF-funded work of the investigators on accelerating RL on CPU-FPGA platforms.

It will be portable to various heterogeneous cyber infrastructures and support a wide range of RL hyperparameters, algorithms, and policy models. Furthermore, it will offer APIs to facilitate productive development and seamless integration with existing RL ecosystems. Additionally, the project will include the following community interactions and sustainability plans: 1.

Interactions with key processor, GPU, and FPGA vendors (AMD, Intel and NVIDIA) to integrate the proposed library into their software development tools (AMD-Xilinx Vitis, Intel oneAPI and NVIDIA CUDA-X). 2. Collaborations with NSF Open Cloud Testbed, and NSF NCSA to integrate the library into their cyber infrastructures. 3. Making the library compatible with existing RL frameworks (e.g., RLlib) and RL simulation toolkits (e.g., Gymnasium). 4.

Ensuring the availability of the library to a broader audience by collaborating with commercial cloud service providers such as Microsoft and Amazon. 5. Demonstrating end to end applications in various domains through collaborations with NSF AI Institutes including ACTION, AgAID, and ICICLE.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.