Collaborative Research: RI: Small: Active Curriculum and Environment Design for Reinforcement Learning

Recent advances in deep reinforcement learning (RL) have shown impressive results across a variety of applications. However, the broader application of RL often faces significant challenges, particularly in real-world scenarios such as robots interacting with the environment or autonomous driving. The challenges in these complex environments are due factors such as the intricacy of the task requirements and/or few opportunities for the system to know the response is correct (i.e., sparse reward functions).

Moreover, extensive physical interactions with the environment are costly because they take considerable time and staffing and sometimes even unsafe due to potential physical interactions during exploration. Leveraging the principles of active learning and curriculum RL, the project seeks to enhance the performance of RL systems in completing difficult tasks in complex environments, while optimizing resource allocation and reducing the need for expensive environment interactions.

This project intends to fundamentally reshape the RL landscape by developing task and environment representations specifically for active design in RL. More concretely, this project is structured around four interconnected thrusts. First, Active Environment Design for RL (ACED-RL) aims to identify a sequence of auxiliary environments that best facilitate learning in the target environment.

Second, active task design for RL seeks to establish a scalable active task selection strategy, allowing the learner to be trained sequentially in these tasks, facilitating RL, and transferring the acquired knowledge to the target problem. Third, active joint task and environment Design combines active task and environment design to generate RL curricula.

This approach extends to settings involving multiple agents, accounting for challenges posed by simultaneous learning and non-stationary agent behaviors. Finally, the project will evaluate the proposed approaches across various high-impact machine learning applications, including standard RL benchmarks, autonomous driving, robotic manipulation, and scientific experimental design.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.