NSF-AoF: RI: Small: Safe Reinforcement Learning in Non-Stationary Environments With Fast Adaptation and Disturbance Prediction

Reinforcement learning (RL) has shown impressive performance in the control of complex robotic systems for various tasks such as locomotion, manipulation, and playing sports, e.g., table tennis. Reinforcement learning enables a robot to autonomously discover an optimal behavior through trial-and-error interactions with its environment. However, the environmental perturbations could easily cause a behavior policy trained in an old environment to fail in a perturbed environment.

The failure is unacceptable for safety-critical robotic systems such as self-driving cars, drones, flying taxies and construction machines. Existing robust methods try to consider all scenarios during the training phase and seek a fixed policy, leading to conservative behaviors. Existing adaptive methods try to update their behavior policies in the perturbed environment, but will only do that after the robot has “felt a difference” through its interaction with the environment.

In contrast, a human could leverage his/her perception for prediction in the new environment and adjust his/her behavior accordingly even before interacting with it. In light of these conditions, this project envisions a new framework for safe and efficient RL in the presence of environmental changes leveraging fast adaptation and perception-based prediction.

The framework will enable robotic and autonomous systems robustly and safely operate, learn and adapt in the real world.

This project relies on the following thrusts: i) hybrid RL for safe and efficient policy updates, ii) robust adaptive control with safety guarantees; iii) vision-based disturbance prediction. More specifically, the project will develop robust adaptive control algorithms that ensure that the executed trajectory of a robot remains safe in the presence of disturbances induced by environmental changes.

It will spur hybrid model-free/model-based RL algorithms that are capable of efficiently and safely updating the behavior policies with the help of the control algorithms. The project will advance novel methodologies for predicting the key parameters of the disturbances (e.g., the weight of a package) directly from the image observations, leading to new scalable methods for efficiently learning the mathematical model of the disturbances with quantified error bounds.

All the ingredients will be holistically integrated to build a framework to enable robots to safely, robustly, and efficiently operate and adapt in real-world environments. Aerial and ground vehicles will be used for experimental validation.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.