CAREER: Hierarchical Reinforcement Learning Framework for Safe Dynamic Bipedal Locomotion

The anthropomorphic design of bipedal robots equips these machines with unique advantages navigating challenging terrains (e.g., natural uneven grounds, hills, and stairs), operating in restricted environments (e.g., narrow vertical spaces in house or warehouse designed for human operators), and interacting with other humans in a natural manner. However, the technological realization of safe and dynamic behaviors in bipedal robots remains challenging due to the fundamental lack of understanding of underlying mechanisms of locomotion controllers.

Such understanding will also help improve assistive devices such as lower-limb exoskeletons, which could help restore mobility lost due to stroke or other movement disorders. This Faculty Early Career Development (CAREER) project aims to significantly advance the technology of bipedal robots and lower-limb exoskeletons through a novel learning-based feedback motion control framework, with a specific focus on experimentally realizing safe and dynamic bipedal locomotion in real-world settings.

Inspired by how humans learn complex tasks hierarchically, this project will make major innovations to motion control for safe bipedal locomotion through a physics-inspired hierarchical structure. Moreover, we will leverage the innate appeal of bipedal robots in our coordinated education and outreach plans to engage students of various levels in STEM education and research programs.

The successful completion of this project has the potential to accelerate real-world applications of bipedal robots in industry and public health and improve the depth of the STEM talent pool.

Bipedal robots are inherently unstable and consist of many degrees of freedom. Despite current achievements in robot manipulation and mobile robots, typical reinforcement learning (RL) algorithms are prone to fail and are difficult to scale when used for bipedal robots. The robot will fall without proper control, resulting in very sparse and discontinuous rewards, causing the RL algorithms not to converge.

The high dimensionality of bipedal robots also increases the search space of the classic “flat” RL algorithms, leading to sampling inefficiency and a lengthy (potentially unsuccessful) training process. Many existing applications of RL on bipedal robots do not respect the physical limitations of the robot, and consequently, cannot be implemented on robot hardware.

This research will therefore address these scientific challenges by pursuing the following four research goals: (G1) develop a hierarchical learning structure that enables the robot to efficiently explore the high-dimensional behavior space by reducing the task complexity through the temporal abstraction of skills at different levels, (G2) design a probabilistically safe “safety filter” to ensure and guide safe policy learning via control barrier functions, (G3) improve learning efficiency through guided policy exploration that imitates physics-inspired template models in a layered fashion, and finally (G4) bridge the “sim-to-real” gap for real-world deployments. The resulting framework will be experimentally demonstrated with a 3D bipedal robot (Digit) in real-world settings and a lower-limb exoskeleton (ATALANTE) in pre-clinical trials with non-Spinal-Cord-Injury healthy human subjects.

This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.