NRI: Integrating Perception and Manipulation of Deformable Objects by Learning Implicit Representations

The goal of this National Robotics Initiative 3.0 project is to develop models and algorithms that enable autonomous robotic manipulation of deformable objects by integrating sight and touch for contact-rich interactions. Manipulation of deformable objects is essential in many day-to-day tasks ranging from cooking (e.g., using a compliant spatula to scrape a wok) to manufacturing (e.g., assembly tasks involving flexible parts and materials such as composites and cables).

These tasks require the robot to perform complex maneuvers to bring deformable objects into contact and apply forces to bend and shape them to achieve tasks. Recent advances in collaborative robots (e.g., the Franka Emika Panda and Kuka LBR iiwa robots) have made robotic manipulation of deformable objects physically possible but robots still lack the models and algorithms necessary to perform practical tasks such as cooking, cleaning, and flexible manufacturing.

This project will have broad societal impact through its applications in in-home assistive and manufacturing robotics. The ability to robustly manipulate deformable structures is an important precursor technology towards realizing intelligent robotic assistants. Robotics and their assistive applications have the potential to inspire children to pursue careers in STEM fields and meet the needs of America's growing assistive and manufacturing robotics industry.

Integration of the research activities with education will emphasize actively involving undergraduates in research activities and introducing new lecture material and projects into undergraduate and graduate courses. Also, special emphasis will be given to recruit qualified students from under-represented groups.

This project will develop integrated methods for perception and manipulation of elastically deformable objects via novel implicit function representations. Elastic objects are ubiquitous in people's lives, from spatulas and sponges in the kitchens to elastic rods on manufacturing floors to surgical tools and tissue. Their manipulation is an essential skill for practical robotic systems.

The proposed integrative approach brings together methods from computer vision and robotics to address the need for seamless visio-tactile reasoning and acting that is essential for robotics applications. Specifically, this approach integrates and extends recent theoretical and computational advances in implicit function learning to model, perceive, and intelligently manipulate elastic objects.

Implicit representations of 3D geometries have recently gained traction in computer vision owing to their compactness and computational efficiency; however, they have yet to be deployed successfully for robotics. Towards their successful integration, the outcomes of this project are: 1) A generalization of implicit function theory and algorithms for 3D visio-tactile deformable object representations.

This addresses the need for dynamic and multi-modal deformable object representations particularly suited for robotics; specially to streamline the sense-reason-act pipeline. 2) Algorithmic tools for implicit state-estimation. These algorithms will bridge inherently uncertain robotic sensing modalities with implicit representations. 3) Theoretic and algorithmic foundations for implicit control and planning.

The resulting tools from this project will enable real-time closed-loop control of deformable objects for practical robotic systems. For evaluation, this project will explore the integration of these algorithms for assistive robotics (food preparation) and warehouse logistics (dense packing). The research outcomes contribute to several related fields including continuum mechanics, computer vision, and learning theory where object deformations subject to boundary conditions and/or implicit function theory are studied.

In particular, this approach will offer a flexible, generalizable, and computationally efficient alternative to the current state-of-the-art methods using finite element analysis or particle models.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.