NeRD: Neural Representations for Dexterity

Grasping and using objects and tools is a remarkable skill that humans possess, but it remains a challenge to replicate this ability in robotic hands. Achieving such a capability in robots would have a profound impact on autonomous robotic manipulation, revolutionizing various fields like domestic services, manufacturing, and logistics. This project has the potential to advance national prosperity and welfare by enabling robots to perform intricate object manipulation using multi-fingered hands.

To reach this objective, this project identified as a core challenge the design of an appropriate interface between robot's sight sense and control of the hand. This interface is riddled by uncertainty because information that can be extracted from cameras through which the robot observes the environment is not precise and incomplete. Most existing approaches to robot manipulation with multifingered hands oversimplify this interface, often relying on perfect simulation or demonstrating limited tasks on real robotic hardware that often appear brittle.

In contrast, proposed research supported by this award will look to develop a novel concept for robot manipulation, that captures both object shape and appearance. By integrating both types of information, novel method that is intended to be developed, can be used for keeping track of objects, inferring manipulation strategies, and accounting for uncertainty - all at the same time.

The project’s novelties are in the algorithms for dexterous manipulation. Because it aims to enable complex manipulation skills such as tool use, assembly, object inspection or dense packing, this project’s impacts are to improve domestic services, manufacturing and logistics, as well as to develop courses on robotics, and provide immersive and engaging programming activities for K-12 students.

The primary objective of this research project is to develop a novel approach for robotic dexterous manipulation that leverages the powerful new object representation of Neural Radiance Fields (NeRFs). NeRFs have recently emerged as a new representation of photorealistic 3D scenes constructed from RGB images alone. NeRFs are able to render synthetic photorealistic RGB images of the scene from camera view points not previously seen in the data.

The core hypothesis is that not only appearance, but also full object shape is accurately encoded in the NeRF representation despite not being explicitly labeled in the training images. Therefore, NeRFs can provide the basis for novel grasping and manipulation algorithms that operate on unlabelled images alone, without requiring accurate object mesh models.

The challenge is that this representation is nontraditional, as it represents the object as a volumetric density field without a distinct object "surface," and therefore does not immediately lend itself to use with current grasping and manipulation algorithms. In this project, it is intended that fundamentally different ways of formalizing grasping and manipulation can be utilized by manipulation robots.

Furthermore, it is hypothesized that the NeRF density output in fact reflects a measure of uncertainty about the location of the true object surface. A sharp density transition indicates high confidence in the surface location, while a gradual transition indicates low certainty about the surface location. Therefore, developed grasping and manipulation algorithms will aim for robustness with respect to this measure of surface uncertainty.

Robustness to object shape and pose uncertainty is essential for precision robot manipulation in the wild, and is largely unsolved. Ultimately, using the NeRF as an object representation, it is intended that an algorithmic pipeline to go from pixels to grasp to manipulation trajectory will be developed and validated in the hardware.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.