CAREER: Visual Manipulation Learning for Challenging Object Grasping

This Faculty Early Career Development (CAREER) project seeks to significantly enhance the capabilities of robotic systems to grasp objects. Object grasping is an important prerequisite for various manipulation tasks. Humans are capable of grasping diverse objects dexterously even if their workspaces are cluttered.

When an object is in a constrained space, such as a cardboard box or a shelf, humans often take advantage of the constraint by pushing the object toward a wall or a corner to grasp it. Even if a targeted object is hidden in a pile of objects, humans actively search for it by removing the clutter objects. While object grasping in such challenging scenarios seems effortless and natural for humans, current robots are still grasping objects in moderately cluttered tabletop environments.

This project will develop computational algorithms to allow robots to perform such challenging object grasping. This project has the potential to broaden the application domains of robotic manipulation, such as flexible manufacturing (e.g., supporting human worker by providing right parts), agriculture (e.g., automated fruit and vegetable harvesting), warehouse fulfillment centers or grocery shopping (e.g., pick-and-place ordered items in shelves or bins), and eldercare (e.g., fetching a remote controller or pills), which have not been feasible with current robotic labor.

The objective of this project is to develop novel computational algorithms that enable robots to visually understand scenes, learn to perform a proper manipulation action sequence, and adapt to different environmental settings. This project will address three fundamental challenges in robotic object grasping: (1) context-aware object grasping -- considering spatial contexts related to grasping, such as clutteredness, reachability, and collision, (2) object grasping via leveraging fixtures -- making use of environmental fixtures to grasp challenging objects by learning pixel-level or object-level affordances, and (3) object searching and grasping -- searching and grasping a hidden target object queried by either a reference image or a human natural language.

All the algorithms and systems are designed to be self-supervised, meaning that they can learn and adapt to novel objects, environments, and tasks with minimal human interventions or guidance. This project will lay a solid foundation for the affordable and reliable robotic labor beneficial to the broad areas of our society, such as homes, factories, farms, warehouses, and eldercare facilities.

This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.