CAREER: Inferring Minimal but Sufficient Environment Models from Natural Language and Semantic Perception for Collaborative Robots in Dynamic Environments

Collaborative robots are poised to have a transformative impact on society. Significant advances in perception, planning, mapping, localization, and actuation over the past decade have accelerated the migration of robots from controlled laboratory environments to diverse applications in a complex and dynamic human world. The ability to effectively communicate has become increasingly important as robots transition from specialized applications performed in isolation to significant roles in human-robot teams.

While smart voice assistants have become prevalent in recent years due to advances in automatic speech recognition, those interactions are not conditioned on a model of the environment inferred from sensor observations and do not lead to the physical manipulation of their surroundings. Recent advances in grounded language communication for human-robot interaction, where speech and text both inform and guide a robot's understanding of and actions in the world, have transformed the limited space of concepts that robots and humans could exchange into rich representations that could be composed of many different objects, spatial relationships, planning constraints, and actions.

Point solutions with bespoke environment representations or models tailored to specific applications may exist, but no current approach is able to efficiently interpret the meaning of human instructions across many scenarios over non-trivial timescales. Fundamentally new approaches to grounded language communication that fluidly reason about the past dynamics, present configuration and/or future state of objects from select observations are needed for collaborative robots to interpret linguistic interactions quickly and accurately in human-robot teams.

This Faculty Early Career Development (CAREER) project seeks to revisit algorithms, models, and intelligence architectures for collaborative robots and reformulate how probabilistic graphical models are constructed and solved for efficient grounded language communication in environments with complex dynamics, non-trivial affordances, and rich semantic representations.

This research will investigate this topic by developing algorithms that address two critical technical gaps. First, this research will develop algorithms that infer which past and/or present observations and/or future projections in time are needed to construct minimal but sufficient environment models for interpreting natural language. Second, this research will develop algorithms that iteratively update their solutions based on minor modifications of the environment model and augmentation of symbolic representations from incrementally updating observations and environment projections.

These algorithms will enable a novel intelligence architecture for natural language symbol grounding that reasons about which observations and/or environment projections and what kinds of information are needed from these dynamic environment representations to enable scalable and capable models for grounded language communication with collaborative robots. Experimental evaluation of these ideas will be performed on a variety of robot manipulators, field robots, and mobile manipulators in addition to purposefully built collaborative mobile robots that will be developed for teaching and outreach activities.

This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.