CAREER: Cognitively-Informed Memory Models for Language-Capable Robots

Robots that can communicate with people through spoken language stand to advance the future of human work and to assist the most vulnerable members of society, including children and older adults, people with disabilities, autism, or mental illness, and people experiencing isolation, bullying, or trauma. One of the key tasks that robots will need to do when talking with everyday people is referring expression generation, which is the process of creating descriptions like "the office at the end of the hallway." When robots generate such descriptions, they need to do so in a way that is accurate (the description shouldn't be wrong), natural (the description shouldn't sound awkward), understandable (the listener should be able to interpret the description quickly and effortlessly), and efficient (the robot should be able to generate the description without having to pause and think for too long).

To understand how robots might generate descriptions in a way that satisfies these properties, we can start by trying to understand how people do so. One reason we are good at generating referring expressions may be because of our working, or short term, memory, which we use to keep a small amount of timely and important information available in a way that we can quickly and effortlessly access.

The key idea of this project is to give robots the same type of working memory capabilities, and the same ways of thinking about what might be in peoples' working memories, so they will be able to use that timely and important information to do a better job at generating referring expressions. By taking this cognitively inspired approach, this work will advance the state of the art of multiple fields, including AI, robotics, and psychology.

In addition, the educational aspect of this project aims to develop materials that will help train the next generation of students working at the intersection of these fields. To ensure the broadest possible impact, these efforts will be integrated with the PI's department's activities relating to Broadening Participation in Computing so that they reach currently underrepresented groups.

From a technical perspective, the key goal of this research is to show how models of working memory that appropriately cache task-relevant beliefs about goal-relevant objects will enable robots to better perform referring expression generation. To this end, the work will assess two key hypotheses: that cognitively inspired models of working memory will enable robots to generate referring expressions in a way that is more accurate, natural, computationally efficient to generate, and cognitively efficient for the listener to process; and that goal relevance can be leveraged to ensure that the most task-relevant information is retained within those models.

By addressing these hypotheses, the research will develop: (1) the first algorithms for referring expression generation in robot cognitive architectures that are informed by current psychological theories of human working memory; (2) a fundamental new understanding of how robots can intelligently manage and allocate resources within artificial working memory models, (3) an understanding of which memory models will produce optimal performance from both robotics and cognitive modeling perspectives; (4) fundamental new understanding of how the goal relevance of entities and their properties can be automatically assessed within integrated cognitive architectures; (5) understanding of how goal relevance can be used to allocate cognitive resources within robotic models of working memory; (6) understanding of which goal-driven resource allocation strategies will produce optimal performance from both robotics and cognitive modeling perspectives; and (7) freely-available datasets of human-robot dialogues, and a freely-available experimental framework to allow other researchers to collect additional such dialogues, both of which will be permanently archived via the Open Science Framework.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.