CIRC: Planning-C: Infrastructure for Natural Communication with Robots: Building the Retico Community

When two people speak with each other, it is common for whomever is speaking to say one word at a time. As the speaker speaks, the listener hears and understands what is being said as it is being said, word for word. In contrast, chatbots and spoken assistants, wait until a sentence is complete.

This project will help enable researchers to build systems (e.g., virtual agents or robots) to speak and understand word-by-word. The project team will develop modules that are important to the research community. A goal is also to create instructional tools so anyone can use the platform.

Summer internships will provide students opportunities to learn about the tools and platform. The project team will host a tutorial at an academic conference. The project tools and outreach activities will make it easier for people to talk to spoken dialog systems.

This project will help allow people to talk to virtual agents or robots. Another aim is to build a community of users. Dialogue systems on virtual or robot platforms must be modular, multimodal, and time-aligned.

Modules should process at word level (i.e., incremental). To fulfill these requirements, the research team will extend the Retico platform. We will also couple Retico with the Platform for Situated Intelligence.

The team has developed modules as part of the core platform for this project. The modules will include speech recognition, language understanding, language generation, and dialog management. The platform will also include modules for object detection, and many others.

The project will extend the platform with community input. This work will impact research on spoken dialog systems, robotics, and virtual agents. The project team and research community will develop and maintain the modules.

As part of this effort, we will develop instructional tools to help people learn about how to use the tools. The project will offer internships for students and host a tutorial session. We will also develop visualization tools to enable rapid research. Finally, we will make these tools available via Dialport.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.