NSF Convergence Accelerator Track H: Next Generation Augmentative and Alternative Communication Technology Powered by Artificial Intelligence

Communication is a basic human need, a fundamental human right, and an elemental human power. It is through communication that all individuals participate and affect change within their daily lives. For millions of Americans, communication is restricted by limitations in speech and language.

These restrictions can include a lack of speech, speech that most communication partners do not understand, and limited language understanding. Individuals with communication limitations are severely restricted in their participation in life, including in educational, vocational, and social activities. Augmentative and alternative communication (AAC) devices aim to help people with speech and language limitations express their thoughts, needs, and ideas.

Despite the evidence that AAC devices can improve social interaction with family, friends, and communication partners; participation in educational contexts; and participation in vocational life, the existing technology has yet to reach its full potential. The challenges include teaching users to interact with AAC devices, personalizing the devices to fit the communication needs of each user, and making communication as effortless as possible.

We hypothesize that by leveraging recent advances in Artificial Intelligence (AI), we can enhance AAC technology and redefine how the users interact with it, ultimately achieving a transformative impact on the lives of many people with communication limitations and their communication partners, including their families.

This convergence project aims to spearhead a new generation of AI-powered AAC technology through a convergent and multidisciplinary effort involving experts in AAC, AI, human-computer interaction, and software design, as well as AAC end users, caregivers, and stakeholders. The AAC embodiment studied in this project will be a prototype of an AAC application for mobile touchscreen devices.

The envisioned AI-powered AAC application will be able to (1) ingest and analyze multi-modal contextual information about the user and their environment to provide personalization, (2) produce dynamic grids and visual scene displays informed by best practices in AAC and inclusive design and driven by AI to maximize communication efficiency, (3) include a virtual conversational partner able to guide the AAC user, and (4) support virtual activities aimed at encouraging frequent AAC use and promoting language development. Through a collaborative, iterative process, the research team will gather stakeholder input, conduct a use-inspired exploration of AI-powered AAC concepts, and evaluate the concepts through rapid prototyping and user testing.

Improved communication support would result in better educational outcomes and increased participation in the workforce for individuals with communication limitations, thus leading to a more abundant society. This project stands to support a more equitable world for the millions of Americans with communication limitations who benefit from AAC. The project team will interact with technology transfer experts and AAC industry leaders to lay out the best path towards developing and deploying the next-generation AAC devices and putting them into the hands of end users.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.