Breaking new ground in augmentative and alternative communication: developing wearable technology for automatic collection and analysis of language data using artificial intelligence

Project Summary In the US, over five million individuals, including 450,000 children, lack or have limited language and speech abilities and could benefit from Augmentative and Alternative Communication (AAC). Evaluating and monitoring the language performance of these individuals when using AAC is crucial for providing the appropriate

intervention to improve language and communication functions. However, traditional language sample analysis (LSA) tools and procedures do not adequately capture language samples generated by AAC devices, such as tracking key presses and timings. Additionally, the time-consuming nature of the traditional LSA process hinders

the motivation of speech-language pathologists to perform regular LSA evaluations over time. Therefore, there is a critical need to build an easy-to-use and objective AAC language sample collection and analysis (ALSCA) system that includes a wearable finger ring system and an AI-implemented data portal to efficiently automate

the AAC data collection and analysis process. Without such a system, the promise of AAC intervention for those who need it will remain limited. Our central hypothesis is that automated AAC data collection and analysis using the ALSCA system will provide valid and sufficient expressive lexical measures of AAC language samples to

build more extensive databases to guide the development of future intervention strategies. We will evaluate the central hypothesis with three aims. In Aim 1, we will develop and test a smart finger ring using an inertial measurement unit (IMU) and computational methods for automatic data collection and classification. In Aim 2,

we will refine a pilot AAC automatic speech recognition model using a dataset from various AAC speakers for automatic AAC data transcription. In Aim 3, we will evaluate the validity and reliability of the analytic result of the ALSCA data portal. This innovative project will be the first study integrating an IMU finger ring, machine-learning

algorithms, an ASR model, and a data portal that allows SLPs and researchers to collect automated, valid, reliable, and meaningful data from AAC users to develop and test targeted interventions. This project also aims to automate the language sample transcription and analysis process, freeing clinicians and researchers from

tedious and time-intensive LSA tasks. Successful development of the project could significantly advance AAC LSA and enable clinicians to provide meaningful, timely interventions that appropriately address individual AAC users’ needs without investing time in the tedious LSA steps. Furthermore, this project aims to develop an

efficient AAC data collection and analysis solution to facilitate AAC studies and address crucial knowledge gaps in language and communication development and rehabilitation in AAC populations.