CAREER: Brain-Machine Interfaces for Speech

Decoding what a person intends to say, through analysis of electrical signals recorded directly from the brain, has transformative potential to restore the ability to communicate through speech to those who have lost it. Neural speech decoders have been deployed along these lines in academic studies, but they are not yet good enough to displace non-invasive, low-tech alternatives: extremely high accuracy has been achieved only at the price of task complexity or generality.

This project aims to soften this trade off considerably by collecting much more data and making better use of it. This will entail the integration of recent ideas in machine learning, efficient experimental design, and data collection from a large pool of volunteers. The ultimate goal of the project is to enable implantable decoders for restoring speech to persons who have lost it through ALS, stroke, or other traumatic brain injury.

Modern machine learning relies on techniques (artificial neural networks) that scale very favorably with the amount of available training data, but intracranial recording (iEEG) data are scarce. Accordingly, the project is organized around a set of methods for increasing the amount of effective training data. A major focus is pre-training, including self-supervised learning, i.e., training models to solve "pretext" tasks involving iEEG data but no annotations, speech or otherwise; generative models for speech audio, trained entirely on audio waveforms, for iEEG-to-audio; and large language models (trained entirely on text) for iEEG-to-text.

A second major focus is transfer learning across multiple subjects; that is, training a single model on multiple subjects' data. At least two dozen participants from several collaborating hospitals and research groups are anticipated to provide data. The project also includes an educational component aimed at creating an interdisciplinary workforce in the areas of neuroscience and machine learning, and for making the problem of algorithm design for brain-machine interfaces accessible to a wider audience.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.