HCC:EAGER:Panoptic Studio: A Multiview System for Artificial Intelligence, Sign Languages, Linguistics, Education, and Social Interactions for Deaf and DeafBlind community members

This EAGER project explores the use of multiple cameras to record three-dimensional (3D) videos of social interactions and dialogues in American Sign Language (ASL). ASL is the third most used language in the United States, after English and Spanish. There are 500,000 deaf, deafblind, and hard of hearing (DDBHH) signers, and far more who learn ASL as a foreign language in school or college.

DDBHH people experience many educational, workplace, linguistic, health, economic and social disparities. There is a critical need not only for more data but also better data about use of American Sign Language by individuals, groups and communities across diverse communication settings such as home, health or the workplace. Communication and social interaction among people who use ASL is hindered by a lack of three-dimensional video about DDBHH signers’ communication and interaction in these settings.

This research is expected to have strong benefits and impact how people communicate with DDBHH signers.

This proposal aims to pool institutional resources for both equipment and personnel to create larger, more systematic 3D video datasets of social interactions for collective use by researchers and community members. This research establishes an infrastructure to collect this critical data that can be shared. This consolidation of resources, including pooling of resources and personnel, can eliminate the time and resources that individual researchers spend on compiling small, idiosyncratic, one-off datasets for their personal use.

The pooled data will be higher quality and more durable, and facilitate reproducibility and replication of key research results, leading to better, more reliable science and to leverage these datasets to build the intellectual, social and developmental capital for the DDBHH communities and American Sign Language.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.