HNDS-I: A Collaborative Annotation System for Language Research

Human communication happens both when we speak to each other and when we write things down. Researchers studying how people communicate can easily obtain many examples of written material. Written material is also easy to analyze because it is easy to scan it for computer use or to download it from the Internet.

It is far more difficult to do research on spoken language because it is much harder to record speech and convert those recordings into a format a computer can use, especially speech that occurs in natural situations such as a family conversation around a dinner table or an argument between children in a classroom. This project will build a new system that will let researchers and students examine recorded conversations and provide additional data about details of those conversations, such as the talkers’ emotions, that cannot otherwise be measured.

The new system to be built will be part of the TalkBank system at Carnegie Mellon University, the largest publicly accessible databank for the study of spoken language in the world. TalkBank includes audio and video recordings from children, classrooms, bilinguals, second language learners, and persons with language disorders in 34 different languages, together with written transcripts of those recordings.

The new system will allow scientists and researchers to study human communication by looking at patterns of speech in individual conversations, which is a level of detailed analysis that is not currently possible. Studying communication at the conversation level is important because fine details in speech patterns can help answer important questions such as how people resolve confusion and misunderstandings as a conversation unfolds.

The new system will be based on a concept called Collaborative Commentary, which allows research groups and students to add comments about important conversation details, such as talkers’ motivations, directly into the database by adding links to the conversation transcript. This same system will also permit users to provide comments on media that has not yet been transcribed and create new transcripts through a web browser interface.

This project will engage the public with the data through Citizen Science and provide a resource for teaching about language and communication through guided commentary. Completion of this work will open up a remarkable new window to the study of spoken language and communication.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.