CRII: HCC: Modeling computer-mediated task-oriented dialogues with multi-modality information theoretic approaches

Effective real-time remote communication is challenging because unlike face-to-face conversations, information is exchanged indirectly through computer screens and speakers. In these situations, rich non-verbal information such as body poses, hand gestures, and facial expressions is often blocked or neglected. In order to improve the quality of existing remote communication systems, this project will develop an in-depth quantitative understanding of the kinds of information non-verbal aspects of communication might add to the conversation.

The research team will address this issue by developing novel measures of non-verbal communication content based on state-of-the-art machine learning and computer vision techniques. The team will also develop software toolkits that can support both behavioral scientists who study conversation and designers of future computer-mediated communication systems.

The research team proposes a series of studies that use information theoretic models to distill information from the non-verbal communication channels, including facial expressions, upper-body gestures, and whole-body poses, and synthesize them into the ultimate goal of predicting the success of computer-mediated collaborative tasks. As part of the work the team will develop several novel analysis methods.

These include discretizing and symbolizing non-verbal information through joint representation learning and aggregation, using neural sequential models to estimate the amount of information in parallel discretized series, and using the temporal-spectral patterns in all communication channels to predict the task success. A multimodal information incorporated corpus of task-oriented dialogues will also be collected as part of the work.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.