Interpersonal behavioral synchrony in virtual and in-person dyadic conversation

PROJECT SUMMARY/ABSTRACT Human dyadic social communication entails a rich repertoire of expression, including not only face expression (and gaze), but also acoustics (prosody and pauses) turn-taking, gestures and language. Communication has evolved in humans within a social context, beginning with the parent-infant dyad, with mirroring of facial

expressions and sounds. Its natural ecology is face-to-face dyadic interactions, both in-person and increasingly via remote platforms for teleconferencing and telehealth. Social communication is a “complex orchestration” in real time: its signals are multiple and temporally offset. It is a continuous exchange that is highly coordinated

between speakers, with norms for turn-taking and alignment of face expression, gesture, semantic content and speech rates. As yet, a critical gap exists in that we lack the tools to quantify and analyze temporal patterns of multimodal communication behavior between two individuals in face-to-face communication, in an ecologically

valid setting, that have the same rigor and reproducibility as do hyperscanning approaches to record brain activity during dyadic conversation. This tool must be developed to realize the true potential of second-person neuroscience. This planning proposal for tool development entails several key activities, beginning with the

convening of a diverse multidisciplinary team of experts from various fields, including ethics/regulatory, anthropology, cognitive neuroscience, computer science, engineering, physics, mathematics, psychiatry and neurology. This team will discuss ethics, diversity, paradigm development, and computational frameworks, and

providing iterative feedback and convening also with advocacy groups. Also, we will build two testing rooms for multimodal recording of dyadic communication, to demonstrate feasibility of acquiring and synching high temporal resolution data. Pilot EEG hyperscanning will be done concurrently in a subcohort. Further, given

increased use of teleconferencing, dyadic communication data will be collected via remote platform and compared with in-person data, to determine how information may be degraded by differences in resolution and streaming delays. We will also develop computational frameworks for analyses of multimodal data.