Active Social Vision: How the Brain Processes Visual Information During Natural Social Perception

Summary/Abstract The dominant paradigm in social visual neuroscience has been to present simplified stimuli, often consisting of static images in isolation presented without context, passively, to subjects required to maintain gaze fixation. While much has been learned using this paradigm, it cannot capture all aspects of how social vision (SV)

works. Specifically, active observers combine motoric information gathering behaviors (eye and head movement) to optimally sample perceptual information in a complex, natural multisensory social environment. Indeed, there are few places where active vision is more critical than in the study of the neurobiology of SV. Real world SV is a

prototypic active sensing process where we use our experience, along with all of our sensing and interpretive capacities to gather social and affective information from other people in a complex and dynamic setting. Simply put, SV, as typically studied, does not adequately approximate actively interacting with friends, family, or your

doctor. Recent advances in computer vision, machine learning, and computational analysis now provide means to attack a critical goal of social neuroscience: how the social and affective system guides active information gathering in natural social contexts. We propose a novel natural approach to studying SV.

This project will leverage a powerful technique to measure activity in the human brain: direct recording from electrodes implanted in the brains of surgical epilepsy patients. Surgical epilepsy patients who spend 1-2 weeks in the hospital while having their brain activity monitored afford the unique opportunity to record

multiscale neural activity during natural interactions with friends, family, doctors, nurses, experimenters, etc. In addition, patients will play a social game to allow for the study of real world SV using a semi-controlled and repeatable task. Furthermore, participants will engage in more standard laboratory SV paradigms to assess how

results from real world conditions compare to the results from traditional experiments. The neural recordings will be acquired simultaneously with video and audio monitoring, and eye tracking. State-of-the-art computer vision analysis will provide a continuous assessment of social cues (e.g., eye gaze and facial expression) from

people with whom the patients are interacting. Computational analyses and mechanistic neural measurements will determine the correspondence between what is perceived by the patient, the neural coding, and the neurobiological implementation of that code. This approach will allow us to bridge across three critical levels of

analysis required for understanding the SV information processing systems. We hypothesize that real world SV is an extended, iterative process that combines active, dynamic motor/attentional sampling strategies with prior and contextual information to actively plan information gathering, which both enables and constrains the flow

of information through the system. Our results will have fundamental implications both for basic social neuroscience and for our understanding of how these processes may be altered in disorders of SV, such as mood and anxiety disorders and schizophrenia.