Exploring How Scene Segmentation Circuits Aid in Predicting Future Visual Input

PROJECT SUMMARY/ABSTRACT The dynamics of typical visual input challenge the brain: movement of objects or caused by navigation shifts visual information over the retina, and complicates the brain’s goal of understanding visual input as an organized collection of objects. Visual scene segmentation occurs in the ventral - ‘what’ - stream in the primate

brain, hence prior studies have studied these circuits mostly under static conditions, since motion is thought to be mainly processed elsewhere, in the dorsal – ‘where’ – stream. These prior studies found that the earliest surface segmentation signal in the primate brain is selectivity for border ownership. Border ownership neurons

respond differently to an identical border in their receptive field depending on which side of the border is owned by foreground. It is poorly understood how these circuits contribute to the processing of the dynamic, complex visual scenes in everyday environments. Recently the applicants found that border ownership units with similar

properties as those in the brain emerge in an artificial neural network trained to predict the next frame in natural videos, even though the network was not trained to segment objects. This indicates that these units aid in predicting future input. The overarching goal of this proposal is to understand how border ownership units

contribute to this objective. Specific aims are to understand 1) the stimulus diet that drives the emergence of border ownership units in artificial neural networks (Aim 1); and 2) how these units benefit the prediction of future input in natural videos (Aim 2). To this end the applicants will perform experiments in artificial neural

networks (in silico) as well as in the non-human primate brain. This exploratory research may thus lead to a paradigm shift in the understanding of scene segmentation circuits, which traditionally have been assumed to perform a function typically considered under static conditions: segmenting objects from background. This

research addresses several research needs recognized by the National Eye Institute, including exploring the connections between biological measurements and theoretical models of vision processes, and understanding processing in higher brain areas to inform the design of next-generation cortex prostheses. Moreover, it may

lead to better diagnostic and therapeutic tools for disorders characterized by disrupted perception of complex visual input, such as visual agnosias and schizophrenia. This project is a collaboration between Dr. Franken (PI), an electrophysiologist with expertise in border ownership circuits in the primate brain, and Dr. Wessel (co-

I), a neurophysicist with expertise in studying visual processing (including natural videos) in a variety of systems, including in artificial neural networks. The experiments will break new ground and extend previous work in a new and promising direction. Our ultimate goal is to understand why neural signals that always had

been assumed to segment objects emerge in networks that have a fundamentally different objective – predicting future input. Thus this research is high-risk high-reward. For these reasons, the proposal is closely aligned with the purposes of the R21 funding mechanism.