CRCNS": Brain-derived network architectures for deciphering and applying brain algorithms

Our LONG-TERM GOAL is to extend current biological and artificial vision research from a focus on 2D image recognition toward visual understanding of 3D structure in the real world. 3D object perception is the essence of real-world vision, underlying the “thousand words” of information the brain generates about

precise object geometry on large and fine scales, structural design, mechanics, material composition, biological morphology and functionality, physical state, pose, mass distribution, balance/support against gravity, potential for movement from passive falling/rolling to self-generated motion and complex interactive

behaviors, age, beauty, damage, value, etc. Our first AIM is to use artificial vision networks to decipher brain algorithms underlying biological 3D vision. We will train novel, analyzable network architectures developed for 3D vision by the Yuille lab, to replicate 3D shape tuning functions of individual neurons recorded in the Connor lab, from successive

stages of object processing in macaque monkeys: area V4, posterior inferotemporal cortex (PIT) and anterior inferotemporal cortex (AIT). The Connor and Yuille labs will analyze the 3D shape processing algorithms learned by these neuron-trained networks (NTNs), tracing the pathways of information from V1-

like 2D Gabor filters in layer 1 to the output neuron response, using a novel method for back-tracing the sources of excitatory and inhibitory signals through the network.

Our second AIM is to develop artificial networks that implement biological algorithms to achieve humanlevel 3D visual performance. The Yuille lab will build on preliminary work proving the potential of novel network architectures performing analysis-by-synthesis with Neural Textured Deformable Meshes

(NTDMs). These networks are designed for internal reconstruction of and high-level inference from 3D shape in natural scenes. The Connor lab will use components of these networks, in addition to more standard architectures like AlexNet, to train using neural responses (AIM 1). The trained network components and algorithms deciphered from them (AIM 1) will be incorporated by the Yuille lab into their

evolving NTDM network designs to search for higher performance on larger 3D shape domains under a variety of real-world viewing conditions. Training and testing in larger 3D shape domains will depend on a unlimited photorealistic 3D ground truth-parameterized stimuli produced by genetic algorithms developed

by the Connor lab (PRGAs).