EAGER:Using Network Analysis And Representational Geometry To Learn Structure-Function Relationship In Neural Networks

Over the past few years, neural networks have revolutionized the fields of computer vision and natural language processing and are now becoming commonplace in many scientific domains. Despite their successes, understanding how to design or build a neural network solution remains challenging and often results in a game of guess and check. This process is incredibly inefficient, and in the end, does not provide any insights into why a model is either good or bad.

Thus, new approaches are needed to characterize the relationship between structure (how the network is constructed) and function (how the network performs on a task) in neural networks, and use this information to design learning systems that are more efficient and stable. The overarching goal of this project is to develop tools to model the relationship between the structure and function of deep neural networks.

This project will generate a rich toolkit for extracting low-dimensional features from neural networks and will produce new insights that can be used to drive progress in the future design of systems capable of modifying their own architecture to adapt to new data streams.

Given the dimensionality of the problem, the discovery of compact (low-dimensional) representations and metrics that can adequately capture signatures of ``learning'' will be critical. When learning is unsuccessful, these metrics will be used to diagnose problems inherent to the network structure, such as its depth, width, and density of connections.

The first part of the project will use tools in network science to discover how concepts such as network sparsity or path diversity between inputs and outputs affect the network's learning performance and efficiency (e.g., the number of examples required to learn a modular task, or whether the network can learn continually without catastrophic forgetting). The second part of the project will develop tools to study how the geometry of representations formed within networks can be used to predict learning outcomes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.