Collaborative Research: CIF: Medium: Analysis and Geometry of Neural Dynamical Systems

The complexity of modern neural nets, with their millions of parameters and unprecedented computational demands, has been a major hurdle for the conventional approaches which had been successfully applied in machine learning over the past decades. This project aims to develop new mathematical and computational foundations for the analysis and design of these systems through a radically new conceptualization of their architectures as continuous dynamical systems.

The key pillar of this framework is the idealization of depth as a continuum of layers and width as a continuum of neurons. Infinitesimal abstractions of this type have successfully unlocked many disciplines throughout the twentieth century, including probability, optimization, control, and many more. This collaborative project involving UIUC and MIT will push the boundaries of the theory and practice of deep learning, while sparking sustained interactions between the communities of electrical engineering, mathematics, statistics, and theoretical computer science.

The project will also have broad impacts through a deliberate approach to education and training. The education and outreach activities will include research opportunities for undergraduate students at both institutions, as well as an exchange program to foster the collaboration and exchange of ideas.

This project on Analysis and Geometry of Neural Dynamical Systems is developing the mathematical foundations of deep learning by synthesizing tools from probability, statistics, dynamical systems, geometric analysis, partial differential equations, and optimal transport. The research program is articulated around three major directions: (1) continuous models of neural dynamical systems; (2) discretization schemes; and (3) algorithms.

The first direction is focusing on characterizing the tradeoffs between the expressive power and complexity of idealized infinitely wide and deep neural nets. The second direction builds on these continuous abstractions to develop, from first principles, mathematically rigorous and practically implementable techniques for analyzing large but finite neural nets.

The third direction emphasizes algorithmic and computational aspects, such as the computational complexity of numerical methods, stability, and implicit regularization, using a novel synthesis of analytic and geometric methods developed as part of the project.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.