Adaptive Neural Networks for Partial Differential Equations

Neural networks have achieved astonishing performance in computer vision, natural language processing, and many other artificial intelligence tasks. Despite their great successes in many practical applications, it is widely accepted that approximation properties of neural networks are not yet well-understood. This project would have the potential to understand why and how neural networks work, particularly, why a neural network is much better than other functional classes for computer simulations of problems with singularities and discontinuities.

The self-adaptive neural network method developed in this project would have the potential to dramatically improve the computational capabilities for computer simulations of complex physical, biological, and human-engineered systems exhibiting computational difficulties. Understanding of the role of neural network width and depth in approximation and the new ideas gained in this project would have significant impacts on many other artificial intelligence tasks such as transfer learning, online control, and pattern recognition. Training of at least one graduate on the topics of the proposed research is expected.

A fundamental, open question in machine learning is on how to design the architecture of neural networks, in terms of their width and depth, in order to approximate functions or numerically solve partial differential equations accurately and efficiently. Addressing this issue for applications to computationally challenging problems is the focus of this project.

More precisely, for any given partial differential equation, this project is going to develop an adaptive neuron enhancement (ANE) method that adaptively constructs a neural network with a nearly minimum number of neurons and parameters such that its approximation accuracy is within the prescribed tolerance. A key component of developing ANE methods is an error indicator for determining the number of new neurons to be added at either the current or next layer.

This project plans to develop efficient indicators by studying the role of additional neurons in the current and next layers. This knowledge is crucial for designing efficient and accurate ANE methods, but is generally not provided by the standard a priori error estimates. The exceptional approximation powers of neural networks come with a price: the procedure for determining the values of the parameters is now a problem in nonlinear optimization even if the underlying partial differential equation is linear.

Nonlinear optimizations usually have many solutions, and the desired one is obtained only if one starts from a close enough first approximation. The ANE method to be developed in this project is a good continuation process. This project will focus on how to initialize parameters of new neurons at each adaptive stage of the ANE method. This will be done by analyzing the physical meanings of neurons at each layer.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.