OAC Core: The Best of Both Worlds: Deep Neural Operators as Preconditioners for Physics-Based Forward and Inverse Problems

Many physical systems across all areas of science, engineering, medicine, and defense are modeled with high accuracy by partial differential equations (PDEs) and solved on advanced computing systems. Often the ultimate goal is to repeatedly solve the PDEs to explore parameter uncertainties. Settings in which this arises are inverse problems (inferring uncertain parameters of a model from data), optimal experimental design (determining the optimal data acquisition to learn the most about the model), optimal design (finding the optimal configuration of a system to maximize performance), and optimal control (determining the optimal operation of a system to achieve a desired behavior).

These problems are often characterized by high dimensional uncertain parameter spaces, since the parameters typically represent initial conditions, boundary conditions, material properties, or source terms and vary in space and/or time. As a result, the PDEs often have to be solved thousands or even millions of times to adequately represent uncertainties in the parameters.

When the systems that are modeled involve coupled multiple physics or behavior occurring on multiple space and time scales, repeated solution of the PDE models becomes prohibitive, even on the latest supercomputers. The development of deep neural networks in recent years shows promise in overcoming the intractability of repeated solution of the PDE models, by learning the relationships between the input parameters and the outputs of interest (e.g., temperature, velocity, pressure, stress, electric field, magnetic field, chemical species).

Once trained on PDE solution data, the networks can evaluate the outputs for any given inputs in milliseconds, compared to hours or days to solve the PDE models themselves. However, despite much progress in the development of these so-called neural network surrogates, they typically deliver just 1-2 digits of accuracy, which is not sufficient to replace the PDE solver.

Instead, this project is developing hybrids of neural network surrogates and PDE models that combine the best properties of each: the accuracy of the PDEs with the speed of the neural networks. The impact is that many problems in technology, health, the environment, and society that were not amenable to complex model-based inference and decision making will now become tractable.

The algorithms developed in this project are being released as open-source software so that a broad community of researchers and practitioners can apply them to a spectrum of scientific and engineering problems. In addition, the surrogate methods developed in this project are being incorporated into a popular graduate course on inverse problems taught at University of Texas, Austin.

Neural network approximations of high fidelity PDE solutions, i.e., neural operators, have gained popularity in recent years due to their ease of implementation, adaptability to varied settings, and seeming ability to mitigate the curse of dimensionality. Significant recent research has attempted to establish "universal approximation" properties of these surrogates for various classes of maps.

While theory suggests that neural operators can in principle achieve arbitrary accuracy, realizing this in practice remains a significant challenge. The reasons for this include the enormous costs of generating sufficient training data, and confounding relations between statistical sampling errors, approximation errors, and nonconvexity of the training problem.

Often neural operators can achieve just 1-2 digits of accuracy relative to high fidelity PDE solvers, with little hope of further reducing this accuracy. On the other hand, high fidelity PDE models (particularly conservation and balance laws) are often known with very high confidence and high precision is necessary due to sensitivity of PDE solutions to small perturbations in the inputs.

The modest accuracies of neural operators are often insufficient for the demands of inference, control, and decision making for critical systems. This project is developing hybrids of neural operators and high fidelity PDE models to realize the best features of each, by retaining accuracy via the PDE residual and speed via use of the neural operator as a preconditioner.

The project targets linear and nonlinear parametric neural preconditioners for PDEs, and neural preconditioners for Metropolized Langevin methods to accelerate the solution of Bayesian inverse problems. A further advantage of using neural operators as preconditioners is that they map well onto GPU architectures.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.