Collaborative Research: Frameworks: Convergence of Bayesian inverse methods and scientific machine learning in Earth system models through universal differentiable programming

Understanding and quantifying parameter sensitivity of simulated systems, such as the numerical models of physical systems and mathematical renderings of neural networks, are essential in simulation-based science (SBS) and scientific machine learning (SciML). They are the key ingredients in Bayesian inference and neural network training. Seizing on the opportunity of emerging open-source Earth system model development in the Julia high-level programming language, this project is endowing these open-source models with automatic differentiation (AD) enabled derivative information, making these converging data science and simulation-based science tools available to a much broader research and data science community.

Enabling a general-purpose AD framework which can handle both large-scale Earth system models as well as SciML algorithms, such as physics-informed neural networks or neural differential equations, will enable seamless integration of these approaches for hybrid Bayesian inversion and Bayesian machine learning. It merges big data science, in which available data enable model discovery with sparse data science, and the model structure is exploited in the selection of surrogate models representing data-informed subspaces and fulfilling conservation laws.

The emerging Julia language engages a new generation of researchers and software engineers, channeling much needed talent into computational science approaches to climate modeling. Through dedicated community outreach programs (e.g., Hackathons, Minisymposia, Tutorials) the project team will be working toward increasing equity, diversity, and inclusion across the participating disciplines.

The project is developing a framework for universal differentiable programming and open-source, general-purpose AD that unifies these algorithmic frameworks within Julia programming language. The general-purpose AD framework in Julia leverages the composability of Julia software packages and the differentiable programming approach that underlies many of the SciML and high-performance scientific computing packages.

Compared to most current modeling systems targeted for HPC, Julia is ideally suited for heterogeneous parallel computing hardware (e.g., CUDA, ROCm, oneAPI, ARM, PowerPC, x86 64, TPUs). The project is bringing together expertise in AD targeted at Earth system data assimilation in high performance computing environments with SciML expertise. The project team is working with the Julia Computing organization and package developers to ensure sustainability of the developed frameworks.

The project’s Earth system flagship applications consist of (i) an open-source, AD-enabled ocean general circulation model that is being developed separately as part of the Climate Modelling Alliance (CliMA), and (2) an open-source, AD-enabled ice flow model. Each of these application frameworks is being made available to the community for science application, in which derivative (gradient or Hessian) information represent key algorithmic enabling tools.

These include SciML-based training of surrogate models (data-driven and/or model-informed), parameter and state estimation, data assimilation for model initialization, uncertainty quantification (Hessian-based and gradient-informed MCMC) and quantitative observing system design. Academic and industry partners are involved, who are using the frameworks for developing efficient power grids, personalized precision pharmacometrics, and improved EEG design.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.