Elements: Scaling MetPy to Big Data Workflows in Meteorology and Climate Science

MetPy is a Python-based software package for atmospheric science; it provides modern, well-tested, domain-specific software tools for reading data formats, performing calculations, and creating visualizations. MetPy builds upon an extensive set of community-developed scientific Python tools, and leverages technologies such as continuous integration, automated documentation generation, screencasts, and Jupyter notebook tutorials.

This project advances MetPy to address some current limitations in supported data formats, scalability, and run-time performance; it makes MetPy more suitable for working on much larger datasets, frequently encountered in climate science and ensemble-based modeling studies. Addressing these areas allows MetPy to continue to be a powerful tool for Python users in the atmospheric sciences for both small and large datasets.

When equipped with modern and well-engineered tools, researchers will be able to better utilize the large amounts of data available in a more time-efficient way.

This project has three main goals: enabling efficient access to big and small datasets, enabling access to cloud-based datasets, and creating training resources for using MetPy with big data. In tackling these challenges, the project is creating a benchmark suite for MetPy, to quantify the current performance of MetPy in important workflows as well as the performance improvements that occur as a result of further development.

Using performance profiling tools, bottlenecks in MetPy are identified and optimized using Python performance tools like Cython and Numba. MetPy is also being refactored to work better with the Dask library, which provides facilities for distributed computing in Python and would allow MetPy to work more effectively with large datasets. The World Meteorological Organization (WMO) has made GRIB (GRIdded Binary) its standard format for gridded model output, and BUFR (Binary Universal Form for the Representation of meteorological data) the standard format to encode meteorological observational data.

MetPy enhancements developed in this project will enable reading of additional datasets used in large cloud-based data holdings, such as GRIB and BUFR. All of this work will be featured in additional, freely available training materials in MetPy’s online documentation.

This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Physical and Dynamic Meteorology Program and the Division of Integrative and Collaborative Education and Research within the NSF Directorate for Geosciences.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.