Cross-modality foundation models in Earth observation

In recent years, the efficacy of large-scale pretrained deep learning models, termed "foundation models", has been demonstrated in various downstream tasks.

However, their application to Earth observation still poses significant challenges due to the multi-modal nature of Earth observation data (multispectral, hyperspectral, and synthetic aperture radar).

Since the existing foundation models are primarily trained on RGB images from ground-level perspectives, they usually struggle to adapt to this complexity. To address this challenge, this project proposes to develop cross-modality foundation models in Earth observation.

By establishing a joint embedding space across different modalities and sensor types, the developed foundation models aim to enhance synergy among various sensors.

This advancement is expected to revolutionize the interpretation and analysis of Earth observation data, and thereby improve downstream applications like climate monitoring in the geoscience and remote sensing fields.