CAREER: Adaptive Multimodal Fusion for Robust Robot Perception in Underwater Environments

This Faculty Early Career Development Program (CAREER) project will support research that attempts to provide new capabilities for marine robotic systems to map, visualize, and navigate underwater sites autonomously; that is, without the need for direct control by a human operator. This will advance scientific knowledge, bolster maritime defense, and strengthen the economy.

These new capabilities will be achieved using innovative machine learning methods to simultaneously capture the robot’s surroundings and pinpoint the robot’s location by combining acoustic and visual information. Project outcomes will be demonstrated for use in archaeology by mapping historically significant shipwreck sites. The results will also apply to tasks such as underwater construction, infrastructure maintenance, and emergency response.

The underwater environment is a challenging workplace for robots and humans. Most electromagnetic radiation is absorbed or scattered over short distances, precluding reliance on satellite navigation signals or ambient illumination. Other effects due to water movement or variable transmission properties can further distort interpretation of sensor measurements.

Because different types of signals respond differently to these effects, the fusion of diverse imaging modes such as light and sound offers the promise of improved mapping and localization error, while also reducing the amount of data required. Reducing or eliminating the need for a human pilot allows allocation of human expertise to higher-value tasks.

This project will develop virtual resources for K-12 education, university curricula, and public outreach platforms, making marine robotics research and education accessible to a broader population. Software and datasets developed throughout this work will be published to foster innovation in robotics for ocean exploration.

Marine environments pose unique challenges to robot perception due to the complex nature of underwater sensing across multiple modalities. Imagery collected with cameras provides high resolution observations at close range but can be highly degraded due to the effects of underwater light propagation. This results in loss of information about the scene, especially at longer ranges.

Sonar can provide long-range sensing capabilities but has low resolution and viewpoint-dependent appearance. While acoustic and visual sensors appear complementary, multi-sensor fusion is non-trivial due to the differences in geometry and appearance across these modalities. This research project investigates implicit neural representations for multimodal fusion of underwater imagery and acoustic data.

This project will contribute new knowledge in adaptive multimodal fusion and uncertainty estimation for deep learning to leverage the learned implicit neural representations for real-time simultaneous localization and mapping (SLAM). The developed methods will be demonstrated in real underwater environments through the task of autonomous real-time dense mapping of shipwreck sites in Thunder Bay National Marine Sanctuary.

This work will be broadly applicable to survey and inspection of complex marine structures beyond shipwrecks, including offshore energy infrastructure, pipelines, and underwater caves. Knowledge developed through this project will be transferable to mobile robots operating in challenging terrestrial environments, such as in poor weather conditions or GPS-denied environments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.