Collaborative Research:CIF:Small: Acoustic-Optic Vision - Combining Ultrasonic Sonars with Visible Sensors for Robust Machine Perception

Computer vision has primarily used optical information to understand the environment, from object detection to robotic localization and mapping. However, many animals in the natural world use multiple sensing modalities to explore their environments. In particular, hearing and acoustics have been used effectively with vision to enhance navigation in the wild.

Dolphins and bats both use sonar echolocation to help triangulate their position in visually degraded environments such as underwater or at night. Inspired by their mammalian sensing systems, this project investigates the fusion of optical and ultrasonic sensors for enhanced computer vision and computational imaging. The team of researchers will develop novel hardware platforms for performing joint optical and acoustic sensing along with the corresponding graphics and vision algorithms to support simulation and analysis of this multimodal data.

The resulting sensing platforms will be leveraged for applications in 3D scanning, material recognition, non-line-of-sight imaging, and robotic mapping. Further, the project features dissemination and education goals through incorporation of bio-inspired vision and acoustics into computer graphics and vision courses, undergraduate research, engagement with members of an industry-university consortium, and a middle-school outreach program.

The project will focus on three main objectives. The first objective will be to design new rendering algorithms that can jointly synthesize optical and acoustical information in an environment. These algorithms will be made efficient by the importance sampling of optical rays and sound waves in an environment and will inform sensor models and sampling criteria to recover the environment information.

The second objective will be to develop new sensing models that trade off sampling across optical and acoustic modalities. These systems will leverage multiple image sensors and acoustic transducers which will be prototyped in real hardware. The final objective will focus on the mutual benefit of combining vision and acoustics for robotics systems.

Applications in 3D sensing, material identification, non-line-of-sight/occluded imaging, and acoustic-optic robotic navigation will be investigated. Research insights will be disseminated to the robotics, computer-vision, acoustics, and computational-imaging communities through scientific publications and presentations as well as open-source software and hardware designs for reproducibility.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.