Collaborative Research: OP: Meta-optical Computational Image Sensors

In modern daily life, cameras are indispensable, and they truly serve an excellent purpose to capture a scene as perceived by a human eye. Digital photography became a disruptive technology when it was first introduced almost 30-years ago. From that time, cameras have undergone dramatic miniaturization.

With these cameras readily available to consumers, professionals and hobbyists are able to experience how easily a photo can be captured, viewed, and shared. But many emerging applications in machine vision, robotics or internet of things require ever more advanced (smaller, lower power and intelligent) cameras. These cameras are expected not just to capture images, but also to provide information on how a machine must function, like for example in autonomous navigation.

For this type of scene-understanding or object-detection problems, current systems employ bulky cameras combined with a computer or graphical processing unit. Unfortunately, most of these systems consume significant amounts of energy, and often are not optimized for specific tasks. By co-designing the hardware and software together, this project aims to create computational machine vision sensors, capable of low-power, low-latency operation and compact in size.

The resulting sensors can revolutionize the field of autonomous navigation and machine vision. Furthermore, this project will improve the training and education of undergraduate and high school students in multi-disciplinary research in optics and machine learning. Through the PI’s active involvement with industrial laboratories working on automotive, imaging and augmented reality visors, the scientific results will be disseminated to a wider scientific audience via seminars, workshops, peer-reviewed publications, and conferences.

There is a tremendous need for compact, low-power, and ubiquitous image sensors for applications in autonomous transportation, smart homes and cities, and the Internet of Things. Many of these machine vision applications require an electronic back-end to interpret the captured images or need more information than just the two-dimensional intensity information usually captured in cameras.

Current approaches for solving these problems employ high-end, bulky cameras to capture high-quality images and then exploit computationally expensive and power-hungry computer vision algorithms. Both the size and power consumption of these imaging systems can be drastically reduced via co-optimizing the optics and computational imaging algorithms for specific applications, including depth sensing and directly solving higher-level computer vision tasks such as object segmentation, detection, and classification.

This project aims to research and develop such a co-optimization algorithm for an optical front-end and complementary computational back end. The optical elements are implemented via high-efficiency dielectric meta-optics, where each scatterer constitutes a design parameter. Combining numerical simulation, device fabrication, and optical characterization, this project aims to develop an inverse design framework for optimizing the sensor’s meta-optics; expand the design framework to co-optimize both the meta-optics and computational algorithms without placing prohibitive constraints on intermediate representations, as well as fabricate and characterize the meta-optical sensors for 3D imaging and object detection.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.