RI: Small: Physically-Based Learning for Shape, Lighting and Material in Complex Indoor Scenes

Indoor scenes involve visible and invisible light sources interacting with spatially-varying materials and shapes, together with interreflections and shadows, to produce images that display complex local variations. Applications such as augmented reality require editing an indoor scene to insert new objects, changing materials in some parts of the scene, or visualizing a room under a different illumination.

Estimating those factors of image formation constitutes the problem of inverse rendering, which is especially ill-posed when the input is only a few images of an indoor scene acquired with a commodity camera. While traditional measurement-based methods need expensive devices and do not scale to large-scale scenes, conventional learning methods suffer from a lack of data and expressive power to handle the above complex visual effects.

This research addresses the longstanding computer vision and graphics challenge of inverse rendering through a holistic approach of modeling complex materials and illumination, developing novel physically inspired deep networks and generating large-scale training data. The project will have a transformative effect on industry and society through photorealistic augmented reality and image editing applications such as view synthesis, object insertion, material, and light source editing, with accurate shadows and interreflections.

The project also incorporates experiential insights from the above technological advances into college and K-12 education through a new interdisciplinary curriculum in computer vision and computer graphics.

This project develops physically based deep networks that incorporate the inductive bias of image formation to achieve generalizable representations that estimate shape, material and especially lighting with unprecedented detail, by accounting for visibilities in a single image, consistency in multiple images and complex light transport in challenging or dynamic scenes. It designs novel modules such as neural rendering layers and reflectance volumes that incorporate the domain knowledge of image formation.

It achieves scalability through parsimonious representations that retain descriptive power while generalizing to complex lighting effects such as refraction, scattering and dynamic light transport. It develops large-scale training datasets with realistic spatially varying lighting and complex high-quality material.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.