Collaborative Research: HCC: Medium: Differentiable Rendering for Computer Graphics

Creating realistic images in computer graphics has historically relied on accurately computing the values at each point or pixel in the image based on physically accurate simulation of lighting in the scene, but recently it has become clear that simply computing image values is not adequate. One needs to also be able to understand how these values change with changes in the environment, for example as the sun moves across the sky, a door is opened letting light into the scene, or the material properties of an object are gradually changed from velvet to metallic.

Mathematically, this involves computing the derivatives of the image to determine how it changes with respect to the input parameters. This research will create a class of differentiable renderers that compute both images and their derivatives. Project outcomes will have broad impact because the computation of derivatives is increasingly central to many areas of computer graphics, computer vision, robotics and machine learning, with potential benefit to applications as diverse as perception control in self-driving cars and robots, optimization of indoor lighting for architecture, fabrication of 3D objects with a desired appearance, statistics and epidemiology.

Additional impact will derive from the fact that the PIs are educators committed to broadening participation in computing who participate in early research scholars programs and will develop new online courses in rendering.

Computing the derivatives or gradients of general light transport involves tackling fundamental challenges of differential calculus, Monte Carlo integration, signal processing, automatic differentiation, and metaprogramming systems. One challenge is in handling discontinuities of various forms, which lead to Dirac delta terms that require careful and analytic treatment that cannot be provided by traditional automatic differentiation.

Even for the smooth variation, computing gradients involves a large number of intermediate variables that necessitate tradeoffs across bias, variance, compute and memory. Moreover, full generality requires differentiable rendering in new representations such as implicit surfaces and procedural materials, as well as new problem domains such as transient rendering for non-line-of-sight imaging and geometrical diffraction for acoustics.

One also needs to effectively apply the gradients for optimization in inverse problems. This project will develop a broad transformative agenda, seeking to enable differentiable renderers to efficiently reconstruct billions of varied primitives from millions of pixels under general and diverse light transport situations. The research plan consists of four interconnected components involving computational foundations and efficient algorithms for solving visibility gradients including: analytic and area sampling methods; a unified system for exploring computational and memory tradeoffs in differentiable rendering algorithms; generalizations to new physical phenomena such as transient rendering and geometrical diffraction; and advances in inverse problems and deep learning including new approaches to continuous optimization involving Euler-Lagrange equations.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.